HTTP 1.1 Health checks failing

Rancher 1.x
1

Health check is configured as follows:

health_check:
  port: 18080
  request_line: "GET \"integration-server/agents/health/${ANSWER_HEALTH}\" \"HTTP/1.1\\r\\nHost: iserver\""
  interval: 10000
  initializing_timeout: 120000
  response_timeout: 5000
  unhealthy_threshold: 3
  healthy_threshold: 2
  strategy: recreate

However, server never gets passed initializing before it is removed. An important factor may be that iserver is service alias to a service in another stack, where port 18080 is private container port. HEALTH_ANSWER is provided during stack creation as an answer

In the health check logs I see this:

time=“2017-06-22T00:27:24Z” level=info msg=“healthCheck – reloading haproxy config with the new config changes\n[WARNING] 172/002724 (18954) : config : ‘option forwardfor’ ignored for proxy ‘web’ as it requires HTTP mode.\n[WARNING] 172/002724 (18954) : config : ‘option forwardfor’ ignored for backend ‘cattle-e0e3104e-6e3b-48a3-8cca-7edf9fc290ad_c3b102eb-131d-4e93-91e3-3a4a25f1158c_1’ as it requires HTTP mode.\n[WARNING] 172/002724 (18954) : config : ‘option forwardfor’ ignored for backend ‘cattle-e0e3104e-6e3b-48a3-8cca-7edf9fc290ad_32cdbc83-68c2-47ea-b603-2ed805746f78_1’ as it requires HTTP mode.\n[WARNING] 172/002724 (18954) : [haproxy.main()] Cannot raise FD limit to 10274, limit is 4096.\n[WARNING] 172/002724 (18954) : [haproxy.main()] FD limit (4096) too low for maxconn=4096/maxsock=10274. Please raise ‘ulimit-n’ to 10274 or more to avoid any trouble.\n”

Manually executing the endpoint from the server when its initializing is working fine.

Its unclear whether or not:

  • My request line is formatted properly
  • Whether the warnings presented in the health check log are more than that.

Any help would be appreciated.

1 Like
2

The request line is wrong in multiple ways…

request_line: "GET /integration-server/agents/health/${ANSWER_HEALTH} HTTP/1.1\r\nHost: iserver"
3

Thanks Vincent I have used this verbatim, but am still faced with issues on the haproxy side:

6/22/2017 8:12:51 AMtime=“2017-06-22T12:12:51Z” level=info msg="Scheduling apply config"
6/22/2017 8:12:51 AMtime=“2017-06-22T12:12:51Z” level=info msg="healthCheck – reloading haproxy config with the new config changes\n[ALERT] 172/121251 (20739) : parsing [/etc/haproxy/haproxy.cfg:45] : unknown keyword ‘Host:’ in ‘backend’ section\n[ALERT] 172/121251 (20739) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg\n[WARNING] 172/121251 (20739) : config : ‘option forwardfor’ ignored for proxy ‘web’ as it requires HTTP mode.\n[WARNING] 172/121251 (20739) : config : ‘option forwardfor’ ignored for backend ‘cattle-e0e3104e-6e3b-48a3-8cca-7edf9fc290ad_c3b102eb-131d-4e93-91e3-3a4a25f1158c_1’ as it requires HTTP mode.\n[WARNING] 172/121251 (20739) : config : ‘option forwardfor’ ignored for backend ‘cattle-e0e3104e-6e3b-48a3-8cca-7edf9fc290ad_32cdbc83-68c2-47ea-b603-2ed805746f78_1’ as it requires HTTP mode.\n[ALERT] 172/121251 (20739) : Fatal errors found in configuration.\n"
6/22/2017 8:12:51 AMtime=“2017-06-22T12:12:51Z” level=error msg=“Error applying config: error reloading healthCheck – reloading haproxy config with the new config changes\n[ALERT] 172/121251 (20739) : parsing [/etc/haproxy/haproxy.cfg:45] : unknown keyword ‘Host:’ in ‘backend’ section\n[ALERT] 172/121251 (20739) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg\n[WARNING] 172/121251 (20739) : config : ‘option forwardfor’ ignored for proxy ‘web’ as it requires HTTP mode.\n[WARNING] 172/121251 (20739) : config : ‘option forwardfor’ ignored for backend ‘cattle-e0e3104e-6e3b-48a3-8cca-7edf9fc290ad_c3b102eb-131d-4e93-91e3-3a4a25f1158c_1’ as it requires HTTP mode.\n[WARNING] 172/121251 (20739) : config : ‘option forwardfor’ ignored for backend ‘cattle-e0e3104e-6e3b-48a3-8cca-7edf9fc290ad_32cdbc83-68c2-47ea-b603-2ed805746f78_1’ as it requires HTTP mode.\n[ALERT] 172/121251 (20739) : Fatal errors found in configuration.\n: exit status 1”

Healthcheck is running at v0.3.1 if that matters. Is this a bug or a configuration issue at this point? Either way can you suggestion any path forward?

1 Like
5

In order to troubleshoot and narrow down the issues I have taken the following steps:

  1. Installed a new rancher server and added 1 host
  2. Selected a plain docker public container ( tomcat )
  3. Run this container with no health checks and its starts fine
  4. Cloned the container and configured http 1.1 health check in the UI, so that it goes to an external service where I can inspect the health check requests coming in. This service is simply ngrok pointed to my local host. Inspection occurs via the standard ngrok webapp.

Observations:

  • No requests seem to be generated to my external endpoint
  • Service flaps between initializing and restarting
  • Health check logs show this warning:

time=“2017-06-25T13:58:20Z” level=info msg=“healthCheck – reloading haproxy config with the new config changes\n[WARNING] 175/135820 (678) : config : ‘option forwardfor’ ignored for proxy ‘web’ as it requires HTTP mode.\n[WARNING] 175/135820 (678) : config : ‘option forwardfor’ ignored for backend ‘cattle-76d6f144-b907-46e9-b788-0404aab71362_2c6e2697-6502-4f6f-b207-7cfefb5c306b_1’ as it requires HTTP mode.\n[WARNING] 175/135820 (678) : [haproxy.main()] Cannot raise FD limit to 10274, limit is 4096.\n[WARNING] 175/135820 (678) : [haproxy.main()] FD limit (4096) too low for maxconn=4096/maxsock=10274. Please raise ‘ulimit-n’ to 10274 or more to avoid any trouble.\n”

Been at this for a few days now, and I would be more than happy to share access to this environment with anyone from the rancher team in order to resolve this.

Thanks!

1 Like