However, server never gets passed initializing before it is removed. An important factor may be that iserver is service alias to a service in another stack, where port 18080 is private container port. HEALTH_ANSWER is provided during stack creation as an answer
In the health check logs I see this:
time=â2017-06-22T00:27:24Zâ level=info msg=âhealthCheck â reloading haproxy config with the new config changes\n[WARNING] 172/002724 (18954) : config : âoption forwardforâ ignored for proxy âwebâ as it requires HTTP mode.\n[WARNING] 172/002724 (18954) : config : âoption forwardforâ ignored for backend âcattle-e0e3104e-6e3b-48a3-8cca-7edf9fc290ad_c3b102eb-131d-4e93-91e3-3a4a25f1158c_1â as it requires HTTP mode.\n[WARNING] 172/002724 (18954) : config : âoption forwardforâ ignored for backend âcattle-e0e3104e-6e3b-48a3-8cca-7edf9fc290ad_32cdbc83-68c2-47ea-b603-2ed805746f78_1â as it requires HTTP mode.\n[WARNING] 172/002724 (18954) : [haproxy.main()] Cannot raise FD limit to 10274, limit is 4096.\n[WARNING] 172/002724 (18954) : [haproxy.main()] FD limit (4096) too low for maxconn=4096/maxsock=10274. Please raise âulimit-nâ to 10274 or more to avoid any trouble.\nâ
Manually executing the endpoint from the server when its initializing is working fine.
Its unclear whether or not:
My request line is formatted properly
Whether the warnings presented in the health check log are more than that.
Thanks Vincent I have used this verbatim, but am still faced with issues on the haproxy side:
6/22/2017 8:12:51 AMtime=â2017-06-22T12:12:51Zâ level=info msg="Scheduling apply config"
6/22/2017 8:12:51 AMtime=â2017-06-22T12:12:51Zâ level=info msg="healthCheck â reloading haproxy config with the new config changes\n[ALERT] 172/121251 (20739) : parsing [/etc/haproxy/haproxy.cfg:45] : unknown keyword âHost:â in âbackendâ section\n[ALERT] 172/121251 (20739) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg\n[WARNING] 172/121251 (20739) : config : âoption forwardforâ ignored for proxy âwebâ as it requires HTTP mode.\n[WARNING] 172/121251 (20739) : config : âoption forwardforâ ignored for backend âcattle-e0e3104e-6e3b-48a3-8cca-7edf9fc290ad_c3b102eb-131d-4e93-91e3-3a4a25f1158c_1â as it requires HTTP mode.\n[WARNING] 172/121251 (20739) : config : âoption forwardforâ ignored for backend âcattle-e0e3104e-6e3b-48a3-8cca-7edf9fc290ad_32cdbc83-68c2-47ea-b603-2ed805746f78_1â as it requires HTTP mode.\n[ALERT] 172/121251 (20739) : Fatal errors found in configuration.\n"
6/22/2017 8:12:51 AMtime=â2017-06-22T12:12:51Zâ level=error msg=âError applying config: error reloading healthCheck â reloading haproxy config with the new config changes\n[ALERT] 172/121251 (20739) : parsing [/etc/haproxy/haproxy.cfg:45] : unknown keyword âHost:â in âbackendâ section\n[ALERT] 172/121251 (20739) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg\n[WARNING] 172/121251 (20739) : config : âoption forwardforâ ignored for proxy âwebâ as it requires HTTP mode.\n[WARNING] 172/121251 (20739) : config : âoption forwardforâ ignored for backend âcattle-e0e3104e-6e3b-48a3-8cca-7edf9fc290ad_c3b102eb-131d-4e93-91e3-3a4a25f1158c_1â as it requires HTTP mode.\n[WARNING] 172/121251 (20739) : config : âoption forwardforâ ignored for backend âcattle-e0e3104e-6e3b-48a3-8cca-7edf9fc290ad_32cdbc83-68c2-47ea-b603-2ed805746f78_1â as it requires HTTP mode.\n[ALERT] 172/121251 (20739) : Fatal errors found in configuration.\n: exit status 1â
Healthcheck is running at v0.3.1 if that matters. Is this a bug or a configuration issue at this point? Either way can you suggestion any path forward?
In order to troubleshoot and narrow down the issues I have taken the following steps:
Installed a new rancher server and added 1 host
Selected a plain docker public container ( tomcat )
Run this container with no health checks and its starts fine
Cloned the container and configured http 1.1 health check in the UI, so that it goes to an external service where I can inspect the health check requests coming in. This service is simply ngrok pointed to my local host. Inspection occurs via the standard ngrok webapp.
Observations:
No requests seem to be generated to my external endpoint
Service flaps between initializing and restarting
Health check logs show this warning:
time=â2017-06-25T13:58:20Zâ level=info msg=âhealthCheck â reloading haproxy config with the new config changes\n[WARNING] 175/135820 (678) : config : âoption forwardforâ ignored for proxy âwebâ as it requires HTTP mode.\n[WARNING] 175/135820 (678) : config : âoption forwardforâ ignored for backend âcattle-76d6f144-b907-46e9-b788-0404aab71362_2c6e2697-6502-4f6f-b207-7cfefb5c306b_1â as it requires HTTP mode.\n[WARNING] 175/135820 (678) : [haproxy.main()] Cannot raise FD limit to 10274, limit is 4096.\n[WARNING] 175/135820 (678) : [haproxy.main()] FD limit (4096) too low for maxconn=4096/maxsock=10274. Please raise âulimit-nâ to 10274 or more to avoid any trouble.\nâ
Been at this for a few days now, and I would be more than happy to share access to this environment with anyone from the rancher team in order to resolve this.