Load Balancer doesn't start

Rancher 1.x
1

One of my load balancers refuses to start and is constantly stuck in the “Activating” phase. I get the following logs from the container:

2/11/2016 1:49:46 PMINFO: Downloading agent https://rancher.deepgenomics.com/v1/configcontent/configscripts
2/11/2016 1:49:46 PMINFO: Updating configscripts
2/11/2016 1:49:46 PMINFO: Downloading https://rancher.deepgenomics.com/v1//configcontent//configscripts current=
2/11/2016 1:49:46 PMINFO: Running /var/lib/cattle/download/configscripts/configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319/apply.sh
2/11/2016 1:49:46 PMINFO: Sending configscripts applied 1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319
2/11/2016 1:49:46 PMINFO: Updating agent-instance-startup
2/11/2016 1:49:46 PMINFO: Downloading https://rancher.deepgenomics.com/v1//configcontent//agent-instance-startup current=
2/11/2016 1:49:46 PMINFO: Running /var/lib/cattle/download/agent-instance-startup/agent-instance-startup-1-be633e113b20a4d8473083d27c28e6fac9761b174fdee8a518cbe003bb3a3a9a/apply.sh
2/11/2016 1:49:46 PMINFO: Updating services
2/11/2016 1:49:46 PMINFO: Downloading https://rancher.deepgenomics.com/v1//configcontent//services current=
2/11/2016 1:49:46 PMINFO: Running /var/lib/cattle/download/services/services-1-061405f3edd960bfdfe1cfb8447be40eab5b4b608731608e224cc51c5dc30b91/apply.sh
2/11/2016 1:49:46 PMINFO: HOME -> ./
2/11/2016 1:49:46 PMINFO: HOME -> ./services
2/11/2016 1:49:46 PMINFO: Sending services applied 1-061405f3edd960bfdfe1cfb8447be40eab5b4b608731608e224cc51c5dc30b91
2/11/2016 1:49:46 PMINFO: Getting agent-instance-scripts
2/11/2016 1:49:46 PMINFO: Updating agent-instance-scripts
2/11/2016 1:49:46 PMINFO: Downloading https://rancher.deepgenomics.com/v1//configcontent//agent-instance-scripts current=
2/11/2016 1:49:46 PMINFO: Running /var/lib/cattle/download/agent-instance-scripts/agent-instance-scripts-1-4b5124bd74cd423f98d57550b481ec77ec3a7135c6a650886ab95c043305d642/apply.sh
2/11/2016 1:49:46 PMINFO: HOME -> ./
2/11/2016 1:49:46 PMINFO: HOME -> ./events/
2/11/2016 1:49:46 PMINFO: HOME -> ./events/config.update
2/11/2016 1:49:46 PMINFO: HOME -> ./events/ping
2/11/2016 1:49:46 PMINFO: Sending agent-instance-scripts applied 1-4b5124bd74cd423f98d57550b481ec77ec3a7135c6a650886ab95c043305d642
2/11/2016 1:49:46 PMINFO: Getting monit
2/11/2016 1:49:46 PMINFO: Updating monit
2/11/2016 1:49:46 PMINFO: Downloading https://rancher.deepgenomics.com/v1//configcontent//monit current=
2/11/2016 1:49:46 PMINFO: Running /var/lib/cattle/download/monit/monit-1-4bf21eaa37373e2770bfb922848a07f801e32826d5a1048c98e5c2b8e7ff2dcc/apply.sh
2/11/2016 1:49:46 PMINFO: ROOT -> ./
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/logrotate.d/
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/logrotate.d/rancher-logs
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/monit/
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/monit/monitrc
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/monit/conf.d/
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/monit/conf.d/logrotate
2/11/2016 1:49:46 PMINFO: Sending monit applied 1-4bf21eaa37373e2770bfb922848a07f801e32826d5a1048c98e5c2b8e7ff2dcc
2/11/2016 1:49:46 PMINFO: Getting haproxy
2/11/2016 1:49:46 PMINFO: Updating haproxy
2/11/2016 1:49:46 PMINFO: Downloading https://rancher.deepgenomics.com/v1//configcontent//haproxy current=
2/11/2016 1:49:46 PMINFO: Running /var/lib/cattle/download/haproxy/haproxy-2-3e0d61c5cd2851e80416ea7bdd7aa59b417a3f11fa7852d0f0f91a26a6cdadb9/apply.sh
2/11/2016 1:49:46 PMINFO: ROOT -> ./
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/default/
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/default/haproxy
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/monit/
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/monit/conf.d/
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/monit/conf.d/haproxy
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/haproxy/
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/haproxy/certs/
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/haproxy/certs/certs.pem
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/haproxy/certs/default.pem
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/haproxy/haproxy.cfg
2/11/2016 1:49:46 PMINFO: Sending haproxy applied 2-3e0d61c5cd2851e80416ea7bdd7aa59b417a3f11fa7852d0f0f91a26a6cdadb9
2/11/2016 1:49:46 PMINFO: HOME -> ./
2/11/2016 1:49:46 PMINFO: HOME -> ./etc/
2/11/2016 1:49:46 PMINFO: HOME -> ./etc/cattle/
2/11/2016 1:49:46 PMINFO: HOME -> ./etc/cattle/startup-env
2/11/2016 1:49:46 PMINFO: ROOT -> ./
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/init.d/
2/11/2016 1:49:46 PMINFO: ROOT -> ./etc/init.d/agent-instance-startup
2/11/2016 1:49:46 PMINFO: Sending agent-instance-startup applied 1-be633e113b20a4d8473083d27c28e6fac9761b174fdee8a518cbe003bb3a3a9a
2/11/2016 1:49:46 PMmonit: generated unique Monit id 9051ae6df028c4b37e2f1caa8c33ca20 and stored to '/var/lib/monit/id'
2/11/2016 1:49:46 PMStarting monit daemon with http interface at [localhost:2812]
2/11/2016 1:51:36 PM
The system is going down NOW!
2/11/2016 1:51:36 PM
Sent SIGTERM to all processes
2/11/2016 1:51:37 PM
Sent SIGKILL to all processes
2/11/2016 1:51:37 PM
Requesting system reboot

Have you seen this before?

2

I have similar “unstability” issues, sometimes it’s stuck in activating, sometimes in “networking”. And it stays like this forever. I usually need to delete the service and recreate it to make it work. Restarting all the network agents also sometimes solves the issues.

3

What versions of Rancher are you running? Is this an upgraded setup?

4

I’m running v0.56.1, going to upgrade to v0.59.0 to see if it solves the issue. It’s an upgraded version.