URGENT: Can't get Load Balancer to work after 0.59 upgrade

Archive Rancher 1.x
1

Hi folks,

My load balancer (the config of which worked fine in 0.56) now doesn’t get past Initializing in the new 0.59 version.

I’ve confirmed communication between containers (pinging between active containers)

Can anybody help me get this running? I’m pretty desperate…

2

For reference, here’s the communications test:

root@8a43e534fe49:/# nc -v 10.42.205.218 42                                     
nc: connect to 10.42.205.218 port 42 (tcp) failed: Connection refused           

root@8a43e534fe49:/# nc -v 10.42.205.218 80                                     
nc: connect to 10.42.205.218 port 80 (tcp) failed: Connection refused           

root@8a43e534fe49:/# ping  10.42.205.218                                        
PING 10.42.205.218 (10.42.205.218) 56(84) bytes of data.                        
64 bytes from 10.42.205.218: icmp_seq=1 ttl=63 time=0.220 ms                    
64 bytes from 10.42.205.218: icmp_seq=2 ttl=63 time=0.214 ms

This is a similar test to a container with port 3000 open to http traffic:

root@8a43e534fe49:/# nc -v 10.42.187.201 3000                                   
Connection to 10.42.187.201 3000 port [tcp/*] succeeded!
3

@drmikecrowe could you check if haproxy is running inside the lb instance?

service haproxy status

if haproxy process is not running, nothing listens on port 42, therefore health check fails

4

That appeared to be the issue.

I’m actually downgrading to 0.56.1 now to try and get the system back online.

5

If you run into this issue again, could you try to execute the following command from inside haproxy:

service haproxy start

and see what error you get.

Could you also share your haproxy config? I would like to try it on my side. We’ve upgraded haproxy to 1.6, and I suspect that there might be some parameter that haproxy 1.6 doesn’t support any more. We would have to fix it on our side.

6

FWIW, 0.56 brought my site back up

7

@drmikecrowe any chance you can share your haproxy config (/etc/haproxy/haproxy.cfg from lb instance)? I would like to debug it on the latest, with haproxy 1.6

8

The problem is with this param in your custom haproxy_config:

listen stats :1936

haproxy 1.6 no longer supports this directive, and errors out with

[ALERT] 054/035008 (518) : parsing [/etc/haproxy/haproxy.cfg:10] : please use the ‘bind’ keyword for listening addresses.

Custom config should look as follows:

		listen stats 
                bind 0.0.0.0:1936
...........other parameters

I’ve tested it locally, works fine after the fix. You can try manually upgrading one of your existing lb instances to 1.6

9

Awesome, thanks @alena. I’ll try a manual upgrade tonight

10

Upgrade went smooth. Thanks for your help @alena