Rancher Server in HA

We have used exact commands to create two rancher nodes in fully active HA and we were able to do it with remote mysql. However, after adding hosts, we tried to remove the first rancher server from haproxy, but its not working as expected. Even though rancher UI is accessible(sometimes its showing 503), all hosts are saying its disconnected while using just the second server. While checking from the hosts to connect to rancher server, its simply failing with “503 service unavailable” which means some issues with rancher container itself. Error is below

ERROR [:] [] [] [] [TaskScheduler-2] [i.c.p.a.s.ping.impl.PingMonitorImpl ] Failed to get ping from agent [205] count [7]

If we enable just the first server in haproxy, its working smoothly. Is there anything else need to be taken care? Will Rancher HA work fine with just two nodes?

Can you share how you setup your HA cluster? The docker run commands you used and the haproxy config? This would help to get to the bottom of this.