I had exactly this issue in a local environment.
When adding a second rancher server on the same database, it gets registered immediately in the gui.
From within the second rancher server container, I can perfectly ping the other dockerhost on 10.10.31.11.
In the logs however, the message is repeated:
…
CATTLE_CATTLE_VERSION=v0.181.13
…
CATTLE_RANCHER_CLI_VERSION=v0.6.2
CATTLE_RANCHER_COMPOSE_VERSION=v0.12.5
…
[main] INFO ConsoleStatus - DB migration done
[main] INFO ConsoleStatus - Cluster membership changed [10.10.31.11:9345, 10.10.32.11:9345]
[main] INFO ConsoleStatus - Checking cluster state on start-up
[main] ERROR ConsoleStatus - Failed to connect to master at 10.10.31.11
[main] ERROR ConsoleStatus - Failed to connect to master at 10.10.31.11
[main] ERROR ConsoleStatus - Failed to connect to master at 10.10.31.11
SOLUTION: unblock traffic on port 9345 via iptables.
quick check telnet master docker host from other host on port 9345
If refusal -> this issue