Cross-host communication - ping only works between some containers

I know this is a rehashed question, but I can’t find anyone with my specific problem… Links between services do not work when containers are on different hosts.

I have three hosts (all Ubuntu 16.04) in my Cattle environment:

h1 - rancher server v1.6.17
h2 - rancher agent v1.2.10. cattle agent ip = 192.168.x.9
h3 - rancher agent v1.2.10. cattle agent ip = 192.168.x.10

Service container on h3 has a link to service container on h2. But the named link doesn’t work and pinging from the container on h3 to the container on h2 says Destination Host Unreachable.

Some diagnostics:

  • Nmap reports UDP 500 and 4500 open between all servers. All infrastructure containers including health checks are green and report no errors in logs.
  • Any container on any host can ping the hosts on their 192.168.x addresses
  • Ipsec container on h2 can ping ipsec container on h3
  • Ipsec container on h3 can ping ipsec container on h2
  • Ipsec container on h2 can ping service container on h3
  • Ipsec container on h3 can NOT ping service container on h2
  • Service container on h2 can ping ipsec container on h2 but NOT on h3
  • Service container on h3 can ping ipsec container on both h2 and h3

I am so confused… Any ideas where to dig?

Apparently upgrading from docker 17 to 18.03 made it work…