Ensuring Cross-Host Communication

I am having a problem establishing cross-host communication on my rancher set up.

I have two physical hosts running RancherOS and on the first host I have Rancher server running.
The problem I am encountering is the healthcheck container on host #2 is stuck in an ‘initializing’ state. I have read in a few places that this indicates there is a problem communicating between hosts. I followed the steps in the Toubleshooting section to try pinging the healthcheck container running host #1 from the healthcheck container on host #2 and I found that I could not reach the other container.

At this point, my suspicion is that it could be a firewall issue but I am not sure where to look or what to try next. Are there any specific ports that need to be open on a router to allow cross-host communication?

For clarity, both these hosts are plugged into a physical hardware firewall router.

Thanks!

The ports needed are described here: http://rancher.com/docs/rancher/v1.6/en/hosts/custom/#security-groupsfirewalls

For ipsec (default overlay network) it’s UDP/500 and UDP/4500 between all the IPs that are shown in the Infrastructure -> Hosts view.