Cross Host Communication Problem [FIXED]

Archive Rancher 1.x
1

I’ve run into an issue where cross host communication has stopped working. I’ve followed the steps outlined on the troubleshooting section of the docs and confirmed that Host B cannot ping containers on Host A, and visa versa. The troubleshooting section outlines how to confirm the issue exists, but doesn’t tell you how to resolve the issue.

I’ve tried:

  • Restarting the EC2 instances for each host
  • Creating fresh hosts through the Rancher UI (Provider: AWS, AMI: default, OS: Ubuntu 15.10)
  • Updating Rancher Server version to the latest stable release (v1.1.1)
  • Upgrading Docker to version 1.11.2 on the Rancher server
  • Ensure UFW is disabled
  • Restarted the “Network Agent” container on each host
  • Launched new containers on each host

Any ideas on how to get cross host communication working again would be greatly appreciated.

2

I found the issue. I allocated an Elastic IP to one of the hosts, so the IP that was being reported in the UI was no longer used by host.

I tried following the steps under the heading “ARE THE IPS OF THE HOSTS CORRECT IN THE UI?” in the troubleshooting section of the docs, but it was unclear what http://SERVER_IP:8080/v1/scripts/xxxx was supposed to be replaced with. I found the proper URL in the Rancher UI under the add a custom host section.

Once the proper IP was set cross host communication started working again.