How to troubleshoot: Containers cannot ping some other containers in the same stack, but Infrastructure Stacks seem fine

CentOS 7.3 VMs in VMware. Rancher 1.5.10.

I have three Rancher environments in two Rancher clusters, and two environments have the following problem.

  1. I read Troubleshooting FAQs: CROSS HOST COMMUNICATION, and everything looks okay.
  • The Infrastructure Stacks are all green
  • The Healthcheck stack is green
  • The IPs of the hosts are correct in the UI

In this case, I have a cluster of three VMs. I created a new stack with one service, and one container on each node. From each node, I pinged all three internal IPs for each container in the service. The results are:

  • container-1 can ping itself and container-2
  • container-2 can ping container-1, itself and container-3
  • container-3 can ping container-2 and container-3

Why can containers 1 & 3 not ping each other? What other troubleshooting can I do? Can I check anything in iptables?