Cross-Host communication

Hello together,

I have kind a question about cross-host communication between containers.

Scenario:

  • I have 10 AWS-EC2 Instances with Ubuntu15.10, Docker 1.12.5 and Rancher 1.2.1
  • By set up all clusters (kafka, zookeeper and applications using kafka), everything is working fine
  • After three days I can observe following behaviour:
    ** I name the ec2 instances, where kafka-containers are run, kafka-1…kafka-x
    ** Communication from kafka-1…kafka-x containers to other EC2 instances’ containers is working
    ** Communication from kafka-1 containers to kafka-2,…,kafka-x is not working
    ** ping command confirms the issue: I can ping from kafka containers to other instances’ containers. If i want to ping from kafka-1 containers to other kafka-instances containers, it is not working
    ** I use managed network for kafka containers. Default Network for every stack/service

Solution:

  • A restart of ipsec (rancher/net:v0.7.5) will restart the healthcheck and kafka-container too and the problem is gone. After 3 days it’s happening again.
  • I went through http://docs.rancher.com/rancher/v1.2/en/faqs/troubleshooting/ and everything seems ok
  • No limitation about security groups and vpn rules on AWS Instances

How can I avoid this behaviour?

The problem seems to be like in this topic: IPSec network fails silently on a host