Containers can't connect via overlay network between hosts

Keiga · June 25, 2016, 9:27pm

Hello.

Need some help with Docker network config. By default my containers don’t have any network\internet access. My host machine has two interface (one(5.61.0.0) to local network, and another one to internet(10.38.0.0))

The only solution i found is to add iptable rule:
iptables -t nat -I POSTROUTING -p all -s 172.17.0.0/16 -j SNAT --to-source <my-host-ip-from-5.61.0.0-network>

But now if use Rancher default overlay network (10.42.0.0), containers can’t communicate with each other on different hosts by using 10.42.X.X ip. How can i configure Docker networking and\or Rancher to make it work together?

My system and network info:

-----
docker info

Containers: 35
 Running: 32
 Paused: 0
 Stopped: 3
Images: 4
Server Version: 1.11.2
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.6.2-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 31.39 GiB
Name: host.dev.dv
ID: W25R:CFB7:J3E7:SGDY:YEBF:BWHR:BR2F:LAJZ:U5FF:ONLM:ZXBQ:ZASQ
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/

-----------------

# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
CATTLE_PREROUTING  all  --  anywhere             anywhere            
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere            !loopback/8           ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
SNAT       all  --  172.17.0.0/16        anywhere             to:5.61.234.27
CATTLE_POSTROUTING  all  --  anywhere             anywhere            
MASQUERADE  all  --  172.17.0.0/16        anywhere            
MASQUERADE  udp  --  172.17.0.2           172.17.0.2           udp dpt:ipsec-nat-t
MASQUERADE  udp  --  172.17.0.2           172.17.0.2           udp dpt:isakmp

Chain CATTLE_POSTROUTING (1 references)
target     prot opt source               destination         
ACCEPT     all  --  10.42.0.0/16         169.254.169.250     
MASQUERADE  tcp  --  10.42.0.0/16        !10.42.0.0/16         masq ports: 1024-65535
MASQUERADE  udp  --  10.42.0.0/16        !10.42.0.0/16         masq ports: 1024-65535
MASQUERADE  all  --  10.42.0.0/16        !10.42.0.0/16        
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x3ad7e to:10.42.241.22
MASQUERADE  tcp  --  172.17.0.0/16        anywhere             masq ports: 1024-65535
MASQUERADE  udp  --  172.17.0.0/16        anywhere             masq ports: 1024-65535
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x34016 to:10.42.213.14

Chain CATTLE_PREROUTING (1 references)
target     prot opt source               destination         
DNAT       tcp  --  10.42.0.0/16         10.42.0.1            tcp dpt:domain to:169.254.169.250
DNAT       udp  --  10.42.0.0/16         10.42.0.1            udp dpt:domain to:169.254.169.250
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:DC:60:F3 MARK set 0x3ad7e
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:69:63:96:CE:07 MARK set 0x34016

Chain DOCKER (2 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            
DNAT       udp  --  anywhere             anywhere             udp dpt:ipsec-nat-t to:172.17.0.2:4500
DNAT       udp  --  anywhere             anywhere             udp dpt:isakmp to:172.17.0.2:500

----------------

# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.38.158.1     0.0.0.0         UG    0      0        0 eth0.3003
10.38.158.0     0.0.0.0         255.255.254.0   U     0      0        0 eth0.3003
10.42.0.0       0.0.0.0         255.255.0.0     U     0      0        0 docker0
10.255.2.0      10.38.158.1     255.255.255.0   UG    0      0        0 eth0.3003
link-local      0.0.0.0         255.255.0.0     U     1002   0        0 eth0
link-local      0.0.0.0         255.255.0.0     U     1029   0        0 extif0
link-local      0.0.0.0         255.255.0.0     U     1041   0        0 eth0.3003
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0

--------------

# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        inet6 fe80::42:97ff:fe54:899a  prefixlen 64  scopeid 0x20<link>
        ether 02:22:97:55:89:9a  txqueuelen 0  (Ethernet)
        RX packets 18078007  bytes 10782154070 (10.0 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 15374194  bytes 23394143099 (21.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::67d:7bff:fef1:a94c  prefixlen 64  scopeid 0x20<link>
        ether 09:7d:bb:f1:a9:4c  txqueuelen 1000  (Ethernet)
        RX packets 27472778  bytes 26086394861 (24.2 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 23364882  bytes 11529001372 (10.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xdfd20000-dfd3ffff  

eth0.3003: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.38.158.34  netmask 255.255.254.0  broadcast 10.38.159.255
        inet6 fe80::67d:7bff:fef1:a94c  prefixlen 64  scopeid 0x20<link>
        ether 09:7d:bb:f1:a9:4c  txqueuelen 1000  (Ethernet)
        RX packets 16103286  bytes 25113755309 (23.3 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 18688617  bytes 11220213569 (10.4 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

extif0: flags=195<UP,BROADCAST,RUNNING,NOARP>  mtu 1500
        inet 5.61.234.27  netmask 255.255.255.255  broadcast 5.61.234.27
        inet6 fe80::a022:eff:fe5e:cf3a  prefixlen 64  scopeid 0x20<link>
        ether a5:22:0d:5e:cf:3a  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3  bytes 210 (210.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 74882  bytes 1025810342 (978.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 74882  bytes 1025810342 (978.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethc57e0d2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::ec92:35ff:fe47:aa44  prefixlen 64  scopeid 0x20<link>
        ether ee:92:35:47:aa:44  txqueuelen 0  (Ethernet)
        RX packets 1144  bytes 95187 (92.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1084  bytes 12703905 (12.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:5e:40:52  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
----

Upayavira · June 26, 2016, 8:20pm

Overlay network uses a VPN under the covers. Traffic is routed from your container to the network agent (rancher/agent-instance) on each node, and from there it gets sent via the VPN using ports 500 and 4500(both UDP) to the network agent on the relevant node, which forwards it to the target container. If the necessary traffic is routable, then the VPN should function and the overlay network should work, although there have been reports of issues.

sjiveson · June 27, 2016, 12:23pm

You shouldn’t need an iptables rule and it’s likely this is breaking the overlay/VPNs between hosts.

The rule is not necessary. All Docker containers (Rancher related or not) will have outbound traffic source NATted to a host IP using a MASQUERADE rule. This is seen in the POSTROUTING table.

Which IP will depending on your routing table and your default route seem to point towards your local network, not your external/internet facing one?

Keiga · June 27, 2016, 1:21pm

The main issue that Docker engine by default can’t route traffic to a local network and to internet at the same time. That why i use additional iptables route. Docker binds on my local interface and there is no internet in containers. If i adding that rule - i have internet and network in the containers, but other rules (like cattle) don’t work properly. I am not strong at iptables configuration and i think there should be another solution.

Is Swarm overlay network work the same VPN-way as Rancher?

leodotcloud · June 27, 2016, 5:19pm

As pointed out by @Upayavira and @sjiveson, you don’t need to do additional configuration to get your containers to connect to the internet.

In the Rancher UI, when you deploy containers using the managed network option and specify which ports are exposed to the internet, they should start working.

Keiga · June 28, 2016, 7:45am

Yes, it should, but not working. Without that rule i can’t even add host to Rancher enviroment, agent can’t connect to master-host.

Upayavira · June 28, 2016, 12:00pm

Have you confirmed that each of your Docker hosts can communicate with each other on ports 500/4500 UDP? Confirm that (make it happen with iptables, etc) and your containers should be come able to communicate with each other.

sjiveson · June 28, 2016, 1:19pm

@Keiga, you should not need an SNAT rule in order to add hosts to the infrastructure and should not be doing it over a public interface either.

Regarding “Docker engine by default can’t route traffic to a local network and to internet at the same time” - this is untrue and I do so all the time both in private/home environments and in corporate ones behind firewalls and the like.

I would kindly and sincerely suggest you have someone who understands your network infrastructure well take a close look at each of your servers’ networking and IP configuration and then discuss your requirements with them to see how best to proceed.

Whilie you’re at it, I would also confirm IP Forwarding is enabled on each host too:

$ sysctl net.ipv4.ip_forward

Keiga · June 28, 2016, 7:12pm

Thank you for advice, i will try to understand my network. I mean that in my case if host machine has two separate interfaces, my docker containers can’t access internet or another host. Ip-forward is enabled on each host.

sjiveson · June 29, 2016, 1:32pm

Hey @Keiga, I thought it would be enabled. I don’t like saying it but it should ‘just work’ so clearly something is up. Do post back when you find the cause. Cheers

Upayavira · July 8, 2016, 9:07pm

A colleague just fixed overlay network on EC2. He had run the Rancher agent installation using the public IP address of the Rancher server. Consequently it registered the instances with Rancher using the external IPs which were not routable between the instances. Once he reinstalled the agent using the internal IP, and made sure that UDP 450/5000 were open internally, the overlay network came to life.

leodotcloud · July 9, 2016, 12:12am

@Upayavira I just launched my cluster on EC2 with the public IP address for the agent installation on the hosts and everything is working as expected. I did launch the ‘Wordpress’ Catalog item with the two containers getting deployed across two different hosts and they are able to reach each other. So I am not sure what issue you are referring to?

leodotcloud · July 9, 2016, 12:13am

$ uname -a
Linux 3.13.0-74-generic #118-Ubuntu SMP Thu Dec 17 22:52:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
$ docker --version
Docker version 1.10.3, build 20f81dd

Upayavira · July 11, 2016, 6:27am

Leo, I’m not suggesting there was a fault in Rancher. Merely responding to my initial comment that i had never seen the overlay network work.

In Amazon, the ip address you use to contact Rancher when registering a node defines the ip address of the node, and thus the security groups that you will require. We had the private network wide open, but used the public to register with Rancher.

leodotcloud · July 11, 2016, 8:18pm

@Upayavira: Cool! Glad to hear that things are working as expected.

Topic		Replies	Views
Overlay network issue Rancher 1.x	2	926	September 7, 2016
Cross-host intercontainer communication trouble Rancher 1.x	27	12102	May 14, 2016
How to bridge the overlay network with the hosts network? Rancher 1.x	2	1966	October 5, 2015
Cross Host Networking Issues Rancher 1.x	3	1923	April 9, 2017
No connection between Host Rancher 1.x	13	4162	April 12, 2016

Containers can't connect via overlay network between hosts

Related topics