Cannot connect between hosts after upgrade from 0.47 to 0.59.1

Archive Rancher 1.x
1

Hi, I upgraded from 0.47 to 0.59.1 today and after a few glitches got things restarted.

Problem now is that I can’t connect between hosts. IP_forward is definitely on but I notice that the IPTABLES rules look slightly different. There’s now a chain DOCKER which forwards UDP 500,4500 to the Docker IP not the Rancher IP. Is it that?

1
[root@ld3-docker-1 ~]# iptables -L -n --line-numbers -t nat
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination
1    CATTLE_PREROUTING  all  --  0.0.0.0/0            0.0.0.0/0
2    DOCKER     all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination
1    DOCKER     all  --  0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination
1    CATTLE_POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0
2    MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0
3    MASQUERADE  udp  --  172.17.0.1           172.17.0.1           udp dpt:4500
4    MASQUERADE  udp  --  172.17.0.1           172.17.0.1           udp dpt:500
5    MASQUERADE  udp  --  172.17.0.1           172.17.0.1           udp dpt:4500
6    MASQUERADE  udp  --  172.17.0.1           172.17.0.1           udp dpt:500
7    MASQUERADE  udp  --  172.17.0.2           172.17.0.2           udp dpt:4500
8    MASQUERADE  udp  --  172.17.0.2           172.17.0.2           udp dpt:500

Chain CATTLE_POSTROUTING (1 references)
num  target     prot opt source               destination
1    ACCEPT     all  --  10.42.0.0/16         169.254.169.250
2    MASQUERADE  tcp  --  10.42.0.0/16        !10.42.0.0/16         masq ports: 1024-65535
3    MASQUERADE  udp  --  10.42.0.0/16        !10.42.0.0/16         masq ports: 1024-65535
4    MASQUERADE  all  --  10.42.0.0/16        !10.42.0.0/16
5    MASQUERADE  tcp  --  172.17.0.0/16        0.0.0.0/0            masq ports: 1024-65535
6    MASQUERADE  udp  --  172.17.0.0/16        0.0.0.0/0            masq ports: 1024-65535

Chain CATTLE_PREROUTING (1 references)
num  target     prot opt source               destination
1    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL tcp dpt:8761 to:10.42.103.22:8761
2    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL tcp dpt:8000 to:10.42.28.184:80
3    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL tcp dpt:8081 to:10.42.190.164:8081
4    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL tcp dpt:8893 to:10.42.62.5:80
5    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL tcp dpt:8084 to:10.42.248.135:8084
6    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL tcp dpt:9104 to:10.42.177.228:9104

Chain DOCKER (2 references)
num  target     prot opt source               destination
1    DNAT       udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:4500 to:172.17.0.2:4500
2    DNAT       udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:500 to:172.17.0.2:500
2

Just so people know. I managed to fix this by rebuilding my hosts. Shouldn’t have to but there we have it. New hosts were constructed using exactly the same puppet scripts as the old ones. I just suspect something was outdated on them.