Iptables NAT forwarding between centos 7 hosts doesn't work

I’m having real issues with networking using a centos 7 docker host. I’ve opened up a lot of rules but I can’t see between hosts. Almost the same iptables setup on rancheros works fine but I have other issues with rancheros. I’m running several hosts and on each is a prometheus monitoring agent. I can see the ones on the rancheros nodes but not on the centos nodes.

[root@ld3-docker-1 ~]# tcping 10.42.199.180 9104 (rancheros)
10.42.199.180 port 9104 open.
[root@ld3-docker-1 ~]# tcping 10.42.61.179 9104 (centos)
(hang) ^C

my iptables config is:

Generated by iptables-save v1.4.21 on Wed Aug 19 16:44:58 2015

*mangle
:PREROUTING ACCEPT [215:28955]
:INPUT ACCEPT [141:15844]
:FORWARD ACCEPT [71:12346]
:OUTPUT ACCEPT [95:21972]
:POSTROUTING ACCEPT [160:31846]
COMMIT

Completed on Wed Aug 19 16:44:58 2015

Generated by iptables-save v1.4.21 on Wed Aug 19 16:44:58 2015

*nat
:PREROUTING ACCEPT [17:4017]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [5:300]
:CATTLE_POSTROUTING - [0:0]
:CATTLE_PREROUTING - [0:0]
:DOCKER - [0:0]
-A PREROUTING -j CATTLE_PREROUTING
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -j CATTLE_POSTROUTING
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.17.0.1/32 -d 172.17.0.1/32 -p udp -m udp --dport 4500 -j MASQUERADE
-A POSTROUTING -s 172.17.0.1/32 -d 172.17.0.1/32 -p udp -m udp --dport 500 -j MASQUERADE
-A CATTLE_POSTROUTING -s 10.42.0.0/16 -d 169.254.169.250/32 -j ACCEPT
-A CATTLE_POSTROUTING -s 10.42.0.0/16 ! -d 10.42.0.0/16 -p tcp -j MASQUERADE --to-ports 1024-65535
-A CATTLE_POSTROUTING -s 10.42.0.0/16 ! -d 10.42.0.0/16 -p udp -j MASQUERADE --to-ports 1024-65535
-A CATTLE_POSTROUTING -s 10.42.0.0/16 ! -d 10.42.0.0/16 -j MASQUERADE
-A CATTLE_POSTROUTING -s 172.17.0.0/16 ! -o docker0 -p tcp -j MASQUERADE --to-ports 1024-65535
-A CATTLE_POSTROUTING -s 172.17.0.0/16 ! -o docker0 -p udp -j MASQUERADE --to-ports 1024-65535
-A CATTLE_PREROUTING -p tcp -m addrtype --dst-type LOCAL -m tcp --dport 9000 -j DNAT --to-destination 10.42.111.51:9090
-A CATTLE_PREROUTING -p udp -m addrtype --dst-type LOCAL -m udp --dport 4500 -j DNAT --to-destination 10.42.63.199:4500
-A CATTLE_PREROUTING -p udp -m addrtype --dst-type LOCAL -m udp --dport 500 -j DNAT --to-destination 10.42.63.199:500
-A DOCKER ! -i docker0 -p udp -m udp --dport 4500 -j DNAT --to-destination 172.17.0.1:4500
-A DOCKER ! -i docker0 -p udp -m udp --dport 500 -j DNAT --to-destination 172.17.0.1:500
COMMIT

Completed on Wed Aug 19 16:44:58 2015

Generated by iptables-save v1.4.21 on Wed Aug 19 16:44:58 2015

*filter
:INPUT DROP [0:0]
:FORWARD ACCEPT [6:2472]
:OUTPUT ACCEPT [95:21972]
:DOCKER - [0:0]
:fail2ban-SSH - [0:0]
-A INPUT -i lo -m comment --comment “000 accept all to lo interface” -j ACCEPT
-A INPUT -p icmp -m comment --comment “001 accept all icmp” -j ACCEPT
-A INPUT -m comment --comment “002 accept related established rules” -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m multiport --ports 22 -m comment --comment “099 send ssh connections to fail2ban chain” -m state --state NEW -j fail2ban-SSH
-A INPUT -s 10.20.0.0/16 -p tcp -m multiport --ports 22 -m comment --comment “100 accept ssh for PI” -j ACCEPT
-A INPUT -s 10.20.0.0/16 -p tcp -m multiport --ports 5666 -m comment --comment “110 accept nrpe for nagios” -j ACCEPT
-A INPUT -s 10.20.0.0/16 -p tcp -m multiport --ports 2375 -m comment --comment “200 docker for PI” -j ACCEPT
-A INPUT -s 10.20.0.0/16 -p tcp -m multiport --ports 5555 -m comment --comment “200 pi_sysadmin accept for PI” -j ACCEPT
-A INPUT -m comment --comment “999 drop all” -j DROP
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.17.0.1/32 ! -i docker0 -o docker0 -p udp -m udp --dport 4500 -j ACCEPT
-A DOCKER -d 172.17.0.1/32 ! -i docker0 -o docker0 -p udp -m udp --dport 500 -j ACCEPT
-A fail2ban-SSH -p tcp -m comment --comment “099 return from fail2ban chain” -j RETURN
COMMIT

Completed on Wed Aug 19 16:44:58 2015

SOLVED: I was looking at the destination host iptables but also had to modify the source host iptables.