Iptables forwarding not working

Werelds · August 22, 2015, 6:23pm

So far my experience with Rancher has been great, except the load balancer. We only started using it very recently, but it has most of what I was looking for. Whether the load balancer will ultimately be all I need remains to be seen, but for this particular project it should be more than sufficient.

Anyway, the stack I’m running is fairly straight forward: http://cl.ly/image/1e120U1O3y1W (API listens on 8080, Frontend on 8081).

The balancer config looks like this (sorry, domain obscured for privacy): http://cl.ly/image/1N080k0d3M0Q

So far so good; the default target port I left empty initially, I only just changed it to see if it made a difference. The host in question is Debian 8 (Jessie), Docker 1.8. Rancher is running off the same. Versions used:

Rancher: v0.32.0
Cattle: v0.75.0
User Interface: v0.40.0
Rancher Compose: beta/latest

The problem is that port 80 isn’t reachable on any IP at all. UFW is enabled right now, with the default FORWARD policy set to ACCEPT; I tried with it disabled as well, but no joy. When I check the routing, it seems fine:

# iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
CATTLE_PREROUTING  all  --  anywhere             anywhere
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere            !loopback/8           ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
CATTLE_POSTROUTING  all  --  anywhere             anywhere
MASQUERADE  all  --  172.17.0.0/16        anywhere
MASQUERADE  udp  --  172.17.0.2           172.17.0.2           udp dpt:ipsec-nat-t
MASQUERADE  udp  --  172.17.0.2           172.17.0.2           udp dpt:isakmp

Chain CATTLE_POSTROUTING (1 references)
target     prot opt source               destination
ACCEPT     all  --  10.42.0.0/16         169.254.169.250
MASQUERADE  tcp  --  10.42.0.0/16        !10.42.0.0/16         masq ports: 1024-65535
MASQUERADE  udp  --  10.42.0.0/16        !10.42.0.0/16         masq ports: 1024-65535
MASQUERADE  all  --  10.42.0.0/16        !10.42.0.0/16
MASQUERADE  tcp  --  172.17.0.0/16        anywhere             masq ports: 1024-65535
MASQUERADE  udp  --  172.17.0.0/16        anywhere             masq ports: 1024-65535

Chain CATTLE_PREROUTING (1 references)
target     prot opt source               destination
DNAT       udp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL udp dpt:ipsec-nat-t to:10.42.119.252:4500
DNAT       udp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL udp dpt:isakmp to:10.42.119.252:500
DNAT       tcp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL tcp dpt:http to:10.42.137.131:80

Chain DOCKER (2 references)
target     prot opt source               destination
DNAT       udp  --  anywhere             anywhere             udp dpt:ipsec-nat-t to:172.17.0.2:4500
DNAT       udp  --  anywhere             anywhere             udp dpt:isakmp to:172.17.0.2:500

Port 80 is listed just fine and points to the LB’s IP. But aside from its actual IP, I can’t reach it from any of the machine’s real IP addresses.

The port isn’t in use yet:

# netstat -plan|grep LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      685/sshd
tcp        0      0 127.0.0.1:9344          0.0.0.0:*               LISTEN      23164/.cadvisor
tcp6       0      0 :::22                   :::*                    LISTEN      685/sshd

Now, our setup may be a little different from most use cases. Our hosting provider gives us the ability of having an actual private network. So both the Rancher host as well as this host have eth0 connected to the web and eth1 connected to the internal network (over eth1) - on this network, I have dnsmasq providing DHCP (192.168.50.0/24)as well as the internal hostname resolving. I don’t think this should be a problem (the actual Rancher <> Host communication works flawlessly), but I figured I’d mention it anyway. That said, eth1 is completely open on eth1 and docker0 anyway - only eth0 is blocked off.

I tried eth0, eth1, docker0, localhost, 127.0.0.1 but the traffic doesn’t get forwarded on any of the IP addresses provided by these interfaces. Tried disabling UFW as well, no effect. Checked the HAProxy config, looks fine:

frontend de8e4b9b-4309-428c-8634-2a0f4d4692f3_frontend                          
        bind 10.42.137.131:80                                                   
        mode http                                                               
        acl 0_host hdr(host) -i ---                                  
        acl 0_host hdr(host) -i ---:80                               
        acl 0_path path_beg -i /api                                             
        use_backend de8e4b9b-4309-428c-8634-2a0f4d4692f3_0_backend if 0_host 0_path                                                                             
        acl 1_host hdr(host) -i ---                                  
        acl 1_host hdr(host) -i ---:80                               
        use_backend de8e4b9b-4309-428c-8634-2a0f4d4692f3_1_backend if 1_host    
                                                                                
backend de8e4b9b-4309-428c-8634-2a0f4d4692f3_0_backend                          
        mode http                                                               
        balance roundrobin                                                      
        server 1c8436a3-500d-4217-b81a-73b3e025a288 10.42.85.65:8080            
backend de8e4b9b-4309-428c-8634-2a0f4d4692f3_1_backend                          
        mode http                                                               
        balance roundrobin                                                      
        server 51078afd-8549-4d42-a168-10c97aa3f116 10.42.112.121:8081

I’m sure I’m overlooking something really dumb, but I don’t see it right now

flaccid · August 23, 2015, 12:02am

I may have run into this before myself as well. In some cases, it was solved by restarting the docker daemon, it did however look like the NAT in iptables failing and the restart reset it so it worked again. Why and how it got screwed up for me I’m not sure but next release I’m keen to see if it ever happens with docker >= 1.8.1.

Werelds · August 23, 2015, 10:57am

Just tried that, no luck with that either.

Werelds · August 23, 2015, 2:19pm

Definitely a wrong rule in iptables somewhere, I need to do an iptables-save and go over the whole stack.

When accessing port 80 from the outside (i.e. from another server in the private network, via the VPN or just on its public IP) it does work. It just doesn’t work from the host itself, which is what was so confusing to me.

vincent · August 23, 2015, 4:08pm

What you’re trying to do is called “Hairpin NAT”. There’s nothing wrong with the existing iptables rules, but an additional rule that Rancher does not currently setup is required to support this.

Werelds · August 24, 2015, 3:02pm

See, told you my brain was fried. Completely forgot about that.

Thanks!

Topic		Replies	Views
[Fixed] Problem Publishing public LoadBalancer Port on host Rancher 1.x	2	1571	November 6, 2015
"Host IP" in load balancer configuration not working Rancher 1.x	3	2653	January 8, 2017
No access to containers without "iptables -P FORWARD ACCEPT" Rancher	1	867	September 30, 2020
LoadBalancers fail after a day or so Rancher 1.x	4	950	April 4, 2017
Rancher agent not communicating across public<->private subnets on AWS Rancher 1.x	4	1926	November 18, 2016

Iptables forwarding not working

Related topics