So far my experience with Rancher has been great, except the load balancer. We only started using it very recently, but it has most of what I was looking for. Whether the load balancer will ultimately be all I need remains to be seen, but for this particular project it should be more than sufficient.
Anyway, the stack I’m running is fairly straight forward: http://cl.ly/image/1e120U1O3y1W (API listens on 8080, Frontend on 8081).
The balancer config looks like this (sorry, domain obscured for privacy): http://cl.ly/image/1N080k0d3M0Q
So far so good; the default target port I left empty initially, I only just changed it to see if it made a difference. The host in question is Debian 8 (Jessie), Docker 1.8. Rancher is running off the same. Versions used:
- Rancher: v0.32.0
- Cattle: v0.75.0
- User Interface: v0.40.0
- Rancher Compose: beta/latest
The problem is that port 80 isn’t reachable on any IP at all. UFW is enabled right now, with the default FORWARD policy set to ACCEPT; I tried with it disabled as well, but no joy. When I check the routing, it seems fine:
# iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
CATTLE_PREROUTING all -- anywhere anywhere
DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere !loopback/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
CATTLE_POSTROUTING all -- anywhere anywhere
MASQUERADE all -- 172.17.0.0/16 anywhere
MASQUERADE udp -- 172.17.0.2 172.17.0.2 udp dpt:ipsec-nat-t
MASQUERADE udp -- 172.17.0.2 172.17.0.2 udp dpt:isakmp
Chain CATTLE_POSTROUTING (1 references)
target prot opt source destination
ACCEPT all -- 10.42.0.0/16 169.254.169.250
MASQUERADE tcp -- 10.42.0.0/16 !10.42.0.0/16 masq ports: 1024-65535
MASQUERADE udp -- 10.42.0.0/16 !10.42.0.0/16 masq ports: 1024-65535
MASQUERADE all -- 10.42.0.0/16 !10.42.0.0/16
MASQUERADE tcp -- 172.17.0.0/16 anywhere masq ports: 1024-65535
MASQUERADE udp -- 172.17.0.0/16 anywhere masq ports: 1024-65535
Chain CATTLE_PREROUTING (1 references)
target prot opt source destination
DNAT udp -- anywhere anywhere ADDRTYPE match dst-type LOCAL udp dpt:ipsec-nat-t to:10.42.119.252:4500
DNAT udp -- anywhere anywhere ADDRTYPE match dst-type LOCAL udp dpt:isakmp to:10.42.119.252:500
DNAT tcp -- anywhere anywhere ADDRTYPE match dst-type LOCAL tcp dpt:http to:10.42.137.131:80
Chain DOCKER (2 references)
target prot opt source destination
DNAT udp -- anywhere anywhere udp dpt:ipsec-nat-t to:172.17.0.2:4500
DNAT udp -- anywhere anywhere udp dpt:isakmp to:172.17.0.2:500
Port 80 is listed just fine and points to the LB’s IP. But aside from its actual IP, I can’t reach it from any of the machine’s real IP addresses.
The port isn’t in use yet:
# netstat -plan|grep LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 685/sshd
tcp 0 0 127.0.0.1:9344 0.0.0.0:* LISTEN 23164/.cadvisor
tcp6 0 0 :::22 :::* LISTEN 685/sshd
Now, our setup may be a little different from most use cases. Our hosting provider gives us the ability of having an actual private network. So both the Rancher host as well as this host have eth0
connected to the web and eth1
connected to the internal network (over eth1
) - on this network, I have dnsmasq providing DHCP (192.168.50.0/24
)as well as the internal hostname resolving. I don’t think this should be a problem (the actual Rancher <> Host communication works flawlessly), but I figured I’d mention it anyway. That said, eth1
is completely open on eth1
and docker0
anyway - only eth0
is blocked off.
I tried eth0
, eth1
, docker0
, localhost
, 127.0.0.1
but the traffic doesn’t get forwarded on any of the IP addresses provided by these interfaces. Tried disabling UFW as well, no effect. Checked the HAProxy config, looks fine:
frontend de8e4b9b-4309-428c-8634-2a0f4d4692f3_frontend
bind 10.42.137.131:80
mode http
acl 0_host hdr(host) -i ---
acl 0_host hdr(host) -i ---:80
acl 0_path path_beg -i /api
use_backend de8e4b9b-4309-428c-8634-2a0f4d4692f3_0_backend if 0_host 0_path
acl 1_host hdr(host) -i ---
acl 1_host hdr(host) -i ---:80
use_backend de8e4b9b-4309-428c-8634-2a0f4d4692f3_1_backend if 1_host
backend de8e4b9b-4309-428c-8634-2a0f4d4692f3_0_backend
mode http
balance roundrobin
server 1c8436a3-500d-4217-b81a-73b3e025a288 10.42.85.65:8080
backend de8e4b9b-4309-428c-8634-2a0f4d4692f3_1_backend
mode http
balance roundrobin
server 51078afd-8549-4d42-a168-10c97aa3f116 10.42.112.121:8081
I’m sure I’m overlooking something really dumb, but I don’t see it right now