Load balancer, no host IP

Rancher 1.x
1

Hi.

I have a two hosts rancher test environment: one server is running rancher server and the other the agent and my containers.

Server is running on 192.168.88.190 and agent in 192.168.88.200.

I’m running:

Rancher v0.28.0
Cattle v0.64.0
User Interface v0.34.0
Rancher Compose beta/latest

When I create a load balancer (ie the Wordpress “Multi-Container Application” described in the docs), I can’t access the application using the “public” host IP address. My test host IP address is 192.168.88.200.

I can only access the app using the lb agent’s IP address: 10.42.13.49:8083 (not 192.168.88.200:8083).

In fact, clicking the load balancer’s link, it shows:

Info

Host IP
Container IP 10.42.13.49
Image rancher/agent-instance:v0.3.1
Entrypoint None
Command None

I guess it should show “Host IP 192.168.88.200”.

When I created the load balancer, I specified name (wordpresslb), scale, (1), public port (8083) and, in targets, I added one service with target service (mywordpress) and target port (80).

What am I doing wrong? I like to view the wordpress app in http://192.168.88.200:8083 (host’s IP address).

2

It’s strange that, in fact, the load balancer container shows the following:

Ports
State IP Address Public (on Host) Private (in Container) Protocol
ACTIVE 192.168.88.200 8083 8083 TCP

But 8083 port doesn’t exist in the real host:

$ netstat -lnp

(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:9344 0.0.0.0:* LISTEN -
udp 0 0 0.0.0.0:5353 0.0.0.0:* 1573/chromium
udp6 0 0 :::4500 :::* -
udp6 0 0 :::500 :::* -
raw6 0 0 :::58 :::* 7 -
…
[snip]

3

All containers are missing the “host IP” in the details screen of v0.28 due to a UI bug; this does not indicate an actual problem.

On the host the traffic is directed to the container by iptables… iptables -L -t nat (I’m on my phone but something like that :smile:)

4

I understand.

Then, I must manually add the necessary iptables rules on the host to forward traffic to the balancer’s container. I thought rancher itself would do that.

Thanks a lot.

5

It does (well, should) do that… I just meant to look at iptables on the host to see if the rules for your port exist, not netstat.

6

@marceloaguero Rancher is supposed to publish the port to the host for the LB listening ports. If it didn’t, its a bug. Working it around by adding the iptables rules manually won’t be sufficient as Rancher will overwrite them on the next port publish for any containers in the network.

7

Yes, alena & vincent. You’re right.

I’m trying rancher at my work, is addition to my home. Here (at work) it works as it’s supposed to.

I will review what I did at home. Perhaps I’ll upgrade to the latest version and tell you.

Thanks again.

8

I am having the same issue as the OP. Here is a printout of my iptables. Any suggestions would be great.

[root@echo-docker02 ~]# iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
CATTLE_PREROUTING  all  --  anywhere             anywhere            
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere            !loopback/8           ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
CATTLE_POSTROUTING  all  --  anywhere             anywhere            
MASQUERADE  all  --  172.17.0.0/16        anywhere            
MASQUERADE  udp  --  172.17.0.5           172.17.0.5           udp dpt:ipsec-nat-t
MASQUERADE  udp  --  172.17.0.5           172.17.0.5           udp dpt:isakmp

Chain CATTLE_POSTROUTING (1 references)
target     prot opt source               destination         
ACCEPT     all  --  10.42.0.0/16         169.254.169.250     
MASQUERADE  tcp  --  10.42.0.0/16        !10.42.0.0/16         masq ports: 1024-65535
MASQUERADE  udp  --  10.42.0.0/16        !10.42.0.0/16         masq ports: 1024-65535
MASQUERADE  all  --  10.42.0.0/16        !10.42.0.0/16        
MASQUERADE  tcp  --  172.17.0.0/16        anywhere             masq ports: 1024-65535
MASQUERADE  udp  --  172.17.0.0/16        anywhere             masq ports: 1024-65535

Chain CATTLE_PREROUTING (1 references)
target     prot opt source               destination         
DNAT       udp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL udp dpt:ipsec-nat-t to:10.42.192.152:4500
DNAT       udp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL udp dpt:isakmp to:10.42.192.152:500
DNAT       tcp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL tcp dpt:http to:10.42.248.137:80

Chain DOCKER (2 references)
target     prot opt source               destination         
DNAT       udp  --  anywhere             anywhere             udp dpt:ipsec-nat-t to:172.17.0.5:4500
DNAT       udp  --  anywhere             anywhere             udp dpt:isakmp to:172.17.0.5:500
9

This is a PHP issue as I am getting pages through html. Disregard.

10

I’m facing similar issue. I’m trying to loadbalance redis. When I try to ping the loadbalancer directory it works but, when i try to hit it through host it fails.
[root@perfdell15 /]# /var/lib/docker/devicemapper/mnt/84e4cfe177dcd0b9299b0991e00c4c28e4de6fcf00a8852b1c7cfd9d357be375/rootfs/usr/local/bin/redis-cli -h 10.42.188.171 -p 16379
10.42.188.171:16379> PING
PONG

[root@perfdell15 /]#
/var/lib/docker/devicemapper/mnt/84691672116cb8ba90ced799b3fe3d3040103b7e0d5d3d15cde4224f0f0a475f/rootfs/usr/local/bin/redis-cli -h 127.0.0.1 -p 16379
Could not connect to Redis at 127.0.0.1:16379: Connection refused
not connected> exit
[root@perfdell15 /]# /var/lib/docker/devicemapper/mnt/84691672116cb8ba90ced799b3fe3d3040103b7e0d5d3d15cde4224f0f0a475f/rootfs/usr/local/bin/redis-cli -h perfdell15 -p 16379
Could not connect to Redis at perfdell15:16379: Connection refused

My IPtables grep :
DNAT tcp – anywhere anywhere ADDRTYPE match dst-type LOCAL tcp dpt:16379 to:10.42.188.171:16379

Am I missing anything?

11

@kirannk It sounds like you are hitting this issue.

12

Yes, when I tried to ping the redis from an external client it works fine. For now, is there any workaround we can use?