Load Balancer in another network

Archive Rancher 1.x
1

Hello,

On Rancher 0.37, I’ve setup 5 hosts on three different networks :

  • DMZ (wich could be connected to Internet)
  • APP (where application servers live)
  • PROD (where ressources used by application servers live)

The communication between each network is routed and filtered.
We have allow the following traffic between each zone has bidirectional (in and out, just for tests) :

  • TCP/8080
  • UDP/500
  • UDP/4500

As a test, I just tried to deploy a wildly server in APP zone and access them from DMZ through a load-balancer.

I obtain systematically the follwing message when I try to access to the service from DMZ LB :

503 Service Unavailable

No server is available to handle this request.

*Note, the Wildly container is in managed network mode.

I’ve try to expose the standard wildly port (8080) to his host in 8081 and access with different LB either on 8080 or 8081, same result from DMZ, but from APP (same network zone) I success to access to the service via a LB configured to access on 8080 wildly port (I.E. the internal container port not the host redirected port).

This configuration doesn’t work from DMZ.
I suspect an issue when rancher setup LB or “managed network” on different networks.

Here is my docker-compose.yml :

LB:
  ports:
  - 80:8081
  restart: always
  labels:
    io.rancher.scheduler.affinity:host_label: zone=dmz
  tty: true
  image: rancher/load-balancer-service
  links:
  - Portal:Portal
  stdin_open: true
Portal:
  ports:
  - 8081:8080/tcp
  restart: always
  labels:
    io.rancher.scheduler.affinity:host_label: zone=app
  tty: true
  image: jboss/wildfly
  stdin_open: true
LBApp8080:
  ports:
  - 81:8080
  restart: always
  labels:
    io.rancher.scheduler.affinity:host_label: zone=app
  tty: true
  image: rancher/load-balancer-service
  links:
  - Portal:Portal
  stdin_open: true
LBDMZ8080:
  ports:
  - 81:8080
  restart: always
  labels:
    io.rancher.scheduler.affinity:host_label: zone=dmz
  tty: true
  image: rancher/load-balancer-service
  links:
  - Portal:Portal
  stdin_open: true
LBapp:
  ports:
  - 80:8081
  restart: always
  labels:
    io.rancher.scheduler.affinity:host_label: zone=app
  tty: true
  image: rancher/load-balancer-service
  links:
  - Portal:Portal
  stdin_open: true

No error message from agents logs or rancher server logs.

2

We are a little confused at your overall setup due to the complex nature, but we are trying to figure it out using your docker-compose.yml . :smile:

  1. If LBapp and Portal are on the same host, this will not work due to this open issue that hasn’t been implemented:
    https://github.com/rancher/rancher/issues/1920 Are they on the same host or different hosts in the same network?

  2. Just to confirm, you are saying LBApp8080 is able to access Portal.

  3. LBDMZ8080 should work based on the docker-compose.yml. Can you exec into the LB Agent container to see if you can ping the Portal container? Can you see if you can ping the port 8080 on the Portal container?

  4. Does LB work? If it’s not in the same host, then it should be different from #1. Once again, can you exec into LB Agent container to see if you can ping the Portal container? Can you see if you can ping the port 8081 on the Portal container?

3

Hi denise,

If LBapp and Portal are on the same host, this will not work due to this open issue that hasn’t been implemented:
Hairpin NAT · Issue #1920 · rancher/rancher · GitHub Are they on the same host or different hosts in the same network?

In the final configuration, in different networks, the LB is on DMZ network and the Portal on APP network.
But I confirm for my tests I’ve put an LB (LBapp) and the Portal in the same host, and then, it works.

Just to confirm, you are saying LBApp8080 is able to access Portal.

Yes, when LB and Portal are in the same network, it works.

LBDMZ8080 should work based on the docker-compose.yml. Can you exec into the LB Agent container to see if you can ping the Portal container? Can you see if you can ping the port 8080 on the Portal container?

I can ping the host IP where the Portal container reside but not the container IP (in my case 10.42.207.132).

Does LB work? If it’s not in the same host, then it should be different from #1. Once again, can you exec into LB Agent container to see if you can ping the Portal container? Can you see if you can ping the port 8081 on the Portal container?

If he can’t ping the container IP, he can’t “ping” the TCP port no ?

4

Okay, it sounds as if your hosts on different networks are having troubles communicating (regardless of load balancers). So let’s troubleshoot that first. :slight_smile:

If you see a host on the UI with IP (172.17.42.1) or starting with 172.17.x.x, then please double check to see if the IP is the actual IP of the host. These IPs tend to be the docker internal IP and will not allow the Rancher managed network to function. You will need to re-register your host with the correct IP.

Have you confirmed that neither container in the DMZ network can ping any container in the APP network?

Basically, exec into 1 container on each host and ping the container IP on 1 of each host and do the same for the other hosts. I noticed you have 5 hosts so we need to isolate if it’s a specific host issue or is it network traffic.

5

Hi Denise,

I think you merge this thread with Firewall rules for managed network , because I suspect this is the same root cause.