Load balancer Target services with unhealthy backend

Hello,

I created a load balancer service and I’m taking advantage of the Targets so that I can route requests to different stack services depending on the hostname match.

When I look at the lb haproxy config it generates for each stacks backend and correctly routes traffic.

However if I stop the backend service for one stack, that backend is removed from the lb haproxy config and the traffic for that stopped stack service (which I expect to return a 502 HTTP error), gets redirected to the other healthy stack backend.

Is there any way to change this default behaviour ?

Otherwise I’ll have to put an nginx container in between the lb and the app container to force a 503.

Thanks,

Can you share an example of your docker-compose.yml?

For load balancer routing, there is a priority order of how it looks up for services.

Hostname and URL
Hostname only
URL
Default (no hostname, no URL)

If you want the traffic to continue to be re-directed to the stopped service, you might need to make a dummy target so that it would have forced the traffic to go there.

This issue might help describe what I’m trying to explain.

Thanks for your response denise.

I see so it’s a limitation of the lb which cannot support multiple front-ends and route traffic to service based on the hostname. Afraid I cannot use the bogus example as it’s the same port number used on the multiple stack services.

I noticed that unhealthy backends are removed only if I restart the LB service.

Below is my docker-compose, I’d be very keen to hear of any suggestions how I could get this architecture tweaked. The idea is I’d like to have one load balancer running on port 80/443 and any stack ID sub-domain points to that LB records, then layer 7 will route to the appropriate stack service depending on the hostname header, if that makes sense.

I managed to implement that using nginx containers in each stack because nginx is always up (the app service may crash), the health check for lb will always see nginx up and keep it as a backend.

Cheers,

Lb:
  ports:
  - 443:9999
  - 80:9999
  external_links:
  - Stack1/app:app
  - Stack2/app:app
  labels:
    io.rancher.loadbalancer.ssl.ports: '443'
    io.rancher.loadbalancer.target.Stack2/app: stack2.domain.com:443=9999
    io.rancher.loadbalancer.target.Stack1/app: stack1.domain.com:443=9999
  tty: true
  image: rancher/load-balancer-service
  stdin_open: true

This seems like a bug if there’s a matching host/path rule and no healthy targets… @alena ?

@vincent it is a bug; we should only include backends having >=1 healthy targets, to haproxy config

Should I raise a bug ticket ?

@alysum sure, thank you!

@alysum link please, I wouldn’t mind watching that issue :slight_smile:

Issue raised: https://github.com/rancher/rancher/issues/3958