Hi there,
I’ve been setting up a few services and I’ve noticed something I’m not 100% sure what is the conventional way to solve. I have multiple services I run via Rancher and I love utilizing the built-in load balancer, but I’m not sure how to reverse proxy my domains to the proper load balancer as if something causes a server to go down and the LB is spun up on a different server I have to repoint my reverse proxy as well. My current solution is putting all my load balancers on 1 server, but that doesn’t seem to be the appropriate way to me. Maybe it is, but I would be curious if someone has a better or different solution to pointing reverse proxies to the appropriate load balancer. Personally I use Caddy as my reverse proxy, but I’m willing to try anything if it will allow me to spread things out so one server going down doesn’t lock out access to everything.
Thanks!
To clarify my situation here it is plainly:
I have a set of servers (VMs) in a single location
They all share a single external IP address that’s accessible from the internet
When traffic comes in the router automatically sends the traffic to a single specific server (VM)
This server (VM) has Caddy installed and running and handles SSL termination and proxying
It hands it off to the load balancers in Rancher
My issue is I don’t know of any way to update what internal ip address to proxy to in the event it changes because the load balancer spins up on a different machine causing me to either go and manually update it’s record, or, as I’m currently doing, put all my load balancers on a single VM making it essentially making a single point of failure for everything.
You can read metadata to find what hosts the balancer is running on. Or use one of the external-dns providers, which essentially do that and keep a set of DNS entries up to date with the same. Or have your frontend balancer attempt to balance to all the hosts and let it manage which ones are healthy/unhealthy because they do/don’t actually have a balancer container running on them.
I would love to use an external DNS provider but I’m not sure I can do that when it’s all internal servers sharing the same external IP address can I? The goal would be to be to have the reverse proxy know when a new LB is added for a specific domain and then have it add that server to the list and remove stagnant ones. Would that be possible with an all internal network using an external DNS provider?
I can tell you the way we manage this.
We have a farm of 5 Hosts. Two of these have the label LB=1, and we schedule all of our load balancers with ‘global’ and ‘require host LB=1’. As a result, they will always run on both of these 2 Hosts.
Now, we have an F5 Load Balancer appliance. This is configured to have a wildcard TCP VIP on a routeable address that forwards to a pool containing both of these Hosts. As a result, we can connect to the VIP and always reach one of the load balancers; we can swap hosts in and out without requiring an outage.
If you do not have an F5, then you could always use an external HAProxy, or just set DNS to resolve to both of these Hosts. Of course, once you’re using DNS then you can look at Router53 or similar, provided that your addresses are routable and you can link your DNS appropriately.
That seems pretty straight forward. My main issue is the fact I have everything (currently) in a home office, so I have a single external IP address and my router is funneling ALL port 80, 443, 22, and a few other ports to my main proxy server currently (kind of like in your example). My main thought was if there was a way I could create some script or something to see when new LBs were added and get info on what application they were a part of so I could modify my Caddy proxy server’s config and restart it when something new comes up OR when something needs to change, Having them all on 2 or 3 servers is fine in my case though, but it would be nice to be able to know when something new spins up so I could have a script add it based on metadata or something like that. Maybe that’s something I can build into my docker images though…