I am trying to add a Load Balancer service that manages traffic for several applications/services/containers in the same or different Rancher hosts. However, at the moment you can only go to “Stack” => Add Service => Add Load Balancer which in turn associates the lb to the Stack so if you delete the stack you delete the Load Balancer Service.
In top of that, if you are mapping two different Stacks (apps) with that Load Balancer service you have to go to the Stack that actually display the instance. Which is definitively not the way to go.
So, I am possibly doing something wrong or else it may be a design flaw. Details:
If I want to have a Load Balancer managing two different Drupal or Wordpress sites I would create a Load Balancer and associate those two services (each one a Stack with its own Mariadb,etc.). And in the Load Balancer I would bind the port 80 and expose within the Host so I can change my DNS and point to the Load Balancer IP then reach my container world-wide. If I escalate more containers those would autoregister into the haproxy and reload it.
As a fact you cannot have multiple Load Balancers listening on the same port 80 on the same HOST and public IP, thus, you cannot serve two different Wordpress sites associating a Service -> lb on each one. Nor you can have (apparently) a global Load Balancer unless you instantiate and configure your own haproxy or nginx reverse proxy… (losing the automagical configuration and discovery that Rancher provides).
Many apps would expect them to be accessed with he port 80/443 on the URL and so on; You really want to have the reverse proxy or your lb listening on port 80/443 from one IP to several containers and unrelated stacks within the same host.
A separate issue: it would be good to be able to have your own Load Balancer somewhere outside your Rancher infrastructure but you can probably not do that because the Docker containers are network isolated so would be unreachable directly from outside the Rancher host. Am I right ? which forces you to entirely use the provided Rancher lb facility (which is a good thing, somehow).
Question
How would you create a global Service -> Load Balancer not associated directly within a stack so I can clearly see that the Load Balancer is running and decouple it from an “Stack” ?
I am not sure if I explained myself good enough so you can understand my question but let me know if you need more details.
I am not sure how you came about those conclusions… here is how the LB works:
You start a Load Balancer service in a stack - any stack really. The load balancer service has mapping that give various input and maps to different services, can be in any stack in the environment.
For example, you may have the following configuration:
@etlweather I got all that much. However, that is not my actual question and regarding what I said on my post I came to those conclusions by pure experimentation. But I clearly did not manage to fully explain myself, apparently.
Imagine you have only one host available within your Rancher setup:
Create a Wordpress A instance that does NOT expose any public port on the host.
Create a Wordpress B instance that does NOT expose any public port on the host.
Try to add a load balancer to simple access Wordpress A and Wordpress B from the outside Internet mapping port 80 to port 80 and then have 20 containers of Wordpress A for load balancing purposes.
You should notice this two issues:
Inside of your Stack (but not from outside) you must Add Service → Load Balancer in one of the two Stack (Wordpress A or Wordpress B). Thus, if you create it under the A Stack you must go there every time you want to manage the Load Balancer, and if deleted, you end up removing the three services including the Load Balancer that may be serving other containers in other stacks.
You cannot have more than one Load Balancer in the same host (and even if you could). Because of this my actual question: can you have a global Load Balancer service?
So if you only have one Load Balancer to map both Wordpress sites, you will have your Load Balancer appearing in Wordpress A or Wordpress B. But it will not a appear as a “Global” service. Unlike things like Convoy, Storage pools, etc.
You can notice all three Stack (in a single host). One of the Load Balancers will never activate, plus, as you can see they appear underneath each individual stack regardless of the fact that you could be mapping several other stacks (unrelated in the hierarchy).
Trying to have multiple Load Balancer instances listening in the same port 80 on the host is - for obvious reasons - not possible. And if you try you will get this error:
There is no option to add a Load Balancer from the main menu without having a Stack or an “orphaned” container created first.
That menu will only appear from inside a Stack but not from an Standalone container or from the “Applications” context directly. So you cannot add another haproxy unless you deploy your own container, losing all the automatic process that Rancher does.
How do you create a Standalone (“global”) Load Balancer in Rancher?
I think I get your problem now. There is I think a misunderstanding that the load balancer has to be in the same stack as the services it will load balance. Service can link to other service in other stacks. You cannot cross environment, but can definitely can cross stack - that is what I do all the time. I have my “common infrastructure” services in one stack and then a different stack of each services.
You can make a stack named “load blancer” or whatever, start your load balancer service in that stack. You will be able to load balance to any of the service in the Wordpress A and Wordpress B stacks.
A “global” service in Rancher’s term is a service that will automatically run one container instance on every host - that is probably part of why I misunderstood your original post. That’s the option at the top when you start your service - you can select a number of containers to run, or to run one container per host.
@etlweather, I knew you could do cross-stack. However, just realized now, you can create a (Standalone) Load Balancer creating a new Stack without any container or additional service on it. I did not realize that the step “import Compose” is actually optional and that I can add the Service->LB to an empty stack.
And I believe that would allow me to achieve what I want, having a global LB would be possible and like you say you can have it in each host. I still think it should be a global setting like infrastructure or storage pool but it is ok.
In the other hand (and a bit off-topic), I understood “Stacks” are for apps that have several components that are isolated via different docker services rather than “environments”. Am I right?
You would not have a “Stack” named “Wordpress” just for Wordpress sites and another named “Drupal” just for Drupal sites? What is the convention here ?
From docs: “A Rancher stack mirrors the same concept as a docker-compose project.
_It represents a group of services that make up a typical application or _ workload.”
Also, would I be able to handle hundreds of containers/services on production with a LB per host ? or the container will not be configured internally to handle “heavy” load by default ?
thanks for your comments and any additional insight on the topic is welcome
Stacks: from my experience in using Rancher now for several months on semi-production environment, stacks are just a grouping - use it which ever way fits best. Sure you can create a stack from rancher-compose/docker-compose files, but you (as you discovered) can just create a blank stack and add services one at a time. You could have 1 stack with all services you want to run, or you could have 1 stack for each “application” made up of one or more service.
It’s really up to how you want to deal with things.
When you group things in a stack, you can start-stop all the services in the stack together, delete the whole stack with one click - is that a feature? - etc.
In terms of Rancher’s internal DNS, the stack has a slight effect on the naming. Services in the same stack can be accessed by service name. Services in other stacks can only be accessed through service.stack notation.
@etlweather I see, my doubt was trying to replicate an external BSD host with haproxy within Rancher. Since you need to rely on the LB Rancher ships (for the auto-configuration and service discovery). The LB in Docker/Rancher is a single point of failure it is totall a really important part of the architecture to keep reliable.
Re: your example, a feature would be more like “upgrading” all the containers at once within the same stack indeed.
Re: LB, Having a few containers would not be a problem, but having hundreds could be very annoying to configure and manage (specially clicking through the UI). What if you remove by accident the LB Stack ? should be a way to quickly restore its configuration.
Also, thanks for the notes on DNS internals from Rancher. Good to be aware.
LB single-point of failure: yes, that was my concern too. Physical, VM or Docker, it’s always been a problem.
What I did is I have two hosts in Rancher. My LB is configured with “global” so that each host run one container of the LB service.
I also run a keepalived service - also global, one per host - which enables me to use one virtual IP that the clients access. Keepalived takes care of seeing that one of the two hosts is responding on that IP. It checks that the LB service is running also on that host, so if the LB service is stopped on the host that is the master (i.e. that is currently servicing the virtual IP), then the master switch to the other machine.
oh, so that way you can have floating IPs across different hosts. Would you mind to add more details about that ? I am not very familiar with Keepalived myself but definitively sounds like a must have and I will check the documentation (I heard about it already but never actually planned to use).
Do you install it as a Docker container ? how it works to change the IP in the internal LB that Rancher includes?
I suppose you can always do DNS round-robin to different LB on each host (different public IP) or add another IP address to the host. Alternatively.
It would be great if any additions/improvements are also sent to the github repo so we can implement and eventually have an image for all… I will add the keepalived image to docker registry later next week.
FWIW I also went down the route of having a single ‘public’ stack that contains a single load balancer listening on 80/443 and sending traffic to other stacks based on host name. It seems to work well but I agree it feels a bit odd.
I wonder if it makes sense and would be possible from the internals to have a “Stack” concept like “CMS, Databases, Core…” and then sub stacks suck as “app1 (nginx, php-fpm, mariadb, varnish), app2 (nginx, ruby)”.
So visually you could expand a Stack and see several other stacks inside keeping things properly organized. However, that is more a visual thing whereas the Stack concept plays an important role for Rancher.
Just FYI @etlweather, I merged your documentation into the repo. Thanks!
I also put the image on the public registry… In case anyone wants to use it (although I strongly advise against it unless its just testing… There is still a lot of fine tuning to go into this one…)
The docker-compose file in the Github repo is really meant to be the starting point for rancher users though…
A problem I ran into: a LB in a stack separate from the service failed to resolve the DN of the service container as resolv.conf was overriden by dhcp. Each stack with an own LB but binding the same port is prohibited by docker.
My solution: a LB of and with Aliases in one stack which each link to services in separate stacks solves this.
- etc.