Scaling of services across hosts is painful

Rancher 1.x
1

I really like the rancher feature “run this container on every container” but it doesnt work with upgrading services or with using docker-compose files. When I create a stack using docker-compose the edit feature only lets me select scale 1+, there is no options for “one on every host”. Looking at the rancher compose files for services with such scaling, there is no scaling option in the rancher compose at all. So I tried creating services with a rancher-compose, and no scaling key, but that didnt work.

I have tried to add scheduling rules for the service so that they go across hosts, but this doesnt work well with sidekicks. I have to add a rule that they “should not run on host with service {my service name}” but this becomes hard to maintain with upgrades as the service name is changing every upgrade.

My main goal is to have high availability for my containers, so I would be happy with any of the following options.

  1. Containers chose to run on other hosts before stacking up on a single host. In other words, a single service should always span multiple hosts (if allowed by the scheduling rules) BEFORE putting multiple containers of the same service on the same host. Right now, everything tries to go to the initial host, creating a lot of imbalance.
  2. I could use “this service” in the scheduling rules. Then I could do a scheduling rule, “should not run on host with this service” and its easier to maintain then having the “hard-coded” service name.
  3. Allow me to switch to “one container per host” after creating a service.

Also as a slightly different issue, sidekicks should run on the same host as the main. So if I have 3 containers for my main service spread across 3 hosts, all three sidekicks are ran on host 1, there should be a 1-1 mapping per hosts. I cant change scheduling rules because a sidekicks inherits scaling rules from the main service.

Any ideas of how I can have HA services across hosts?

2

one on every host is equivalent to having the service being configured as global:
http://docs.rancher.com/rancher/rancher-compose/scheduling/#global-service

I would set the following label on both the service and the sidekicks:
io.rancher.scheduler.global: ‘true’

3

Awesome. Thanks, I will try that.

4

OK, the global service thing seemed to work except:

rancher-compose --upgrade 27852 15915 --update-links -c --interval 30000 --batch-size=1
time=“2015-10-30T19:14:06Z” level=fatal msg=“Upgrade is not supported for global services”

Which means I can not upgrade the service in a rolling upgrade way.

5

In-service upgrade works on Global services. [Rolling] Upgrade starts up new containers before stopping old, and on a global service every host has the public ports for the service in-use so there is nowhere to schedule an additional container.

Sidekicks always get scheduled onto the same hosts and at the same scale as the primary service. If you’re seeing otherwise you likely don’t have the labels right and they are being treated as independent standalone services.