HA with scheduling

Hello There,

Though there is no option for hot/live migrations for docker. I would like to use scheduling as an option for migrating containers.

Example:

I label all my hosts with “status=healthy” and spin up container on a condition with must option setting to scheduling label “status=healthy”. Do containers migrate to healthy instance? If the label of host is changed to “status=unhealthy”

Also I’m using flocker for managing volume so there is no worries about volumes.

Swaroop Kundeti
Site Reliability Engineer • Making Happiness Possible
Voice: +91-9866221131, Skype : swaroop.kundeti
Follow Axelerant on F T L G G+

If the host label is removed, the containers will not get re-scheduled. There is a Github issue that would allow the removal of containers based on label scheduling.

https://github.com/rancher/rancher/issues/2678

Will there be any scheduling based on system resources/best host for the job?

What about for geolocation? (labels could be one thing but it wouldnt provide for fallback)…

thanks

There is a Github request to schedule based on resources.

As for fallback, I think you this request might help.

Further on this topic, is there any plans for container “re-scheduling”? i.e. if I have 1 host and start services which have soft anti-affinity rules with themselves at a scale =2, Host1 gets 2 copies of the service… This is correct as the anti-affinity rules are soft…

However, if I add a host, the services dont ever change to the other host… 2 things come to mind, simple “start then stop” from one host to another, all the way to “live migration” of continaers, which I understand is already a feature of docker but still hasnt seen its way into rancher…

Any idea on these two fronts? I guess some containers would be good to have “start-then-stop” type mechanisms, while others may be crucial to maintain the container running… This could maybe be defined somewhere… anyways, I’ve noticed that if I restart the container it wont move it, so I gather its just not taken into considueration at all currently…?

@RVN_BR what you are referring to is exactly what I have also mentioned in the issue on github about rebalancing

so you could +1 that one :slightly_smiling:

There is currently no way with rancher scheduling to guarantee that a Stack is on at least two hosts. So that one can easily do docker engine upgrades/hosts replacements. Rancher will just reschedule containers if they are found down/unhealthy but no guarantee they will end up spread evenly with soft anti-affinity (and hard-affinity will only put one on each host). I hope Rancher will address this in the future, because right now we have to do silly tricks like “splitting services” or creating two identical Stacks and make sure one Stack is not deployed on the same hosts as the other Stack.

this issue was also very much related but got closed in favor of another more complex one.

I’m going to check and +1 your github issue, but if a soft anti-affinity rule were respected, it would save you from having to do other kinds of “workarounds” such as splitting services, etc, no?

unfortunately not, in my tests soft-anti affinity only works when you just start with fresh hosts (empty hosts) or hosts with equal number of containers on them. As soon as you have hosts with a different numbers of containers, you can end up a stack in a single host. I will try to post an exact compose file to show how to reproduce it.
you can read the entire post, i was quite detailed in there

Hi @demarant I agree with you, and I saw the same effects where for some reason it starts scheduling containers on a single host independent of affinity rules… I posted here somewhere… I will try to find the link soon as I’m mobile right now…

Eitherway what you mention is correct, and my comment was basically that the anti-affinity and other scheduling rules should be reviewed, not that they are working :wink: Re-scheduling should also be part of that… But I’m agreeing with you :slight_smile:

1 Like

Technically, soft affinity rules are working as at the time that you are scheduling your service, it’s honoring your soft affinity rules. Unless you have an exact example that shows me what about soft affinity rules are broken. Also, please remember that it’s soft rules, so technically anything soft could be ignored.

@RVN_BR I can’t remember if your issue is the one that I’m looking at that has soft affinity rules and sidekicks involved. But if so, that’s a more complicated case as sidekicks and links have dependencies on each other of who starts first. I still believe that soft affinity rules are working.

Hi @denise, I stopped using sidekicks as I was having more issues and ended up just sticking everything in the containers manually :frowning: Ideally I’d like to go back to that…

I reported somewhere here about an issue with using variables for stack/service in the affinity rules… I’ll look for that post… (EDIT: here it is What is the service hash / is it important & some other rancher-compose questions? )