Host-required labels are not applied properly?

1

I have an environment with 2 hosts:

a) host-rancher_internal
b) host-ci-workers

host-ci-workers has a host-required label: io.rancher.scheduler.affinity=ci_worker
image

image
image848×657 42.1 KB

while the host-rancher_internal does not have any required label.
image

Today, due to the ros 1.1.3 upgrade, i had top restart rancher. What happened is, that also stacks without the label io.rancher.scheduler.affinity=ci_worker have been scheduled to host-ci-workers.

Why is that - having no label should not meat the requirement having the label io.rancher.scheduler.affinity=ci_worker?

2

The io.rancher.scheduler.require_any label (which is what the UI is setting there) is only looked at by the external resource-based scheduler (the catalog item), do you have it deployed? And is it actually running anywhere?

You don’t have io.rancher.container.system in the list so if it was working you wouldn’t get any of system services on the host(s), which is unlikely to be desirable.

You should not call your labels io.rancher.*. Make up your own prefix or don’t use one at all… But they are namespaced specifically to avoid conflict with arbitrary user entries and by using our prefix you’re asking for trouble. Any future release may decide to take arbitrary actions based on labels in our namespace. And io.rancher.scheduler.affinity specifically is as you know already part of the container scheduling labels for services.

3

@vincent the scheduler is deployed in the default templater, right:
image

I did remove io.rancher.container.system since rancher was deploying my lb on the host, completly taking that part of the infra down, since the IP changes…

Maybe i got confused, but how would i mark a host using a host label that way, that only catalog items are deployed, which have a specific label? I am fine with not taking the io.rancher namespace of course, so what i would use in my case would be

de.kontextwork.concourse.affinity=workers

I would apply this as a required label on that one host, right? This would also make this host not receive any other services which do not have this label? Because that is what happened when i upgraded to rancheros 1.1.3 - due to the restart, all the services have been rescheduled and actually scheduled on the host with that label - instead of staying on the other host without a required label.

Where did i get confused?

4

Just got the same problem here.
Rancher version is 1.6.13.

Historically we had three rancher hosts. Today I added a fourth one with a required label (datacenter=test).
It also have the label “io.rancher.system” as a required label (and by the way, during tests I also did not put that label on it but have seen all infrasructure containers gets scheduled on it without problems…).

Then, I had to reboot one of the 3 host (which not have any required labels), and surprisingly I found out that some services which were on this host have been scheduled on the new one, even if they don’t have the required label (datacenter=test) on them, and they are not system (or infrastructure wide) services.

On the other hand, they have one paticular scheduling policy : we ask that there is 3 instances running (scale), and we also ask that an instance is not located on a host which already have a similar service :
io.rancher.scheduler.affinity:container_label_ne: io.rancher.stack_service.name=blabla/blabla

Don’t know if that might cause the bypassing of the required label rule on the fourth host.

Another thing that might have helped this problem to happen, is that the scheduling service was running on the node I had to reboot, and it then get rescheduled on the fourth node to.

I don’t know if the rescheduling of other services happened during the time the scheduling service was down (is that possible ?).

Anyway, that’s sound very bad to me, but perhaps I am missing something very simple.
Or perhaps it’s a bug ?

I’ll try to do some more tests and will report on this.

@EugenMayer for the load-balancer being scheduled on the required label hosts even without the label, the solution is to add a scheduling rule on the load-balancer service to ask it to NOT be scheduled on a host with a specific label (and of course, to add that label on all hosts that you don’t want your services to go to) :wink:

5

Ok, so I ran some more tests.

I created a new service, with a schedule rule to ask them to not be scheduled on a host where there is already an instance of this service.
I scaled to 3 containers, they all went on the first 3 nodes (those without any required labels rules).
I then scaled to 4, and got an error that it cannot find an host to run the new instance : it works as expected.
Then I temporarily stopped the scheduler service, and tried to scale to 4 my test service again : the container is scheduled on the fourth node which defined required labels.

So that explains what happened to me yesterday : during the time that the scheduler was rescheduled on a host, some other instances have been scheduled on my fourth host, even if they don’t have the required labels.

For me it’s a big disappointment, because that means that in some special cases (losing the host running the instance of the scheduler), the “required labels” policy is not enforced, which could have bad consequences depending how we were relying on it.

Perhaps I am missing something, and it is possible to have the scheduler in some sort of high-availability setup (didn’t find docs about that), but if not, it makes the “required label” feature useless, because it could not be trusted at all time.

6

This issue seems to be known : https://github.com/rancher/rancher/issues/7951
It was rescheduled for an 1.7 milestone, but with 2.0 coming, I doubt it will ever get fixed :frowning: