Nodepool scale priority

As a DevOps of a relativly large cluster there is a point when nodes get named and sorted weired.
When upgrading nodes by just cordon, drain and delete them safely, everything works as expected.

But in case I just want to safely shutdown a node I don’t want to participate in the cluster anymore this is not really possible.

Example:

  • worker1
  • worker2
  • worker3

I want to scale down to 2 worker nodes and I want worker 2 do be deleted.
If the nodepool down scale is used it will just delete worker3.

Of course I shoudn’t care and just look at the last created node. But thats a pretty hard task if you got a

  • worker 1
  • worker 10

and rancher deletes worker 1 instead of 10 because it maximum confused about the naming.

What would be awesome is something like a nodepool downscale delete priority.
Back to the worker1 to worker3 example I could cordon and drain worker 2 and the priority could be to delete drained nodes first instead of healthy, maybe deployment running nodes.

Is this is already implemented? I couldn’t find it.

2 Likes

The sort looks to be “ASCII-betical”, so “9” > “10” and whatever’s at the end of the list is removed.
https://github.com/rancher/rancher/blob/0f65289e1bbcbadb8f71d11775679ed2d53f9b62/pkg/controllers/management/nodepool/nodepool.go#L246-L248

It would need to use the prefix to split out the number and sort those numerically to ensure that the actual highest is deleted first.

Prioritizing cordoned nodes seems reasonable, but does not currently exist.

2 Likes

Should I open a github issue for a feature request? Maybe I could also try to implement that myself.

I would also be interested in this functionality. This would be very useful for implementing with the cluster autoscaler (https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler). Currently since a node is deleted at random on scaledown we cannot control which node is deleted by the cluster-autoscaler.

1 Like