Hi,
I am running some pods with strict-local onereplica volumes because they take part in a cluster of three pods. Redundancy is provided by the pod cluster.
The problem is that I can’t drain the nodes because longhorn is stucked with this strict-local onereplica volumes. I get an error message for the volume: hard affinity cannot be satisfied. The solution is to delete the instance-manager pod of the node and so the volume is moved to another device and the linked pod is started and the original node is finally drained.
Do you know why this happens? is it a bug or expected behaviour due to strict-local onereplica volumes?
To add more information. When I try to drain the node manually I get the following errors:
kubectl drain k8s-servicios-pool1-ff8c1cd5-f2dk2 --ignore-daemonsets
node/k8s-servicios-pool1-ff8c1cd5-f2dk2 already cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/cilium-envoy-htsk6, kube-system/cilium-hpwnb, longhorn-system/engine-image-ei-b0369a5d-8klh9, longhorn-system/longhorn-csi-plugin-t7qrt, longhorn-system/longhorn-manager-sg5n4
evicting pod longhorn-system/instance-manager-b0b6e7294447432b4daeb28703428999
error when evicting pods/"instance-manager-b0b6e7294447432b4daeb28703428999" -n "longhorn-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod longhorn-system/instance-manager-b0b6e7294447432b4daeb28703428999
error when evicting pods/"instance-manager-b0b6e7294447432b4daeb28703428999" -n "longhorn-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod longhorn-system/instance-manager-b0b6e7294447432b4daeb28703428999
error when evicting pods/"instance-manager-b0b6e7294447432b4daeb28703428999" -n "longhorn-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod longhorn-system/instance-manager-b0b6e7294447432b4daeb28703428999
error when evicting pods/"instance-manager-b0b6e7294447432b4daeb28703428999" -n "longhorn-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod longhorn-system/instance-manager-b0b6e7294447432b4daeb28703428999
error when evicting pods/"instance-manager-b0b6e7294447432b4daeb28703428999" -n "longhorn-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I have to delete the pod manually to be able to drain:
kubectl -n longhorn-system delete pod instance-manager-b0b6e7294447432b4daeb28703428999
pod "instance-manager-b0b6e7294447432b4daeb28703428999" deleted
The issue is with this onereplica and strict-local volumes…