I am having issues with the longhorn mounts not moving if a pod gets rescheduled to a new node.
Example. Right now I have a pod that was on node kube-4 and kube-4 got hit with some heavy loads and evicted the pods on it. Said pod got rescheduled to kube-3. Perfect, this is expected behavior. However, it’s been in container starting state for 20 minutes.
kubectl describe pod
Warning FailedAttachVolume <invalid> (x11 over 16m) attachdetach-controller AttachVolume.Attach failed for volume "pvc-0a6dea05-6954-41da-a7a9-6861461d4e2e" : rpc error: code = FailedPrecondition desc = The volume pvc-0a6dea05-6954-41da-a7a9-6861461d4e2e cannot be attached to the node kube-3 since it is already attached to the node kube-4
I can manually detach said pvc from kube-4 and reattach to kube-3 and it’s ok.
Is there something I am missing that is preventing the attachdetach-controller from detaching when the workload is terminated or evicted. My deployment is set for “recreate” to ensure termination of the pod before starting the new one so this is now a RWX collision.