We are using to Longhorn with kubernetes to deliver of persistent volumes. Sometimes when we making some pod updates (change some configuration for example) we have problems. The pod doesn’t start correctly due to PVC state. The persistent volume claim stays with terminating state forever. While this pvc have been terminating, the pod doesn’t start. If I force to delete PVC, through the kubectl, pod starts and mount the persistent volume again, at the same time as we will have data loss.
Someone have this problem too? And how we can fix them without data lost?
Can you file an issue with reproduce steps? What update you’ve done? If the deployment is not using recreate policy, Kubernetes will attempt to start another pod (probably on another node) with the same PVC, which can cause problem since Longhorn volume cannot be mounted on two different nodes.
We have made a pod image update, our PV is using a delete reclaim policy. We’re using this commando: "kubectl set image deployment/jenkins jenkins=my.registry.com/jenkins:2.138_v2" to update our pod image.
I noticed that sometimes if I run this command the image update works and others it doesn’t work. In my opinion, the problem is on Kubernetes and not in Longhorn. Something strange is occurring during this process (image updates or pods recreate) that put PV on delete status forever.
You can try reproduce this problem just making some images update or recreate pods.