Volume is stuck in creating state after server reboot

Longhorn
1

image
image1893×994 81.7 KB

Several volumes are stuck in Creating State(Health: Faulted, Ready for workload: Not Ready).
It is after server reboot. and I suspect wrong longhorn engine caused the problem. I was using longhorn-engine:master-head somehow, and after reboot, the engine became incompatible as the new image for the head downloaded I guess.
After that I tried some to make it back but now it is.

engines.longhorn.io CR shows the pvc is stopped state with no node/instance manager/engine image bound.

And Pod shows event the pvc is not ready for workloads.

What can I do to make it back fine?

2

It’s highly unrecommended to use master-head images. They may be broken images, or incompatible ones.
To rescue the data, I prefer to:

  1. Figure out the failed replica directory and see if there is data inside
  2. Stop the workload pod, which makes the volume detached
  3. Remove the invalid replicas then do the salvage for this volume if possible
  4. If salvage does not work, plz follow this doc to export the data to a new volume
  5. Re-launch the workload pod based on the recovered volume.