I am using Longhorn in my on-prem k8s cluster … I have many applications using longhorn volumes such as mongodb , efk etc… I have set defaultClassReplicaCount = 1 as i dont need replicas of volume. I get below error for one of the volumes , not for all …
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedAttachVolume 42s (x561 over 18h) attachdetach-controller AttachVolume.Attach failed for volume "pvc-71b4a49f-6cc9-4339-b15e-90bb96784828" : rpc error: code = Aborted desc = volume pvc-71b4a49f-6cc9-4339-b15e-90bb96784828 is not ready for workloads
I dont see any out for below command
kubectl get replicas -n longhorn-system | grep pvc-71b4a49f-6cc9-4339-b15e-90bb96784828
Longhorn manager logs show below error
{"file":"/go/src/github.com/longhorn/longhorn-manager/scheduler/replica_scheduler.go:627","func":"github.com/longhorn/longhorn-manager/scheduler.(*ReplicaScheduler).CheckAndReuseFailedReplica","level":"info","msg":"Cannot find a reusable failed replicas for volume pvc-71b4a49f-6cc9-4339-b15e-90bb96784828","time":"2025-01-27T19:56:56Z"}
{"accessMode":"rwo","controller":"longhorn-volume","error":"No available disk candidates to create a new replica of size 2000683008","file":"/go/src/github.com/longhorn/longhorn-manager/controller/volume_controller.go:2305","frontend":"blockdev","func":"github.com/longhorn/longhorn-manager/controller.(*VolumeController).replenishReplicas","level":"warning","migratable":false,"msg":"Unable to create new replica pvc-71b4a49f-6cc9-4339-b15e-90bb96784828-r-472a1547","node":"worker2","owner":"worker2","state":"detached","time":"2025-01-27T19:56:56Z","volume":"pvc-71b4a49f-6cc9-4339-b15e-90bb96784828"}
{"accessMode":"rwo","controller":"longhorn-volume","file":"/go/src/github.com/longhorn/longhorn-manager/controller/volume_controller.go:1339","frontend":"blockdev","func":"github.com/longhorn/longhorn-manager/controller.(*VolumeController).ReconcileVolumeState","level":"info","migratable":false,"msg":"All replicas are failed, set engine salvageRequested to true","node":"worker2","owner":"worker2","state":"detached","time":"2025-01-27T19:56:56Z","volume":"pvc-71b4a49f-6cc9-4339-b15e-90bb96784828"}
{"accessMode":"rwo","controller":"longhorn-volume","file":"/go/src/github.com/longhorn/longhorn-manager/controller/volume_controller.go:1344","frontend":"blockdev","func":"github.com/longhorn/longhorn-manager/controller.(*VolumeController).ReconcileVolumeState","level":"info","migratable":false,"msg":"All replicas are failed, auto-salvaging volume","node":"worker2","owner":"worker2","state":"detached","time":"2025-01-27T19:56:56Z","volume":"pvc-71b4a49f-6cc9-4339-b15e-90bb96784828"}
{"accessMode":"rwo","controller":"longhorn-volume","file":"/go/src/github.com/longhorn/longhorn-manager/controller/volume_controller.go:1392","frontend":"blockdev","func":"github.com/longhorn/longhorn-manager/controller.(*VolumeController).ReconcileVolumeState","level":"warning","migratable":false,"msg":"Failed to auto salvage volume: no data exists","node":"worker2","owner":"worker2","state":"detached","time":"2025-01-27T19:56:56Z","volume":"pvc-71b4a49f-6cc9-4339-b15e-90bb96784828"}