FailedMount - Error attaching RWX Volume

Longhorn
1

Pod fails to attach to existing RWX volume, although several other pods connected.

We have 20 namespaces, each have a RWX volume shared amongst 5 pods 19 are stable, 1 gets the below error, consistently… This is the only one where the volume is over 80GB of data. The rest are 5-10GB.

Events:

  Type     Reason       Age                    From               Message
  ----     ------       ----                   ----               -------
  Normal   Scheduled    10m                    default-scheduler  Successfully assigned technica-prod/bastion-6c5f845599-wtjrb to k8s-devworker2
  Warning  FailedMount  3m59s (x3 over 8m30s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[attachments], unattached volumes=[attachments default-token-z9fsh]: timed out waiting for the condition
  Warning  FailedMount  102s                   kubelet            Unable to attach or mount volumes: unmounted volumes=[attachments], unattached volumes=[default-token-z9fsh attachments]: timed out waiting for the condition
  Warning  FailedMount  26s (x5 over 8m33s)    kubelet            MountVolume.SetUp failed for volume "pvc-7e292083-314d-4e5a-9e8b-d64a27783918" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
2

Would you please provide the support bundle and kubelet log if possible.
And send to longhorn-support-bundle@suse.com, thx.

3

JenTing, we purchased a support bundle and have opened case 00339301, and have uploaded information there.

Thank you.

4

Hello @Andrew_Prowse ,

I also facing similar issue. Did you able to resolved this?. If so, Can you guide me on this.

Best Regards,
Mareeskannan Rajendran.

5

Unfortunately we were never able to fix reliably.

We were able to recover some volumes by mounting the longhorn volume
file locally on the worker node, and copying the data off.

We ended up moving all RWX volumes off longhorn and on to an NFS
server. No more problems after that.

6

I have volume of 5Gb with longhorn and the pod show Bound , without in sistems describe not connect to database. I kill the pod and not connect to volume and message is
Unable to attach or mount volumes: unmounted volumes=[db-storage], unattached volumes=[kube-api-access-6mn6g db-storage]: timed out waiting for the condition

and

MountVolume.MountDevice failed for volume “pv-db-storage” : rpc error: code = Internal desc = format of disk “/dev/longhorn/pv-db-storage” failed: type:(“ext4”) target:(“/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pv-db-storage/globalmount”) options:(“defaults”) errcode:(exit status 1) output:(mke2fs 1.45.5 (07-Jan-2020) /dev/longhorn/pv-db-storage is apparently in use by the system; will not make a filesystem here! )

This problem init for low level storage and I extend space in te server. in time te pod return in ready, 30 and 60 min lost.