Environment
Longhorn version:1.2.4
Kubernetes version:v1.18.20
Node config
OS type and version
CentOS Linux release 7.9.2009 (Core)
CPU per node: 4
Memory per node: 16G
Disk type xfs
Network bandwidth and latency between the nodes:
Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal):
[shutang@node-u23 ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
node-u04 Ready <none> 8d v1.18.20
node-u19 Ready <none> 8d v1.18.20
node-u20 Ready <none> 8d v1.18.20
node-u21 Ready master 8d v1.18.20
node-u22 Ready master 8d v1.18.20
node-u23 Ready master 8d v1.18.20
[shutang@node-u23 ~]$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-bb5f9009-2eb9-4138-96cb-e32eb7f27327 2Gi RWO Delete Bound default/longhorn-volv-pvc longhorn 4h11m
[shutang@node-u23 ~]$ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
longhorn (default) driver.longhorn.io Delete Immediate true 4h19m
[shutang@node-u23 ~]$ kubectl get pods
NAME READY STATUS RESTARTS AGE
longhorn-iscsi-installation-6nwxh 1/1 Running 0 156m
longhorn-iscsi-installation-cnqsw 1/1 Running 0 157m
longhorn-iscsi-installation-fkt2m 1/1 Running 0 156m
longhorn-nfs-installation-6m5gs 1/1 Running 0 26h
longhorn-nfs-installation-kg4gt 1/1 Running 0 26h
longhorn-nfs-installation-lxn2m 1/1 Running 0 26h
longhorn-uninstall-f4llp 0/1 Completed 0 4h31m
volume-test 0/1 ContainerCreating 0 156m
[shutang@node-u23 ~]$ kubectl describe pod volume-test
Name: volume-test
Namespace: default
Priority: 0
Node: node-u19/10.222.175.236
Start Time: Mon, 23 May 2022 01:33:01 -0700
Labels: <none>
Annotations: Status: Pending
IP:
IPs: <none>
Containers:
volume-test:
Container ID:
Image: nginx:stable-alpine
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/data from volv (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-gkhh8 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
volv:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: longhorn-volv-pvc
ReadOnly: false
default-token-gkhh8:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-gkhh8
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 6m10s (x82 over 156m) kubelet MountVolume.MountDevice failed for volume "pvc-bb5f9009-2eb9-4138-96cb-e32eb7f27327" : rpc error: code = Internal desc = mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t xfs -o defaults /dev/longhorn/pvc-bb5f9009-2eb9-4138-96cb-e32eb7f27327 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-bb5f9009-2eb9-4138-96cb-e32eb7f27327/globalmount
Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-bb5f9009-2eb9-4138-96cb-e32eb7f27327/globalmount: wrong fs type, bad option, bad superblock on /dev/longhorn/pvc-bb5f9009-2eb9-4138-96cb-e32eb7f27327, missing codepage or helper program, or other error.
Warning FailedMount 83s (x52 over 152m) kubelet Unable to attach or mount volumes: unmounted volumes=[volv], unattached volumes=[volv default-token-gkhh8]: timed out waiting for the condition
check the pod longhorn-manager-bs99r logs
