Hello,
after upgrading from 2.4.3 to 2.5.1 in a docker setup, Rancher tries to execute helm operations in the local cluster but the operations fail and the log is filled with the following error messages:
E1023 12:20:47.711785 22 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create containerd task: failed to mount rootfs component &{overlay overlay [workdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1015813/work upperdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1015813/fs lowerdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/32049/fs]}: no such device: unknown
E1023 12:20:47.711985 22 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "helm-operation-54wrd_cattle-system(e08ed2df-aeef-4056-8e77-f0f507a985e5)" failed: rpc error: code = Unknown desc = failed to create containerd task: failed to mount rootfs component &{overlay overlay [workdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1015813/work upperdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1015813/fs lowerdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/32049/fs]}: no such device: unknown
E1023 12:20:47.712152 22 kuberuntime_manager.go:727] createPodSandbox for pod "helm-operation-54wrd_cattle-system(e08ed2df-aeef-4056-8e77-f0f507a985e5)" failed: rpc error: code = Unknown desc = failed to create containerd task: failed to mount rootfs component &{overlay overlay [workdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1015813/work upperdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1015813/fs lowerdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/32049/fs]}: no such device: unknown
E1023 12:20:47.712333 22 pod_workers.go:191] Error syncing pod e08ed2df-aeef-4056-8e77-f0f507a985e5 ("helm-operation-54wrd_cattle-system(e08ed2df-aeef-4056-8e77-f0f507a985e5)"), skipping: failed to "CreatePodSandbox" for "helm-operation-54wrd_cattle-system(e08ed2df-aeef-4056-8e77-f0f507a985e5)" with CreatePodSandboxError: "CreatePodSandbox for pod \"helm-operation-54wrd_cattle-system(e08ed2df-aeef-4056-8e77-f0f507a985e5)\" failed: rpc error: code = Unknown desc = failed to create containerd task: failed to mount rootfs component &{overlay overlay [workdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1015813/work upperdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1015813/fs lowerdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/32049/fs]}: no such device: unknown"
The directory /var/lib/rancher is mounted from a directory on ext4 on the host.
Any ideas on how I can further debug this issue?
Thanks,
a13r