Overlayfs: filesystem on ... not supported as upperdir

Rancher
1

Hello,

I’m running rancher 2.5.8 on docker 20.10.2 on Ubuntu LTS 20.04.2.

Logs and the console are flooded with:

overlayfs: filesystem on '/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/12756778/fs' not supported as upperdir

‘docker info’ shows I’m running overlay2 with a backing filesystem of extfs and Supports d_type: true.

An online search isn’t returning much of anything, is this an issue with the way the docker image is configured?

Grateful for any help…

Randy in Seattle

2

Please share the output from docker info and the exact command you used to run Rancher.

3 
root@rancher:~# docker info
Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 20.10.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version:
 runc version:
 init version:
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-77-generic
 Operating System: Ubuntu 20.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.6GiB
 Name: rancher
 ID: RZY3:UOZP:4CII:GFT5:AEVZ:Y4KG:SFIW:ZNR6:T657:WXFK:NG2C:FNRY
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No blkio weight support
WARNING: No blkio weight_device support
root@rancher:~#

I’m configuring the container options form chef, here’s the resource definition (I’ll see if I can fish out the actual docker command too):

docker_container 'rancher' do
  repo 'rancher/rancher'
  tag 'stable'
  port '443:443'
  volumes [ '/rancher:/var/lib/rancher' ]
  restart_policy 'unless-stopped'
  privileged true
end

The “volumes” line is so I can keep the config information within the rancher instance on a persistent NFS mount.

This might be TMI, tried “runlike” to get the actual command:

root@rancher:/var/chef/cache/cookbooks/sc_rancher/recipes# runlike rancher
docker run --name=rancher --hostname=469dac5eb154 --mac-address=02:42:ac:11:00:02 --env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin --env=CATTLE_SYSTEM_CHART_DEFAULT_BRANCH=release-v2.5 --env=CATTLE_CHART_DEFAULT_BRANCH=release-v2.5 --env=CATTLE_PARTNER_CHART_DEFAULT_BRANCH=main --env=CATTLE_HELM_VERSION=v2.16.8-rancher1 --env=CATTLE_K3S_VERSION=v1.19.8+k3s1 --env=CATTLE_MACHINE_VERSION=v0.15.0-rancher55 --env=CATTLE_ETCD_VERSION=v3.4.3 --env=CATTLE_CHANNELSERVER_VERSION=v0.5.0 --env=LOGLEVEL_VERSION=v0.1.3 --env=TINI_VERSION=v0.18.0 --env=TELEMETRY_VERSION=v0.5.14 --env=KUBECTL_VERSION=v1.19.7 --env=DOCKER_MACHINE_LINODE_VERSION=v0.1.8 --env=LINODE_UI_DRIVER_VERSION=v0.3.0 --env=RANCHER_METADATA_BRANCH=release-v2.5 --env=HELM_VERSION=v3.3.1 --env=KUSTOMIZE_VERSION=v3.5.4 --env=CATTLE_FLEET_MIN_VERSION=0.3.500 --env=CATTLE_RANCHER_OPERATOR_MIN_VERSION=0.1.400 --env=CATTLE_RANCHER_WEBHOOK_MIN_VERSION=0.1.000 --env=TINI_URL_amd64=https://github.com/krallin/tini/releases/download/v0.18.0/tini --env=TINI_URL_arm64=https://github.com/krallin/tini/releases/download/v0.18.0/tini-arm64 --env=TINI_URL=TINI_URL_amd64 --env=HELM_URL_V2_amd64=https://github.com/rancher/helm/releases/download/v2.16.8-rancher1/rancher-helm --env=HELM_URL_V2_arm64=https://github.com/rancher/helm/releases/download/v2.16.8-rancher1/rancher-helm-arm64 --env=HELM_URL_V2=HELM_URL_V2_amd64 --env=HELM_URL_V3=https://get.helm.sh/helm-v3.3.1-linux-amd64.tar.gz --env=TILLER_URL_amd64=https://github.com/rancher/helm/releases/download/v2.16.8-rancher1/rancher-tiller --env=TILLER_URL_arm64=https://github.com/rancher/helm/releases/download/v2.16.8-rancher1/rancher-tiller-arm64 --env=TILLER_URL=TILLER_URL_amd64 --env=K3S_URL_amd64=https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/k3s --env=K3S_URL_arm64=https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/k3s-arm64 --env=K3S_URL=K3S_URL_amd64 --env=CHANNELSERVER_URL_amd64=https://github.com/rancher/channelserver/releases/download/v0.5.0/channelserver-amd64 --env=CHANNELSERVER_URL_arm64=https://github.com/rancher/channelserver/releases/download/v0.5.0/channelserver-arm64 --env=CHANNELSERVER_URL=CHANNELSERVER_URL_amd64 --env=ETCD_URL_amd64=https://github.com/etcd-io/etcd/releases/download/v3.4.3/etcd-v3.4.3-linux-amd64.tar.gz --env=ETCD_URL_arm64=https://github.com/etcd-io/etcd/releases/download/v3.4.3/etcd-v3.4.3-linux-arm64.tar.gz --env=ETCD_URL=ETCD_URL_amd64 --env=KUSTOMIZE_URL_amd64=https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/v3.5.4/kustomize_v3.5.4_linux_amd64.tar.gz --env=KUSTOMIZE_URL_arm64=https://github.com/brendarearden/kustomize/releases/download/kustomize/v3.5.4/kustomize_v3.5.4_linux_arm64.tar.gz --env=KUSTOMIZE_URL=KUSTOMIZE_URL_amd64 --env=CATTLE_UI_VERSION=2.5.8 --env=CATTLE_DASHBOARD_UI_VERSION=v2.5.8 --env=CATTLE_CLI_VERSION=v2.4.11 --env=CATTLE_API_UI_VERSION=1.1.9 --env=AUDIT_LOG_PATH=/var/log/auditlog/rancher-api-audit.log --env=AUDIT_LOG_MAXAGE=10 --env=AUDIT_LOG_MAXBACKUP=10 --env=AUDIT_LOG_MAXSIZE=100 --env=AUDIT_LEVEL=0 --env=CATTLE_CLI_URL_DARWIN=https://releases.rancher.com/cli2/v2.4.11/rancher-darwin-amd64-v2.4.11.tar.gz --env=CATTLE_CLI_URL_LINUX=https://releases.rancher.com/cli2/v2.4.11/rancher-linux-amd64-v2.4.11.tar.gz --env=CATTLE_CLI_URL_WINDOWS=https://releases.rancher.com/cli2/v2.4.11/rancher-windows-386-v2.4.11.zip --env=CATTLE_SERVER_VERSION=v2.5.8 --env=CATTLE_AGENT_IMAGE=rancher/rancher-agent:v2.5.8 --env=CATTLE_SERVER_IMAGE=rancher/rancher --env=ETCD_UNSUPPORTED_ARCH=amd64 --env=ETCDCTL_API=3 --env=SSL_CERT_DIR=/etc/rancher/ssl --volume=/rancher:/var/lib/rancher --volume=/var/lib/cni --volume=/var/lib/kubelet --volume=/var/lib/rancher --volume=/var/log --network=bridge --privileged --workdir=/var/lib/rancher -p 443:443 --restart=unless-stopped --label='org.opencontainers.image.url=https://github.com/rancher/rancher' --label='org.opencontainers.image.source=https://github.com/rancher/rancher.git' --label='org.opencontainers.image.revision=cf16ca13d02315b7a6c1641e5556db969a408e53' --label='org.opencontainers.image.created=2021-05-06T05:41:58Z' --runtime=runc --detach=true rancher/rancher:stable
root@rancher:/var/chef/cache/cookbooks/sc_rancher/recipes#

Thanks!

4

You got me wondering if chef was starting the container in some wrong way so I rm’d it and restarted it with:

docker run --name rancher --privileged -d --restart=unless-stopped -p 443:443 -v /rancher:/var/lib/rancher rancher/rancher:stable

Same results: still get the overlay/fs errors at the console (I also still get the flapping network errors detailed in my other recent post)

5

What filesystem is on / on the host? (or /rancher if thats a separate filesystem)

6

It’s an NFS v3 mount to a NetApp filer

root@rancher:~# mount | grep rancher
redacted.org:/systems/services/rancher on /rancher type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.6.165.26,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=10.6.165.26)

and the container runs with -v /rancher:/var/lib/rancher

from within the container it looks like

root@29f1e9bb3cd9:/var/lib/rancher# mount | grep rancher
redacted.org:/systems/services/rancher on /var/lib/rancher type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.6.165.26,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=10.6.165.26)
7

Pretty sure it won’t like that, can you run it with the mount from the local host’ filesystem and see if that works (just to rule it out).

I think you are hitting something similar as Failed to retrieve agent config: \"overlayfs\" snapshotter cannot be enabled · Issue #3266 · k3s-io/k3s · GitHub (as k3s is used internally)

8

Oh it runs, and mounts/reads/writes its config information. But it floods the console and the logs with the above error.

You link seems to describe running the container itself on an NFS mount. What I’m doing is running the container on a (virtual) machine’s local disk with an NFS mount inside that container where the data directory should be. This lets the VM/container be completely fungible without needing to replace the setup of the rancher/cluster information.

But I’ll try it.

9

Well, crap. The good news is if I run the container, still with the -v argument mounting the local dir /rancher, but with that dir actually local instead of an NFS mount, both the FS and network errors stop.

The bad news is I need some persistent way to keep the config information and just keeping it on a protected NFS volume was pretty damned simple.

What does Rancher offer for automated config backups?