Running k3s one-node cluster with minimal storage

k3s, k3OS, and k3d
1

Hello all,

I am trying to run k3s on a custom computer board where the small amount of embedded flash storage needs to be used as sparingly as possible. This being the case, for my operations the initial rootfs does not get overmounted with a filesystem on a storage device; a separate directory is the mount point for the filesystem existing on the small amount of flash memory. I am in essence trying to k3s totally in RAM. I am running into an issue where the pause containers are failing to start for all the default k3s pods when the container processes are to be jailed, according to the k3s server logs. I see a flood of error messages like the following:

E0413 17:28:26.656835  134774 remote_runtime.go:113] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:117: jailing process inside rootfs caused: pivot_root invalid argument: unknown
E0413 17:28:26.658740  134774 kuberuntime_sandbox.go:69] CreatePodSandbox for pod "helm-install-traefik-bxdcf_kube-system(1bb29148-d543-497e-a2ef-cd91267496b0)" failed: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:117: jailing process inside rootfs caused: pivot_root invalid argument: unknown
E0413 17:28:26.659760  134774 kuberuntime_manager.go:741] createPodSandbox for pod "helm-install-traefik-bxdcf_kube-system(1bb29148-d543-497e-a2ef-cd91267496b0)" failed: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:117: jailing process inside rootfs caused: pivot_root invalid argument: unknown

From the pivot_root(2) man page, I find the following note: " The rootfs (initial ramfs) cannot be pivot_root()ed. I have mounted a tmpfs at /opt/k3s and run k3s with the option -d /opt/k3s and there already exists a tmpfs at /run. To my understanding, then, the containers should not get run in, and so pivot_root should be executed on, the rootfs, but rather in a mounted filesystem (albeit one existing in RAM). At this point, I see two possiblities:

  1. I am misunderstanding, and a mounted tmpfs is still “the rootfs”, which would imply to me, then, that running containers for k3s requires some block storage
  2. The pause containers run to initialize pods are not run in either /opt/k3s (in general, the k3s data directory) or /run

The following is the config.toml.tmpl that I am using to override the k3s-builtin containerd defaults (some auth fields redacted):


[plugins.opt]
  path = "/opt/k3s/agent/containerd"


[plugins.cri]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  sandbox_image = "localhost:5000/rancher/pause:3.1"


[plugins.cri.containerd]
  snapshotter = "native"


[plugins.cri.cni]
  bin_dir = "/opt/k3s/data/current/bin"
  conf_dir = "/opt/k3s/agent/etc/cni/net.d"


[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"


[plugins.cri.registry]
  [plugins.cri.registry.mirrors]
    [plugins.cri.registry.mirrors."docker.io"]
      endpoint = ["https://localhost:5000"]
    [plugins.cri.registry.mirrors."localhost:5000"]
      endpoint = ["https://localhost:5000"]
  [plugins.cri.registry.configs]
    [plugins.cri.registry.configs."localhost:5000".auth]
      auth = *removed by author*
    [plugin.cri.registry.configs."localhost:5000".tls]
      cert_file = *removed by author*
      key_file  = *removed by author*