irlanco
December 16, 2021, 6:26pm
1
Hi everyone,
I’ve seen around that k3s doesn’t support cgroup2.
opened 11:24PM - 19 Oct 21 UTC
closed 09:22PM - 20 Oct 21 UTC
I've a fresh **Debian 11** server and try to install 2.6.1 (stable) with the fol… lowing errors. Did
I also tried "latest" , "v2.6.2-rc2" and "v2.5.10"
What I am missing? Any help would be appreciated.
**EDIT: I tried it now with Ubuntu 20.04 and it works.. so it has definitely to do with Debian, but I still have no clue what causes the error. Can someone expain me what to do?**
**Rancher Server Setup**
- Rancher version: stable ( v2.6.1 (9801a059c))
**Information about the Cluster**
- Docker 20.10.9
- Debian 11
**Describe the bug**
- Fresh server
- fresh installation
- Waiting for server to become available: Get "https://127.0.0.1:6443/version?timeout=15m0s": dial tcp 127.0.0.1:6443: connect: connection refused
- k3s exited with: exit status 1
**To Reproduce**
```yaml
version: '2'
services:
rancher-server:
privileged: true
image: rancher/rancher:stable
command: --acme-domain ${rancher_hostname}
restart: unless-stopped
volumes:
- ./data/rancher:/var/lib/rancher
ports:
- 80:80
- 443:443
```
**Result**
**Expected Result**
Working installation
**Logs**
### Stable
```shell
2021/10/19 22:58:06 [INFO] Rancher version v2.6.1 (9801a059c) is starting
2021/10/19 22:58:06 [INFO] Rancher arguments {ACMEDomains:[rancher.mydomain.at] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features: ClusterRegistry:}
2021/10/19 22:58:06 [INFO] Listening on /tmp/log.sock
2021/10/19 22:58:06 [INFO] Running etcd --data-dir=management-state/etcd --heartbeat-interval=500 --election-timeout=5000
2021-10-19 22:58:06.056798 W | pkg/flags: unrecognized environment variable ETCD_UNSUPPORTED_ARCH=amd64
2021-10-19 22:58:06.056854 W | pkg/flags: unrecognized environment variable ETCD_URL=https://github.com/etcd-io/etcd/releases/download/v3.4.15/etcd-v3.4.15-linux-amd64.tar.gz
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-10-19 22:58:06.057165 I | etcdmain: etcd Version: 3.4.15
2021-10-19 22:58:06.057178 I | etcdmain: Git SHA: aa7126864
2021-10-19 22:58:06.057182 I | etcdmain: Go Version: go1.12.17
2021-10-19 22:58:06.057187 I | etcdmain: Go OS/Arch: linux/amd64
2021-10-19 22:58:06.057241 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-10-19 22:58:06.058102 I | embed: name = default
2021-10-19 22:58:06.058120 I | embed: data dir = management-state/etcd
2021-10-19 22:58:06.058124 I | embed: member dir = management-state/etcd/member
2021-10-19 22:58:06.058128 I | embed: heartbeat = 500ms
2021-10-19 22:58:06.058131 I | embed: election = 5000ms
2021-10-19 22:58:06.058135 I | embed: snapshot count = 100000
2021-10-19 22:58:06.058229 I | embed: advertise client URLs = http://localhost:2379
2021-10-19 22:58:06.064270 I | etcdserver: starting member 8e9e05c52164694d in cluster cdf818194e3a8c32
raft2021/10/19 22:58:06 INFO: 8e9e05c52164694d switched to configuration voters=()
raft2021/10/19 22:58:06 INFO: 8e9e05c52164694d became follower at term 0
raft2021/10/19 22:58:06 INFO: newRaft 8e9e05c52164694d [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
raft2021/10/19 22:58:06 INFO: 8e9e05c52164694d became follower at term 1
raft2021/10/19 22:58:06 INFO: 8e9e05c52164694d switched to configuration voters=(10276657743932975437)
2021-10-19 22:58:06.073687 W | auth: simple token is not cryptographically signed
2021-10-19 22:58:06.075666 I | etcdserver: starting server... [version: 3.4.15, cluster version: to_be_decided]
2021-10-19 22:58:06.082805 I | etcdserver: 8e9e05c52164694d as single-node; fast-forwarding 9 ticks (election ticks 10)
2021-10-19 22:58:06.083900 I | embed: listening for peers on 127.0.0.1:2380
raft2021/10/19 22:58:06 INFO: 8e9e05c52164694d switched to configuration voters=(10276657743932975437)
2021-10-19 22:58:06.084190 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
raft2021/10/19 22:58:07 INFO: 8e9e05c52164694d is starting a new election at term 1
raft2021/10/19 22:58:07 INFO: 8e9e05c52164694d became candidate at term 2
raft2021/10/19 22:58:07 INFO: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 2
raft2021/10/19 22:58:07 INFO: 8e9e05c52164694d became leader at term 2
raft2021/10/19 22:58:07 INFO: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2
2021-10-19 22:58:07.071868 I | etcdserver: setting up the initial cluster version to 3.4
2021-10-19 22:58:07.072756 N | etcdserver/membership: set the initial cluster version to 3.4
2021-10-19 22:58:07.072815 I | etcdserver/api: enabled capabilities for version 3.4
2021-10-19 22:58:07.072842 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
2021-10-19 22:58:07.072926 I | embed: ready to serve client requests
2021-10-19 22:58:07.073702 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
2021/10/19 22:58:07 [INFO] Waiting for k3s to start
time="2021-10-19T22:58:07Z" level=info msg="Acquiring lock file /var/lib/rancher/k3s/data/.lock"
time="2021-10-19T22:58:07Z" level=info msg="Preparing data dir /var/lib/rancher/k3s/data/9df574741d2573cbbe6616e8624488b36b3340d077bc50da7cb167f1b08a64d1"
2021/10/19 22:58:08 [INFO] Waiting for k3s to start
2021/10/19 22:58:09 [INFO] Waiting for k3s to start
2021/10/19 22:58:10 [INFO] Waiting for k3s to start
2021/10/19 22:58:11 [INFO] Waiting for k3s to start
exit status 1
2021/10/19 22:58:17 [FATAL] k3s exited with: exit status 1
2021/10/19 22:58:19 [INFO] Rancher version v2.6.1 (9801a059c) is starting
2021/10/19 22:58:19 [INFO] Rancher arguments {ACMEDomains:[rancher.mydomain.at] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features: ClusterRegistry:}
2021/10/19 22:58:19 [INFO] Listening on /tmp/log.sock
2021/10/19 22:58:19 [INFO] Running etcd --data-dir=management-state/etcd --heartbeat-interval=500 --election-timeout=5000
2021-10-19 22:58:19.118702 W | pkg/flags: unrecognized environment variable ETCD_UNSUPPORTED_ARCH=amd64
2021-10-19 22:58:19.118759 W | pkg/flags: unrecognized environment variable ETCD_URL=https://github.com/etcd-io/etcd/releases/download/v3.4.15/etcd-v3.4.15-linux-amd64.tar.gz
```
### Latest
```shell
2021/10/19 23:11:54 [INFO] Rancher version v2.6.1 (9801a059c) is starting
2021/10/19 23:11:54 [INFO] Rancher arguments {ACMEDomains:[rancher.mydomain.at] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features: ClusterRegistry:}
2021/10/19 23:11:54 [INFO] Listening on /tmp/log.sock
2021/10/19 23:11:54 [INFO] Running etcd --data-dir=management-state/etcd --heartbeat-interval=500 --election-timeout=5000
2021-10-19 23:11:54.943230 W | pkg/flags: unrecognized environment variable ETCD_UNSUPPORTED_ARCH=amd64
2021-10-19 23:11:54.943301 W | pkg/flags: unrecognized environment variable ETCD_URL=https://github.com/etcd-io/etcd/releases/download/v3.4.15/etcd-v3.4.15-linux-amd64.tar.gz
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-10-19 23:11:54.943367 I | etcdmain: etcd Version: 3.4.15
2021-10-19 23:11:54.943386 I | etcdmain: Git SHA: aa7126864
2021-10-19 23:11:54.943394 I | etcdmain: Go Version: go1.12.17
2021-10-19 23:11:54.943402 I | etcdmain: Go OS/Arch: linux/amd64
2021-10-19 23:11:54.943411 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-10-19 23:11:54.944475 I | embed: name = default
2021-10-19 23:11:54.944508 I | embed: data dir = management-state/etcd
2021-10-19 23:11:54.944517 I | embed: member dir = management-state/etcd/member
2021-10-19 23:11:54.944527 I | embed: heartbeat = 500ms
2021-10-19 23:11:54.944546 I | embed: election = 5000ms
2021-10-19 23:11:54.944554 I | embed: snapshot count = 100000
2021-10-19 23:11:54.944570 I | embed: advertise client URLs = http://localhost:2379
2021-10-19 23:11:54.944743 W | pkg/fileutil: check file permission: directory "management-state/etcd" exist, but the permission is "drwxr-xr-x". The recommended permission is "-rwx------" to prevent possible unprivileged access to the data.
2021-10-19 23:11:54.951600 I | etcdserver: starting member 8e9e05c52164694d in cluster cdf818194e3a8c32
raft2021/10/19 23:11:54 INFO: 8e9e05c52164694d switched to configuration voters=()
raft2021/10/19 23:11:54 INFO: 8e9e05c52164694d became follower at term 0
raft2021/10/19 23:11:54 INFO: newRaft 8e9e05c52164694d [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
raft2021/10/19 23:11:54 INFO: 8e9e05c52164694d became follower at term 1
raft2021/10/19 23:11:54 INFO: 8e9e05c52164694d switched to configuration voters=(10276657743932975437)
2021-10-19 23:11:54.954425 W | auth: simple token is not cryptographically signed
2021-10-19 23:11:54.957328 I | etcdserver: starting server... [version: 3.4.15, cluster version: to_be_decided]
2021-10-19 23:11:54.959390 I | etcdserver: 8e9e05c52164694d as single-node; fast-forwarding 9 ticks (election ticks 10)
raft2021/10/19 23:11:54 INFO: 8e9e05c52164694d switched to configuration voters=(10276657743932975437)
2021-10-19 23:11:54.960921 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
2021-10-19 23:11:54.961947 I | embed: listening for peers on 127.0.0.1:2380
raft2021/10/19 23:11:56 INFO: 8e9e05c52164694d is starting a new election at term 1
raft2021/10/19 23:11:56 INFO: 8e9e05c52164694d became candidate at term 2
raft2021/10/19 23:11:56 INFO: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 2
raft2021/10/19 23:11:56 INFO: 8e9e05c52164694d became leader at term 2
raft2021/10/19 23:11:56 INFO: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2
2021-10-19 23:11:56.454147 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
2021-10-19 23:11:56.454261 I | etcdserver: setting up the initial cluster version to 3.4
2021-10-19 23:11:56.454581 I | embed: ready to serve client requests
2021-10-19 23:11:56.454693 N | etcdserver/membership: set the initial cluster version to 3.4
2021-10-19 23:11:56.454914 I | etcdserver/api: enabled capabilities for version 3.4
2021-10-19 23:11:56.456121 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
2021/10/19 23:11:56 [INFO] Waiting for k3s to start
time="2021-10-19T23:11:56Z" level=info msg="Acquiring lock file /var/lib/rancher/k3s/data/.lock"
time="2021-10-19T23:11:56Z" level=info msg="Preparing data dir /var/lib/rancher/k3s/data/9df574741d2573cbbe6616e8624488b36b3340d077bc50da7cb167f1b08a64d1"
2021/10/19 23:11:57 [INFO] Waiting for k3s to start
2021/10/19 23:11:58 [INFO] Waiting for k3s to start
2021/10/19 23:11:59 [INFO] Waiting for k3s to start
2021/10/19 23:12:00 [INFO] Waiting for k3s to start
exit status 1
2021/10/19 23:12:08 [FATAL] k3s exited with: exit status 1
```
**Docker info**
```shell
docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.6.3-docker)
Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 1
Server Version: 20.10.9
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 5b46e404f6b9f661a205e28d59c982d3634148f8
runc version: v1.0.2-0-g52b36a2
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
cgroupns
Kernel Version: 5.10.0-8-amd64
Operating System: Debian GNU/Linux 11 (bullseye)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.745GiB
Name: rancher-server
ID: L3GY:JDLO:EDOZ:X56T:FPMM:3XFQ:JCVA:MJEA:J565:OWYU:EFF4:45CO
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
```
**Additional context**
I have my kernel and docker set to cgroup1 which I can see in docker info
Cgroup Driver: cgroupfs
Cgroup Version: 1
In my k3s logs it seems to still be having issues with cgroups. It continues to restart k3s after this cgroups issue. Is there anything else I need to set? I’m not sure why it still thinks I’m using cgroups v2 when docker is running cgroups v1.
1216 06:11:02.997536 76 node_container_manager_linux.go:57] "Failed to create cgroup" err="cannot enter cgroupv2 \"/sys/fs/cgroup/kubepods\" with domain controllers -- it is in an invalid state" cgroupName=[kubepods]
E1216 06:11:02.997570 76 kubelet.go:1384] "Failed to start ContainerManager" err="cannot enter cgroupv2 \"/sys/fs/cgroup/kubepods\" with domain controllers -- it is in an invalid state"
time="2021-12-16T06:11:20.952587036Z" level=info msg="Starting k3s v1.21.7+k3s1 (ac705709)"
time="2021-12-16T06:11:20.953206502Z" level=info msg="Managed etcd cluster bootstrap already complete and initialized"
{"level":"info","ts":"2021-12-16T06:11:21.130Z","caller":"embed/etcd.go:117","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2021-12-16T06:11:21.131Z","caller":"embed/etcd.go:127","msg":"configuring client listeners","listen-client-urls":["http://127.0.0.1:2399"]}
Thanks!!
irlanco
December 21, 2021, 6:06am
2
Well it just looks like upgrading is rough. I believe I had 5.8 previously.
I ended up having to deleting the volume of the container and starting 6.2 fresh (with the cgroup version at 1).
I was using Linode LKE and I was able to import that back in.