Hello,
i’m a little bit lost - after upgrade rancher from version 2.6.8 to 2.6.14 my cluster (k8n 1.21.8) is not available anymore.
Logs from kublet on a control plane shows:
I0222 09:17:02.671778 1287 scope.go:111] “RemoveContainer” containerID=“02178146f580295425c7f9d9c3a98e56a594c9e4f8b1178d206f9d145470001a”
E0222 09:17:02.672764 1287 pod_workers.go:190] “Error syncing pod, skipping” err=“failed to "StartContainer" for "cluster-register" with CrashLoopBackOff: "back-off 5m0s restarting failed container=cluster-register pod=cattle-cluster-agent-5686fdd6f-bq8vk_cattle-system(f042a8ef-76a7-4417-b5fb-5b4bf306a78a)"” pod=“cattle-system/cattle-cluster-agent-5686fdd6f-bq8vk” podUID=f042a8ef-76a7-4417-b5fb-5b4bf306a78a
I0222 09:17:09.468373 1287 container_manager_linux.go:510] “Discovered runtime cgroup name” cgroupName=“/system.slice/docker.service”
I0222 09:17:15.679038 1287 scope.go:111] “RemoveContainer” containerID=“02178146f580295425c7f9d9c3a98e56a594c9e4f8b1178d206f9d145470001a”
E0222 09:17:15.679602 1287 pod_workers.go:190] “Error syncing pod, skipping” err=“failed to "StartContainer" for "cluster-register" with CrashLoopBackOff: "back-off 5m0s restarting failed container=cluster-register pod=cattle-cluster-agent-5686fdd6f-bq8vk_cattle-system(f042a8ef-76a7-4417-b5fb-5b4bf306a78a)"” pod=“cattle-system/cattle-cluster-agent-5686fdd6f-bq8vk” podUID=f042a8ef-76a7-4417-b5fb-5b4bf306a78a
Any idea what to do? The cattle-cluster-agent pod does not have any logs.
:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4e25a4bfcba5 rancher/mirrored-pause:3.4.1 "/pause" 10 minutes ago Up 10 minutes k8s_POD_cattle-cluster-agent-5686fdd6f-bq8vk_cattle-system_f042a8ef-76a7-4417-b5fb-5b4bf306a78a_4
832537627c1a b77f357cacfc "run.sh" 25 minutes ago Up 25 minutes k8s_agent_cattle-node-agent-67mm6_cattle-system_8a8293ea-b160-4957-bffe-11b12d854b78_0
bc6be195585c rancher/mirrored-pause:3.4.1 "/pause" 25 minutes ago Up 25 minutes k8s_POD_cattle-node-agent-67mm6_cattle-system_8a8293ea-b160-4957-bffe-11b12d854b78_0
8eeada685b7a e6ea68648f0c "/opt/bin/flanneld -…" 18 hours ago Up 18 hours k8s_kube-flannel_canal-wwxln_kube-system_66f1ac20-fe19-41f4-a231-437f94edcc50_6
3138b6edcfaa 7aa1277761b5 "start_runit" 18 hours ago Up 18 hours k8s_calico-node_canal-wwxln_kube-system_66f1ac20-fe19-41f4-a231-437f94edcc50_6
a5613eb55c27 0018da5eed5f "/bin/sh -c 'kube-ap…" 18 hours ago Up 18 hours k8s_kube-api-auth_kube-api-auth-wbb7h_cattle-system_30b00a8a-ceeb-4413-be97-b97a44c44b01_2
6b1f91035ae1 rancher/mirrored-pause:3.4.1 "/pause" 18 hours ago Up 18 hours k8s_POD_kube-api-auth-wbb7h_cattle-system_30b00a8a-ceeb-4413-be97-b97a44c44b01_2
9d594adaadf8 rancher/mirrored-pause:3.4.1 "/pause" 18 hours ago Up 18 hours k8s_POD_canal-wwxln_kube-system_66f1ac20-fe19-41f4-a231-437f94edcc50_6
46e483cc2227 rancher/hyperkube:v1.21.8-rancher2 "/opt/rke-tools/entr…" 2 years ago Up 18 hours kube-proxy
2f959af1605b rancher/hyperkube:v1.21.8-rancher2 "/opt/rke-tools/entr…" 2 years ago Up 18 hours kubelet
38869552a814 rancher/hyperkube:v1.21.8-rancher2 "/opt/rke-tools/entr…" 2 years ago Up 18 hours kube-scheduler
3f9b447a21f7 rancher/hyperkube:v1.21.8-rancher2 "/opt/rke-tools/entr…" 2 years ago Up 18 hours kube-controller-manager
c1947695406c rancher/hyperkube:v1.21.8-rancher2 "/opt/rke-tools/entr…" 2 years ago Up 18 hours kube-apiserver
ea77acc4d66e rancher/mirrored-coreos-etcd:v3.4.16-rancher1 "/usr/local/bin/etcd…" 2 years ago Up 18 hours etcd