Hi,
I was trying to upgrade our K8s cluster from 1.16.8 to 1.16.15 running Rancher 2.4.8.
First node upgrades successfully, second is stuck …
Pls advise what now.
$ docker ps -a -f=name='kubelet|kube-proxy'
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1d10bfad6363 rancher/hyperkube:v1.16.15-rancher1 "/opt/rke-tools/entr…" 52 minutes ago Created kubelet
07898ca4b33d rancher/hyperkube:v1.16.8-rancher1 "/opt/rke-tools/entr…" 5 months ago Up 9 days kube-proxy
6d76bf147e04 rancher/hyperkube:v1.16.8-rancher1 "/opt/rke-tools/entr…" 5 months ago Exited (0) 51 minutes ago old-kubelet
docker logs kubelet
empty
docker logs kube-proxy
I0617 12:40:24.110932 2610 proxier.go:793] Not using `--random-fully` in the MASQUERADE rule for iptables because the local version of iptables does not support it
I0617 12:40:55.195505 2610 proxier.go:793] Not using `--random-fully` in the MASQUERADE rule for iptables because the local version of iptables does not support it
I0617 12:41:25.237148 2610 proxier.go:793] Not using `--random-fully` in the MASQUERADE rule for iptables because the local version of iptables does not support it
Trying to restart the kublet shows:
Error response from daemon: Cannot restart container 1d10bfad6363: error setting label on mount source '/var/lib/kubelet': failed to set file label on /var/lib/kubelet/pods/bb92a9c2-0afa-4280-bce2-e7dabcd68bca/volumes/kubernetes.io~portworx-volume/pvc-ed89a125-8c7a-46bf-b6db-c2f3d9c6e60b: operation not permitted