Hi,
I followed the procedure for rotating expired Webhook certificates on our Rancher 2.6.1 cluster. (Rotation of Expired Webhook Certificates | Rancher Manager).
After doing the procedure the rancher-webhook pod is in redeploy loop.
[divitel@dvsrvrkc12 ~]$ kubectl describe pod rancher-webhook -n cattle-system
Name: rancher-webhook-6979fbd4bf-nq2mg
Namespace: cattle-system
Priority: 0
Node: 10.193.50.165/10.193.50.165
Start Time: Tue, 11 Apr 2023 21:08:54 +0200
Labels: app=rancher-webhook
pod-template-hash=6979fbd4bf
Annotations:
Status: Pending
IP:
IPs:
Controlled By: ReplicaSet/rancher-webhook-6979fbd4bf
Containers:
rancher-webhook:
Container ID:
Image: rancher/rancher-webhook:v0.2.1
Image ID:
Ports: 9443/TCP, 8777/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment:
STAMP:
ENABLE_CAPI: true
ENABLE_MCM: true
NAMESPACE: cattle-system (v1:metadata.namespace)
Mounts:
/tmp/k8s-webhook-server/serving-certs from tls (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-d7hdx (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
tls:
Type: Secret (a volume populated by a Secret)
SecretName: rancher-webhook-tls
Optional: false
kube-api-access-d7hdx:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
Normal SandboxChanged 41s (x11001 over 40h) kubelet Pod sandbox changed, it will be killed and re-created.
Name: rancher-webhook-768d9f664-69mmw
Namespace: cattle-system
Priority: 0
Node: 10.193.50.164/10.193.50.164
Start Time: Tue, 11 Apr 2023 21:08:54 +0200
Labels: app=rancher-webhook
pod-template-hash=768d9f664
Annotations: cattle.io/timestamp: 2023-04-11T16:27:03Z
Status: Pending
IP:
IPs:
Controlled By: ReplicaSet/rancher-webhook-768d9f664
Containers:
rancher-webhook:
Container ID:
Image: rancher/rancher-webhook:v0.2.1
Image ID:
Ports: 9443/TCP, 8777/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment:
STAMP:
ENABLE_CAPI: true
ENABLE_MCM: true
NAMESPACE: cattle-system (v1:metadata.namespace)
Mounts:
/tmp/k8s-webhook-server/serving-certs from tls (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8th2m (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
tls:
Type: Secret (a volume populated by a Secret)
SecretName: rancher-webhook-tls
Optional: false
kube-api-access-8th2m:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
Normal SandboxChanged 40s (x11016 over 40h) kubelet Pod sandbox changed, it will be killed and re-created.
Also I see this in deploy log but now it disappear: “networkPlugin cni failed to teardown pod network: error getting ClusterInformation: connection is unauthorized”.
Any Idea? Need a redeploy of Canal CNI?
Sam.