Hi community,
Cluster information:
Kubernetes version: v1.24.7+k3s1
Installation method: Rancher 2.7
Host OS: RHEL8
I have deployed a K3S cluster using Rancher and on top of it have installed cert-manager using yaml file. All pods are running, but when I install a helm chart I get this error message:
Error: Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": context deadline exceeded )As well, here are some logs from the pod of actions-runner-controller:
As well, the cert-manager-webhook pod is logging some errors:
Trace[1068908304]: [30.003276269s] [30.003276269s] END
E0314 15:02:02.236947 1 reflector.go:140] k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915": dial tcp 10.43.0.1:443: i/o timeout
W0314 15:03:28.953687 1 reflector.go:424] k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1.Secret: Get "https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915": dial tcp 10.43.0.1:443: i/o timeout
I0314 15:03:28.953816 1 trace.go:219] Trace[516939538]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169 (14-Mar-2023 15:02:58.949) (total time: 30004ms):
Trace[516939538]: ---"Objects listed" error:Get "https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915": dial tcp 10.43.0.1:443: i/o timeout 30004ms (15:03:28.953)
Trace[516939538]: [30.004226263s] [30.004226263s] END
E0314 15:03:28.953837 1 reflector.go:140] k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915": dial tcp 10.43.0.1:443: i/o timeout
W0314 15:04:44.919380 1 reflector.go:424] k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1.Secret: Get "https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915": dial tcp 10.43.0.1:443: i/o timeout
I0314 15:04:44.919458 1 trace.go:219] Trace[430405071]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169 (14-Mar-2023 15:04:14.918) (total time: 30000ms):
Trace[430405071]: ---"Objects listed" error:Get "https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915": dial tcp 10.43.0.1:443: i/o timeout 30000ms (15:04:44.919)
Trace[430405071]: [30.000964846s] [30.000964846s] END
E0314 15:04:44.919472 1 reflector.go:140] k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915": dial tcp 10.43.0.1:443: i/o timeout
As well, here are some logs from the pod of actions-runner-controller that I have deployed using helm:
Warning FailedMount 17m kubelet Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[kube-api-access-v48zj secret tmp cert]: timed out waiting for the condition
Warning FailedMount 8m32s kubelet Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[tmp cert kube-api-access-v48zj secret]: timed out waiting for the condition
Warning FailedMount 6m18s (x5 over 19m) kubelet Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[secret tmp cert kube-api-access-v48zj]: timed out waiting for the condition
Warning FailedMount 103s (x2 over 4m1s) kubelet Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[cert kube-api-access-v48zj secret tmp]: timed out waiting for the condition
Warning FailedMount 86s (x18 over 21m) kubelet MountVolume.SetUp failed for volume "cert" : secret "actions-runner-controller-serving-cert" not found
I have tested everything in minikube on my local pc and there everything works fine. However, it’s worth mentioning that the K3S cluster is running in an isolated environment, but probably that shouldn’t be the case.
I was trying to debug the issue and found an official cert-manager guide: Kubernetes - cert-manager Documentation.
The result of verification:
kubectl .kube\config.yaml apply -f test-resources.yaml
namespace/cert-manager-test unchanged
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": context deadline exceeded
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": context deadline exceeded
So, it’s not an issue with actions-runner-controller, but with the cert-manager or K3S deployed using Rancher. Can someone recommend how to fix this issue?
Thanks in advance,