Service rancher-webhook not found after fresh install

Rancher
1

Hello

I am trying to install Rancher 2.5.8 on a AKS cluster. The cluster was being managed by another Rancher deployment on a K3s cluster, and I didn’t unisntall the cattle agent before. The first instalation worked (with the agent still installed) but I noticed that the namespaces where grouped in nameless projects, matching the project groupings on the other Rancher instance.

I’ve tryed to clean all rancher/fleet/cattle resources before installing again (didn’t new about system-tools remove yet), but must have broken something because every single installation after that Rancher fails to startup, with variations on the following error:

2021/06/10 23:54:18 [ERROR] failed to call leader func: failed to add management data: problem reconciling role templates: couldn't create projects-create: Internal error occurred: failed calling webhook "rancherauth.cattle.io": Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s": service "rancher-webhook" not found

Rancher version: 2.5.8
Helm version: 3.6.0
Kubernetes: 1.20.5 (AKS)

Any help would be very much appreciated!

3 Likes
2

Hi I am trying to create a user using Rancher UI which is running on rke cluster and It also shows the same error.

Internal error occurred: failed calling webhook “rancherauth.cattle.io”: Post “https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s”: service “rancher-webhook” not found

Did you get a solution for this?

3

its so annoying that i am not able to do anything related to rancher due to this.

Error from server (InternalError): Internal error occurred: failed calling webhook "rancher.cattle.io": Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation?timeout=10s": service "rancher-webhook" not found
4

Has this issue been resolved?

I am getting the same error.
failed to create fleet-default/test cluster.x-k8s.io/v1alpha4, Kind=Cluster for rke-cluster fleet-default/test: Internal error occurred: failed calling webhook “default.cluster.cluster.x-k8s.io”: Post “hxxps://webhook-service.cattle-system.svc:443/mutate-cluster-x-k8s-io-v1alpha4-cluster?timeout=10s”: service “webhook-service” not found

5

After upgrading Rancher from 2.6 to 2.6.2 I am getting the same Error.
Additionally the local Cluster is stuck in State “Configuring”

Hope there is a solution soon.

6

Here I got the same error as @kbindra12 when add project member via GUI.

Hope there is a solution regarding this error.

7

Hi. I have a similar problem. I searched on a lot of websites with similar error description but so far no luck.

8

Same problem …
Rancher : v2.6.1
k8s : v1.20.10
docker : v1.20.10

If I try to update the rancher version or if I enter the user administration (new) to apply some change I see the error

Internal error occurred: failed calling webhook “rancherauth.cattle.io”: Post “https: //rancher-webhook.cattle-system.svc: 443 / v1 / webhook / validation? Timeout = 10s”: x509: certificate has expired or is not yet valid: current time 2021-11-12T16: 40: 52Z is after 2021-11-06T14: 53: 19Z

9

I make Rotation of Expired Webhook Certificates

kubectl delete secret -n cattle-system cattle-webhook-tls
kubectl delete pod -n cattle-system -l app=rancher-webhook

after this, the initial error disappeared but it is already appearing to me :

“Internal error occurred: failed calling webhook “rancherauth.cattle.io”: Post “https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s”: context deadline exceeded”.

Does anyone know how to solve the latter?

10

I got the same problem: failed calling webhook “rancher.cattle.io”: Post https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation?timeout=10s: service “rancher-webhook” not found

11

Hey Andres,
Check out this workaround by David Noland here:

1 Like
12

thank you very much, it worked for me

13

When trying to apply the workaround with Rancher 2.7.1 I get the following error.

Error from server (InternalError): an error on the server ("unable to create impersonator account: ClusterUnavailable 503: waiting for service account token secret to be populated for cluster") has prevented the request from succeeding
14

I got this timeout error after uninstalling the rancher monitoring and it scared me.

$ kubectl create namespace monitoring
Error from server (InternalError): Internal error occurred: failed calling webhook "rancher.cattle.io.namespaces.create-non-kubesystem": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/namespaces?timeout=10s": context deadline exceeded

This tutorial solved the problem:

https://www.suse.com/support/kb/doc/?id=000020699

Hope it helps somenone.