Rancher-webhook fails due to not existing rancher-webhook-tls secret

Rancher 2.x
1

Hi there,

Our rancher setup is practically unusable since the rancher-webhook workload is not running properly.
It will not start since the rancher-webhook-tls secret does not exist.

MountVolume.SetUp failed for volume "tls" : secret "rancher-webhook-tls" not found

We do not know exactly how this state occurred, presumably it is a state after we tried to upgrade to rancher 2.6 and then downgraded to 2.5 again.

Though we do have backups containing the secret in question, we have not been able to restore it. Oddly, the secret even shows up as a row in the mysql-based etcd of k3s, but neither will show up as a secret in kubectl or in the rancher UI.
We are not able to restore the secret using kubect. It will trigger a
Internal error occurred: failed calling webhook "rancher.cattle.io": Post https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation?timeout=10s:

I assume this is due to the MutatingWebhookConfiguration rancher.cattle.io, which will try to call exactly the currently disfunct webhook :frowning:

What is the recommended way to deal with this issue?

Thanks,
Philipp

2

I fixed it by deleting the MutatingWebhookConfiguration and then created the secret.
The MutatingWebhookConfiguration got recreated automatically.

3

how did you delete it, i am having the same issue now

4

Probably it was kubectl delete -n cattle-system MutatingWebhookConfiguration rancher.cattle.io

3 Likes
5

thanks a lot… are you currently using the Rancher cert manager?

6

How you recreate the secret ?

7

I think I scratched it from some old backup

8

Thanks this worked for me!