Our rancher setup is practically unusable since the rancher-webhook workload is not running properly.
It will not start since the rancher-webhook-tls secret does not exist.
MountVolume.SetUp failed for volume "tls" : secret "rancher-webhook-tls" not found
We do not know exactly how this state occurred, presumably it is a state after we tried to upgrade to rancher 2.6 and then downgraded to 2.5 again.
Though we do have backups containing the secret in question, we have not been able to restore it. Oddly, the secret even shows up as a row in the mysql-based etcd of k3s, but neither will show up as a secret in kubectl or in the rancher UI.
We are not able to restore the secret using kubect. It will trigger a
Internal error occurred: failed calling webhook "rancher.cattle.io": Post https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation?timeout=10s:
I assume this is due to the MutatingWebhookConfiguration rancher.cattle.io, which will try to call exactly the currently disfunct webhook
What is the recommended way to deal with this issue?