Unknown certificate about to expire

Rancher 2.x
1

I’m trying to edit a “local” user’s permissions. Clicking save results in this error:

Internal error occurred: failed calling webhook “rancherauth.cattle.io”: Post “https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s”: x509: certificate has expired or is not yet valid: current time 2022-11-18T01:27:14Z is after 2022-11-09T18:20:28Z

I have tried deleting the user and recreating them. Interestingly, although the same error pops up, the user is created (just not with the permissions), so I’m assuming it is a two-part process. First, creating the user, and second, assigning permissions. It is (apparently) this second part that is failing.

It is unclear what certificate this is referring to, or how to fix it, or where to look for additional data on how to fix it. The RancherUI indicates a valid certificate that won’t expire for another year, so that must not be the cert.

I’m also suspicious of those URLs. They seem like rancher defaults, and maybe they refer to something internal to K3s, and mean the “local” cluster certs need to be rotated. But, if that is the case, unlike with the other clusters there is no easy way to rotate them, and it is not clear that is the problem anyway.

Any additional information on what this error is and why it is occurring would be appreciated. Thank you!

2

I found the certificate in the local k3s cluster:

image
image2786×970 277 KB

For this particular instance of an expired cert, this issue provides guidance:

Here are the steps:

  1. Delete the expired cert (cattle-webhook-tls)
  2. Modify rancher-webhook deployment image to rancher-webhook:v0.1.1, noting the current version
  3. Wait until the cattle-webhook-tls secret is created
  4. You may need to scale down the image, then back up, and wait a minute, a few times before it regenerates the cert. Look for the message “Active TLS secret cattle-webhook-tls” in the logs.
  5. Switch back the rancher-webhook deployment image version again