X509 certificate has expired or is not yet valid

Rancher
1

Hello,

I’m using Rancher in a single docker container running version 2.4.5 and I have imported my ‘clusters’ . I currently receive the following error when I goto the rancher GUI

2021-06-10 17:54:41.461733 I | http: TLS handshake error from 127.0.0.1:59626: remote error: tls: bad certificate
2021-06-10 17:54:43.464172 I | http: TLS handshake error from 127.0.0.1:59628: remote error: tls: bad certificate
2021/06/10 17:54:43 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version?timeout=30s: x509: certificate has expired or is not yet valid

What are the exact steps to rotate the certs , if I can’t connect to the GUI? I read that the certs expire after 1 year and once you rotate they will then expire in 10 years ? What are the exact steps to rotate the certs or create new ones ?

thanks

2

hello

I was able to resolve this by issuing the following commands:

delete certificate template to force re-generation

sudo docker exec -it rancher sh -c “rm /var/lib/rancher/k3s/server/tls/dynamic-cert.json”

delete the currently deployed cert

sudo docker exec -it rancher k3s kubectl delete secret -n kube-system k3s-serving

restart rancher, this triggers the cert re-generation and brings rancher back to life

sudo docker restart rancher

6 Likes
3

Thanks so much for this @JasonK , it saved me today !

4

AWESOME MAN! Thanks for that!

5

Saved my ass! Thank you for sharing.

6

Thank you @JasonK , saved my evening.
Tell me where can i send you a beer :smiley:

7

I’m so happy :smile: that my solution worked for alot of people.

1 Like
8

I’m confused. Is it about the Rancher UI certificate, or some K3s certificate? In my case, the Rancher UI cert is expired. The k3s-serving certificate in the local cluster isn’t expired, so why delete it, and how is it supposed to help?

I can log into the UI when I skip the cert error, but only the local cluster is working - the main cluster is unavailable, because the cattle-cluster-agent pod is crashing due to expired API cert.

9

Hey @JasonK ,
I cannot run kubectl command because of this error
Unable to connect to the server: x509: certificate has expired or is not yet valid

10

@Ahmed_Ramadan You need to add this flag: --insecure-skip-tls-verify

sudo docker exec -it rancher k3s kubectl delete secret -n kube-system k3s-serving --insecure-skip-tls-verify
11

FIXED

Step1. docker exec -it rancher sh -c “rm /var/lib/rancher/k3s/server/tls/dynamic-cert.json”
Step2. delete secrets

serving-cert -n cattle-system & k3s-serving -n kube-system

Step3. docker restart rancher

12

Aamir’s list of secrets to delete is correct, the original answer did not help in our case.

So in easy to follow commands:

docker exec -it rancher sh -c "rm /var/lib/rancher/k3s/server/tls/dynamic-cert.json"
docker exec -it rancher k3s kubectl delete secret -n kube-system k3s-serving --insecure-skip-tls-verify
docker exec -it rancher k3s kubectl delete secret -n cattle-system serving-cert --insecure-skip-tls-verify
docker restart rancher
13

Hello everyone.
What am i doing wrong ?

:~# docker exec -it $ID k3s kubectl delete secret -n kube-system k3s-serving --insecure-skip-tls-verify
error: unknown command "kubectl" for "kubectl"
14

Hi,

After apply the command from group recommendation rancher becomes available but all managed (downstream) cluster status changed from Available to Unavailable, how to fix this ?

Thanks

Nont