I’m using Rancher in a single docker container running version 2.4.5 and I have imported my ‘clusters’ . I currently receive the following error when I goto the rancher GUI
2021-06-10 17:54:41.461733 I | http: TLS handshake error from 127.0.0.1:59626: remote error: tls: bad certificate
2021-06-10 17:54:43.464172 I | http: TLS handshake error from 127.0.0.1:59628: remote error: tls: bad certificate
2021/06/10 17:54:43 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version?timeout=30s: x509: certificate has expired or is not yet valid
What are the exact steps to rotate the certs , if I can’t connect to the GUI? I read that the certs expire after 1 year and once you rotate they will then expire in 10 years ? What are the exact steps to rotate the certs or create new ones ?
I was able to resolve this by issuing the following commands:
delete certificate template to force re-generation
sudo docker exec -it rancher sh -c “rm /var/lib/rancher/k3s/server/tls/dynamic-cert.json”
delete the currently deployed cert
sudo docker exec -it rancher k3s kubectl delete secret -n kube-system k3s-serving
restart rancher, this triggers the cert re-generation and brings rancher back to life
sudo docker restart rancher
Thanks so much for this @JasonK , it saved me today !
AWESOME MAN! Thanks for that!
Saved my ass! Thank you for sharing.
Thank you @JasonK , saved my evening.
Tell me where can i send you a beer
I’m so happy that my solution worked for alot of people.
I’m confused. Is it about the Rancher UI certificate, or some K3s certificate? In my case, the Rancher UI cert is expired. The k3s-serving certificate in the local cluster isn’t expired, so why delete it, and how is it supposed to help?
I can log into the UI when I skip the cert error, but only the local cluster is working - the main cluster is unavailable, because the
cattle-cluster-agent pod is crashing due to expired API cert.
Hey @JasonK ,
I cannot run kubectl command because of this error
Unable to connect to the server: x509: certificate has expired or is not yet valid
@Ahmed_Ramadan You need to add this flag: --insecure-skip-tls-verify
sudo docker exec -it rancher k3s kubectl delete secret -n kube-system k3s-serving --insecure-skip-tls-verify
Step1. docker exec -it rancher sh -c “rm /var/lib/rancher/k3s/server/tls/dynamic-cert.json”
Step2. delete secrets
serving-cert -n cattle-system & k3s-serving -n kube-system
Step3. docker restart rancher
Aamir’s list of secrets to delete is correct, the original answer did not help in our case.
So in easy to follow commands:
docker exec -it rancher sh -c "rm /var/lib/rancher/k3s/server/tls/dynamic-cert.json"
docker exec -it rancher k3s kubectl delete secret -n kube-system k3s-serving --insecure-skip-tls-verify
docker exec -it rancher k3s kubectl delete secret -n cattle-system serving-cert --insecure-skip-tls-verify
docker restart rancher
What am i doing wrong ?
:~# docker exec -it $ID k3s kubectl delete secret -n kube-system k3s-serving --insecure-skip-tls-verify
error: unknown command "kubectl" for "kubectl"
After apply the command from group recommendation rancher becomes available but all managed (downstream) cluster status changed from Available to Unavailable, how to fix this ?