Hi,
today we received error that the certificate of API Server is expired yesterday so the Rancher UI is not accessible.
Is it possible renew the certificate without UI?
Thanks,
Alessio
How did you install Kubernetes? If you used RKE, there is a tool that will rotate the certificates for you.
https://rancher.com/docs/rke/latest/en/cert-mgmt/#certificate-rotation
Hi ,
I have 2.2.4 and am in the same boat. However rke cert rotate does not seem to change the certificates in the pods/containers. Now I have to mention I came from version 2.1.x or so when it was first installed a year ago.
Side effect I see is this regretfully 
Failed to connect to proxy" error="x509: certificate has expired or is not yet valid
The certificate in the apiserver container are right :
root@lc-rancherui01:~# docker exec -ti kube-apiserver /bin/sh
sh-4.4# cd /etc/kubernetes/ssl
sh-4.4# ls
certs kube-apiserver-requestheader-ca-key.pem kube-ca.pem kube-etcd-172-16-1-134.pem kube-etcd-172-16-1-136.pem kube-proxy.pem kube-service-account-token.pem kubecfg-kube-node.yaml
kube-apiserver-key.pem kube-apiserver-requestheader-ca.pem kube-controller-manager-key.pem kube-etcd-172-16-1-135-key.pem kube-node-key.pem kube-scheduler-key.pem kubecfg-kube-apiserver-proxy-client.yaml kubecfg-kube-proxy.yaml
kube-apiserver-proxy-client-key.pem kube-apiserver.pem kube-controller-manager.pem kube-etcd-172-16-1-135.pem kube-node.pem kube-scheduler.pem kubecfg-kube-apiserver-requestheader-ca.yaml kubecfg-kube-scheduler.yaml
kube-apiserver-proxy-client.pem kube-ca-key.pem kube-etcd-172-16-1-134-key.pem kube-etcd-172-16-1-136-key.pem kube-proxy-key.pem kube-service-account-token-key.pem kubecfg-kube-controller-manager.yaml
sh-4.4# openssl x509 -enddate -noout -in kube-ca.pem
notAfter=Oct 16 06:49:14 2028 GMT
sh-4.4# openssl x509 -enddate -noout -in kube-apiserver.pem
notAfter=Nov 2 12:52:20 2029 GMT
sh-4.4# openssl x509 -enddate -noout -in kube-apiserver-proxy-client.pem
notAfter=Nov 2 12:52:21 2029 GMT
sh-4.4# openssl x509 -enddate -noout -in kube-apiserver-requestheader-ca.pem
notAfter=Nov 2 11:44:37 2029 GMT
sh-4.4#
However talking to the api over the proxies :
root@lc-rancherui01:~# curl -vv -k https://rancher.dns.name.here/v3/connect/register
- Trying 172.16.1.132…
- Connected to rancher.dns.name.here (172.16.1.132) port 443 (#0)
- found 148 certificates in /etc/ssl/certs/ca-certificates.crt
- found 592 certificates in /etc/ssl/certs
- ALPN, offering http/1.1
- SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
- server certificate verification SKIPPED
- server certificate status verification SKIPPED
- common name: rancher.dns.name.here (matched)
- server certificate expiration date FAILED
- server certificate activation date OK
- certificate public key: RSA
- certificate version: #3
- subject: O=cert-manager,CN=rancher.dns.name.here
- start date: Fri, 19 Oct 2018 07:08:41 GMT
*** expire date: Sat, 19 Oct 2019 07:08:41 GMT** - issuer: O=the-ranch,CN=cattle-ca
- compression: NULL
- ALPN, server accepted to use http/1.1
GET /v3/connect/register HTTP/1.1
Host: rancher.dns.name.here
User-Agent: curl/7.47.0
Accept: /
< HTTP/1.1 200 OK
< Server: nginx/1.13.12
< Date: Tue, 05 Nov 2019 13:27:19 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 21
< Connection: keep-alive
< Strict-Transport-Security: max-age=15724800; includeSubDomains
<
- Connection #0 to host rancher.dns.name.here left intact
failed authenticationroot@lc-rancherui01:~#
Dues to this my cluster is part way unusable, I woudl assume I have a container I would have to clean up which is holding the old certificates but not sure which one.
Aren’t they 2 different things?
First cert is cert for k8s cluster, second is Rancher’s ingress selfsigned (n this case) cert?
We had a similar problem. Rancher 2.1.0 w/ publicly signed cert with a 5 yr expiry. When the self-signed k8s certs expired, the UI stopped working completely. Not even logging in with a local account worked at that point. It became easier to build & migrate to a new 2.3.1 system than fix the cert issue on the old.
Is there any update on this one or any solution resolve this issue.
You can resolve this error by deleting the Rancher internal “localhost” certificate. A restart will pick up on the missing certificate and it will recreate it. I suggest not to delete the whole folder because then the CA is also recreated and results in other errors 
docker exec -it rancher sh -c "rm -rf ./management-state/tls/localhost.crt"
2 Likes
Very handy tip here, thank you.
I followed this tip and my Rancher got back to work
Thanks for this tip, it restored my Rancher instance