Rancher 2.2.8 - Certificate expired

Rancher
1

Hi,

today we received error that the certificate of API Server is expired yesterday so the Rancher UI is not accessible.

Is it possible renew the certificate without UI?

Thanks,

Alessio

2

How did you install Kubernetes? If you used RKE, there is a tool that will rotate the certificates for you.

https://rancher.com/docs/rke/latest/en/cert-mgmt/#certificate-rotation

3

Hi ,

I have 2.2.4 and am in the same boat. However rke cert rotate does not seem to change the certificates in the pods/containers. Now I have to mention I came from version 2.1.x or so when it was first installed a year ago.

Side effect I see is this regretfully :slight_smile:Failed to connect to proxy" error="x509: certificate has expired or is not yet valid

The certificate in the apiserver container are right :
root@lc-rancherui01:~# docker exec -ti kube-apiserver /bin/sh
sh-4.4# cd /etc/kubernetes/ssl
sh-4.4# ls
certs kube-apiserver-requestheader-ca-key.pem kube-ca.pem kube-etcd-172-16-1-134.pem kube-etcd-172-16-1-136.pem kube-proxy.pem kube-service-account-token.pem kubecfg-kube-node.yaml
kube-apiserver-key.pem kube-apiserver-requestheader-ca.pem kube-controller-manager-key.pem kube-etcd-172-16-1-135-key.pem kube-node-key.pem kube-scheduler-key.pem kubecfg-kube-apiserver-proxy-client.yaml kubecfg-kube-proxy.yaml
kube-apiserver-proxy-client-key.pem kube-apiserver.pem kube-controller-manager.pem kube-etcd-172-16-1-135.pem kube-node.pem kube-scheduler.pem kubecfg-kube-apiserver-requestheader-ca.yaml kubecfg-kube-scheduler.yaml
kube-apiserver-proxy-client.pem kube-ca-key.pem kube-etcd-172-16-1-134-key.pem kube-etcd-172-16-1-136-key.pem kube-proxy-key.pem kube-service-account-token-key.pem kubecfg-kube-controller-manager.yaml
sh-4.4# openssl x509 -enddate -noout -in kube-ca.pem
notAfter=Oct 16 06:49:14 2028 GMT
sh-4.4# openssl x509 -enddate -noout -in kube-apiserver.pem
notAfter=Nov 2 12:52:20 2029 GMT
sh-4.4# openssl x509 -enddate -noout -in kube-apiserver-proxy-client.pem
notAfter=Nov 2 12:52:21 2029 GMT
sh-4.4# openssl x509 -enddate -noout -in kube-apiserver-requestheader-ca.pem
notAfter=Nov 2 11:44:37 2029 GMT
sh-4.4#

However talking to the api over the proxies :

root@lc-rancherui01:~# curl -vv -k https://rancher.dns.name.here/v3/connect/register

  • Trying 172.16.1.132…
  • Connected to rancher.dns.name.here (172.16.1.132) port 443 (#0)
  • found 148 certificates in /etc/ssl/certs/ca-certificates.crt
  • found 592 certificates in /etc/ssl/certs
  • ALPN, offering http/1.1
  • SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
  • server certificate verification SKIPPED
  • server certificate status verification SKIPPED
  • common name: rancher.dns.name.here (matched)
  • server certificate expiration date FAILED
  • server certificate activation date OK
  • certificate public key: RSA
  • certificate version: #3
  • subject: O=cert-manager,CN=rancher.dns.name.here
  • start date: Fri, 19 Oct 2018 07:08:41 GMT
    *** expire date: Sat, 19 Oct 2019 07:08:41 GMT**
  • issuer: O=the-ranch,CN=cattle-ca
  • compression: NULL
  • ALPN, server accepted to use http/1.1

GET /v3/connect/register HTTP/1.1
Host: rancher.dns.name.here
User-Agent: curl/7.47.0
Accept: /

< HTTP/1.1 200 OK
< Server: nginx/1.13.12
< Date: Tue, 05 Nov 2019 13:27:19 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 21
< Connection: keep-alive
< Strict-Transport-Security: max-age=15724800; includeSubDomains
<

  • Connection #0 to host rancher.dns.name.here left intact
    failed authenticationroot@lc-rancherui01:~#

Dues to this my cluster is part way unusable, I woudl assume I have a container I would have to clean up which is holding the old certificates but not sure which one.

4

Aren’t they 2 different things?
First cert is cert for k8s cluster, second is Rancher’s ingress selfsigned (n this case) cert?

5

We had a similar problem. Rancher 2.1.0 w/ publicly signed cert with a 5 yr expiry. When the self-signed k8s certs expired, the UI stopped working completely. Not even logging in with a local account worked at that point. It became easier to build & migrate to a new 2.3.1 system than fix the cert issue on the old.

6

Is there any update on this one or any solution resolve this issue.

7

You can resolve this error by deleting the Rancher internal “localhost” certificate. A restart will pick up on the missing certificate and it will recreate it. I suggest not to delete the whole folder because then the CA is also recreated and results in other errors :slight_smile:

docker exec -it rancher sh -c "rm -rf ./management-state/tls/localhost.crt"

2 Likes
8

Very handy tip here, thank you.

9

I followed this tip and my Rancher got back to work

10

Thanks for this tip, it restored my Rancher instance