SSL certificate renewal

Rancher 2.x
1

Hi!
I am using a letsencrypt SSL certificate for my Rancher (now) 2.4.7 installation. cert-manager version is 0.16.1.
I see that the certificate will expire on September 28.
Will it be automatically updated?
How can I trigger a certificate refresh?

Thanks!

2

I am still curious about this one.
Does anybody have any hint?

Thanks!

3

So the time has come and the letsencrypt cert expired.
It was not renewed automatically.
I tried several things:

  • delete rancher pods
  • delete cert-manager pods
  • delete tls-rancher-ingress cert
    • it was recreated in the expired form
  • delete tls-rancher-ingress and tls-rancher secrets and redeploy rancher
    • it did not recreate the secrets above
      Nothing worked.

In the end I redeployed rancher with our own certificates:
cat ~/certs/<domain>.pem ~/certs/<domain>.intermediate-ca.pem > ./tls.crt cat ~/certs/.key > ./tls.key
kubectl delete secret tls-rancher-ingress kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=tls.crt --key=tls.key
$ helm3 upgrade rancher rancher-stable/rancher --namespace cattle-system --set hostname=. --set ingress.tls.source=tls-rancher-ingress

Still I am very curious how should the letsencrypt cert have been properly renewed and I think there should be something in the Rancher install docs about this.

4

Did you save the cert-manager pods log output? Was the setup installed with the option --set ingress.tls.source=letsEncrypt?

5

Yes, initially it was deployed like this:
$ helm3 install rancher rancher-stable/rancher --namespace cattle-system --set hostname=. --set ingress.tls.source=letsEncrypt --set letsEncrypt.email=

Old logs seem to be gone as I deleted the pods.

6

Hi @superseb!
Now the letsencrypt certificate expired on another Rancher (v.2.4.8) installation.
I uploaded the logs of cert-manager pods and the tls-rancher-ingress yaml file here: https://app.box.com/s/r2nn09fzsfjpwuotn79dkp9d5zvcvsi4
Could you please have a look and let me know what I could do?

For the previous install I redeployed Rancher with certificates from files but I would really like to get the letsencrypt one renewed.

Thanks!

7

cert-manager helm chart:
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
cert-manager cert-manager 2 2020-09-02 11:26:03.75633666 +0300 EEST deployed cert-manager-v0.16.1 v0.16.1

8

Ok, so I finally managed to fix it!
Following instructions here I downgraded the cert-manager from 0.16.1 to 0.15.2 (that worked on the initial install): https://rancher.com/docs/rancher/v2.x/en/installation/resources/encryption/upgrading-cert-manager/

Here are the steps:
kubectl config set-context --current --namespace cert-manager
kubectl get -o yaml --all-namespaces issuer,clusterissuer,certificates,certificaterequests > cert-manager-backup.yaml
helm3 uninstall cert-manager
kubectl delete -f https://github.com/jetstack/cert-manager/releases/download/v0.16.1/cert-manager.crds.yaml
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.2/cert-manager.crds.yaml
helm3 install cert-manager jetstack/cert-manager --namespace cert-manager --version v0.15.2
kubectl apply -f cert-manager-backup.yaml

9

I had another Rancher installation where the letsencrypt ssl cert was not renewed. For that time I switched it to use our own cert:

cat cert.pem cert.intermediate-ca.pem > ./tls.crt
cat cert.key > ./tls.key
kubectl delete secret tls-rancher-ingress
kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=tls.crt --key=tls.key
helm3 upgrade rancher rancher-stable/rancher --namespace cattle-system --set hostname=<hostname> --set ingress.tls.source=tls-rancher-ingress

Now I managed to fix the letsencrypt cert like this:

kubectl config set-context --current --namespace cert-manager
helm3 repo update
#not actually needed#kubectl get -o yaml --all-namespaces issuer,clusterissuer,certificates,certificaterequests > cert-manager-backup.yaml
helm3 uninstall cert-manager
kubectl delete -f https://github.com/jetstack/cert-manager/releases/download/v0.15.2/cert-manager.crds.yaml
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.2/cert-manager.crds.yaml
helm3 install cert-manager jetstack/cert-manager --namespace cert-manager --version v0.15.2
kubectl config set-context --current --namespace cattle-system
kubectl delete secret tls-rancher-ingress
helm3 search repo -l rancher-stable/rancher
helm3 upgrade rancher rancher-stable/rancher --namespace cattle-system --set hostname=<hostname> --set ingress.tls.source=letsEncrypt --set letsEncrypt.email=<email> --version v2.5.3