Cluster disconnected after Rancher K8s certificate change

Our Rancher (v.2.2.2) installation sitting on top of an RKE (v0.1.11) cluster had an issue with its internal kubernetes certificates - they got expired. As a result, the rancher portal displays a bad gateway.
I tried to get the kubeconfig file (https://gist.github.com/superseb/3d8de6092ebc4b1581185197583f472a) of the RKE cluster but it failed with this error.

certificate has expired or is not yet valid

I was able to solve that by rotating the certificate with the following steps.

./rke_0.1.11 up --config=cluster-state.yml
./rke_0.2.4 cert rotate --config=cluster-state.yml --rotate-ca

Now the rancher portal is UP but there are imported clusters that are in “Unavailable” state.

I can surely delete them and readd them again but I will lose all the settings like Projects and member permissions.

How can I make those clusters again available without losing the settings?

I was able to solve this by going to each cluster and recreating the cattle-cluster-agent pods.

1 Like