RKE cluster administration not accessible after changing SSL certificate

Rancher 2.x
1

Hi,
I am new to rancher so I apologize if this is a newbie question.

My environment is like this:

  1. NGINX Loadbalancer (for accessing resources on both RKE clusters)=> this is where I have wildcard ssl certificate from trusted CA.
  2. 3 node RKE Cluster for Rancher.
  3. 3 node RKE cluster for application workloads.

We have bought new wildcard ssl certificate and I have replaced it on NGINX. The new certificate is from different issuer than the old one (before digicert, now Sectigo ).

When I change the certificate and key on NGINX the following happens:

  1. Rancher console and 3 Node cluster for rancher is normally accessible.
  2. Applications running on 3 node RKE cluster, for application workloads, are normally accessible.
  3. I can NOT access 3 node RKE cluster, for application workloads, from rancher console:

ERROR: Failed to ensure monitoring project name: failed to find “cattle-prometheus” Namespace: Get “https://xxx.xxx.xxx.xxx:6443/api/v1/namespaces/cattle-prometheus”: waiting for cluster [c-2nkqn] agent to connect

  1. I can not use kubectl to access or administer the 3 node RKE cluster, for application workloads.

kubectl get nodes
ERROR: Unable to connect to the server: x509: certificate signed by unknown authority

If I change the ssl certificate back to old one, everything is accessible again.

It seems that application rke cluster doesn’t want to access rancher using new ssl certificate.

Can anyone help me figure out where should I put root/intermididate cerst for the new certificate authority so this would work.

Thank you all for any help,
Andrej

2

I have changed all the tls secrets on both clusters and the issue is still the same.

If anyone has any suggestions what to do I would appreciate it very very much ;).

thanks,
Andrej

3

How many certificates are in your server certificate (tls.crt)? I guess one and its missing the intermediate, so adding this to the server certificate and creating the secret with that should be enough to make the agent connect. Based on switching from one recognized CA signed certificate to antother, only the steps in Rancher Docs: Adding TLS Secrets are required.

4

Hi,
thank you for answering. I have done all the steps in Rancher Docs: Adding TLS Secrets

As the manual instructs => Combine the server certificate followed by any intermediate certificate(s) needed into a file named tls.crt . Copy your certificate key into a file named tls.key .

My tls.crt contains:

  • wildcard certificate for my domain
  • intermidiate certificate
    . root ca certificate

Regards,
Andrej

5

Root CA doesn’t need to be in there. Did you verify the ingress controller is serving the correct certificate after changing? If you are running a fairly up-to-date Rancher, the logging of the agent containers on the host should show logging about the certificates state. You can also run

docker run --net=host superseb/ranchercheck https://your_rancher_url
6

Thanks for this. You rule :). I managed to solve the issue.

When I used your script I saw that the pšroblem is not on rke cluster side, but I was actually missing intermediate certificate on nginx loadbalancer.

Thank you again fo your help.

Regrads,
Andrej