Rancher-deployed vSphere-based cluster with custom ca

Rancher 2.x
1

Hi everyone,
it’s my first post and I’m fairly new to Rancher, I read all the docs I could find but I didn’t find the solution to this.

I am running rancher in an air-gapped environment, so to setup rancher itself I’ve used the specific guide for that kind of installation.
I created an RKE cluster with 3 nodes on vSphere VMs, I have my private registry where I stored all the rancher images as per the documentation, and I have my private CA which I used to generate a certificate for rancher UI.
I stored the certificate, key and root ca cert in kubernetes as secrets, and passed those secrets to helm to generate the rancher workloads, and all is working fine.
I have a nginx rev proxy in front or the 3 nodes to have HA and LB, and I’m using a configuration for nginx which supports WSS as per the documentation, in fact I’m able to log on to the UI and everything works fine there.

Now the issue:
I created a template on vSphere using Ubuntu2004 as the OS, I did all the cloud-init cleaning and the template is ready to be used by rancher, In the rancher node template I’m using vApp options to configure the OVF environment and Network Protocol Profiles on vSphere to assign the IP addresses when the nodes are created.
When I create a cluster using that node template, the VMs are built as expected, and rancher is able to deploy rke and the rancher-agent, but it doesn’t get past that because of a ca cert issue.
Looking at rancher UI, the nodes are “Waiting to register with Kubernetes” forever, and looking at the logs of the rancher agent it’s clear it is not able to open the wss connection back to rancher.
I did all my troubleshooting and I found out that actually my root ca certificate is not available inside the rancher-agent container, so I did a test by brutally (docker cp) copying the certificate from the host to the container under /usr/local/share/ca-certificates, and I ran update-ca-certificates to update the local key store, and suddenly it started to work.

How can I have the ca certificate automatically added to the certificate store whenever rancher creates a new cluster node?

Sorry if I’m not sharing any logs but as I said it is an air-gapped environment and I don’t have an easy way to extract files and post them here.

Thank you,
Daniele

2

Hi Daniele,

Are you able to validate that the root CA you provided/placed into the tls-ca file is the same as what is shown in the UI when you go to Settings -> cacerts -> Show cacerts?

Additionally, do you see logs in your rancher-agent around the CA Checksum being used?

3

Hi Oats,

I checked and it was the same, but this led me to inspect the certificate itself and I realized it was the intermediate ca cert rather than the root ca one!
Now I replaced the tls-ca secret and redeployed rancher and it’s showing the correct one.
I’m trying the first cluster deployment and it seems everything is OK now, thank you very much for pointing me in the right direction, very much appreciated.

Daniele