How to create a Kubernetes cluster with existing CA certificate?

Rancher
1

What I have:

  • A running Rancher system, Version 2.4.10.
  • A number of servers for a Kubernetes cluster, standing by and with Docker already installed.
  • A CA certificate: cert.pem (full chain) and key.pem.

What I want:

  • Create a Kubernetes cluster on my servers (with Rancher) that uses my given CA certificate.
  • An easy way to deploy and change the CA certificate, especially when considering that the certificate may have a limited lifetime (~1 year).

What I did:

  1. Prepare my 1st Kubernetes server node (attempt details below).
  2. Sign in into my Rancher system.
  3. In the view “Clusters”, Click “Add cluster”.
  4. In the view “Add Cluster - Select Cluster Type”, select “From existing nodes (Custom)” as cluster type.
  5. In the view “Add Cluster - Custom”, put in a cluster name, leave everything else on default settings and click “Next”
  6. In the view “Add Cluster - Custom / Cluster Options”, select the node roles “etcd” and “Control Plane” and deselect the (preselected) node role “Worker”.
  7. Copy the given command and run it on my 1st Kubernetes node.
    $ sudo docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.4.10 --server https://[...] --token [...] --etcd --controlplane

So far I did several attempts. Everytime I started with a clean environment before running through these steps.

Attempt 1: No preparation

In this scenario, I just left the /etc/kubernetes folder empty before executing the command from step 7 on my 1st Kubernetes node. Everything works fine, except that Kubernetes is running with a self-signed certificate (which was obviously created during installation) instead of my CA certificate.

Attempt 2: Provide CA certificate

According to the Kubernetes documentation, a given certificate is considered by kubeadm. All it would take is to put the certificate in the right place: /etc/kubernetes/pki/ca.crt and /etc/kubernetes/pki/ca.key. Thats what I did. I deployed my CA certificate to that exact location before executing the command from step 7 on my 1st Kubernetes node.

However, this doesn’t seem to work when creating a Kubernetes cluster with Rancher. It looks like Rancher is completely ignoring the given certificate. Instead, Rancher creates the folder /etc/kubernetes/ssl and auto-generates a lot of (self-signed) certificates into that folder, including kube-ca.pem and kube-ca-key.pem.

The cluster creation works fine (just like attempt 1), but still … my CA certificate is not being used.

Attempt 3: Provide CA certificate (differently)

Now I “know” that Rancher is using the /etc/kubernetes/ssl folder to handle certificates. So I copied my certificates to that location: /etc/kubernetes/ssl/kube-ca.pem an /etc/kubernetes/ssl/kube-ca-key.pem before executing the command from step 7 on my 1st Kubernetes node.

This time something clicked, but not in the way I wanted.

In the Rancher web interface this error is shown:

“[etcd] Failed to bring up Etcd Plane: etcd cluster is unhealthy: hosts […] failed to report healthy. Check etcd container logs on each host for more information”

In the logs of the “etcd” container, I found this:

[...]
embed: rejected connection from "[...]:44254" (error "tls: client didn't provide a certificate", ServerName "")
[...]
embed: rejected connection from "127.0.0.1:60168" (error "tls: failed to verify client's certificate: x509: certificate signed by unknown authority", ServerName "")
[...]
grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate". Reconnecting...
[...]

These errors are not in any particular order and they were logged repeatedly.

Obviously something is off with the certificates.

I inspected the generated certificates in the /etc/kubernetes/ssl folder and noticed that they all have the issuer “CN=kube-ca”, except for my pre-provided CA certificate, of course. It seems, all these certificates were not created using the given CA certificate. I suspect that the Rancher agent internally created an own self-signed CA certificate, used that to create all the other certificates (like, kube-apiserver, kube-node, …) but failed to save it to the /etc/kubernetes/ssl folder, because the files kube-ca.pem an kube-ca-key.pem already existed.

This could explain the stange error message, referring a certificate for an emtpy server name.

Conclusion

Whatever the issue is, I still don’t know how to provide an existing CA certificate for the installation of a Kubernetes cluster using Rancher.

Rancher is presenting itself as an enterprise-ready tool to manage Kubernetes clusters. I would expect that in enterprise environments it is common to deal with certificates, signed by official CAs. So I would expect that Rancher is providing an easy-to-use mechanism to deal with that.

What am I missing?
How can I solve this?
How is this scenario supposed to be handled with Rancher?