X509 Certificate signed by unknown authorities

Rancher
1

I have a issue with Rancher tls connection.

Here is my situation:

I have a new Kubernetes installation with 3 x Master Node and 3 x Worker Node hosted on a Proxmox VE cluster.
Rancher 2.8.3 to manage all this. I have a HA proxy on a OPNsense to access to all internal Kubernetes service.There is a reverse proxy to access to the rancher web ui. I created a CA root on opensense and a certificate from it to allow tls connection.
OpenSSL tell me that it is OK. I can go to rancher web ui without issue.

But when I tried to Kubectl get … pods, I get following issue:
E0628 13:03:34.849568 422 memcache.go:265] couldn't get current server API group list: Get "https://rancher.xxxxxxxx.local/k8s/clusters/local/api?timeout=32s": tls: failed to verify certificate: x509: certificate signed by unknown authority

What did I tried to fix that:

I supposed that as my certificate is a self signed, Rancher do not trust it. I followed following link to fix it: Rancher Helm Chart Options | Rancher No result.

Then I supposed that even if I add my own certificate, Rancher do not trust the CA root that signed my certificate. So I followed this tutorial to add my CA root to rancher. Updating the Rancher Certificate | Rancher Now Rancher web interface /v3/settings/cacerts show that my certificate was uploaded. But I have same issue.

I currently wondering if it is not HA Proxy’s fault as it initiates the ssl connection instead of Rancher?
Can we have a Kubernetes cluster without CA / ssl internally and lets HA Proxy encrypt all connections from outside?

There is something I do wrong or I didn’t understand.

Can I have your opinion about my issue?

Regards, Foreman21.

2

Hi Everybody!

I continues to investigate in my issue.

I found two nice articles hosted on SUSE’s blog about that.
TLS Termination in rancher 2.x part 1
TLS Termination in rancher 2.x part 2

Based on my situation I see two possibility to make it works:

1st possibility is to keep SSL offloading on HAProxy and set Rancher tls on external.
With this methode HAProxy manage the encrypted connection by itself and send request to Rancher uncrypted. Rancher documentation. Unfortunalty part 3 that should cover this never was published.

As the connection from client to rancher is not fully encrypted, I prefered to ignore it.

2nd possibility is to lets Rancher manages tls connection by itself. For this case HAProxy must have ssl offloading disabled. Configure HAproxy public server as tcp to allow ssl connection to reach Rancher. Reuse HAProxy’s certificate and CA root into Rancher.

I tried following commands in the same order:

After copy past my certificate into tls.crt, tls.key and my CA root into cacerts.pem,
I update rancher tls ingress to use a secret as tls source the define privateCA cariable as true.

helm upgrade rancher rancher-stable/rancher --namespace cattle-system --set hostname=rancher.xxxxx.local --set ingress.tls.source=secret --set privateCA=true’

Then I update cattle-system already created with my certificate and CA root pem.

kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=tls.crt --key=tls.key --dry-run --save-config -o yaml| kubectl apply -f -

kubectl -n cattle-system create secret generic tls-ca --from-file=cacerts.pem

Then I redeploy Rencher to allow new certificate to be implimented.

kubectl rollout restart deploy/rancher -n cattle-system

I disable offloading on HAProxy but now my browser saw me following error:

Secure Connection Failed
An error occurred during a connection to rancher.xxxxx.local. PR_END_OF_FILE_ERROR
Error code: PR_END_OF_FILE_ERROR

Openssl do not show any certificate at all. No ssl handshake either.

I am sure that my HAproxy setting (tcp mode) is good because before doing those steps, uncrypted connection rancher web page work with http://rancher.xxxxx.local:443 same for kubectl which raised following error: server gave http response to https client.

Currently I wondering if Rancher ingress handle tls termination ?
How can I check that ?
Do you see mistakes or misundestanding in my precess ?

Regards.
Foreman21.

3

Hi Everybody!

I found the issue.

There was a little misunderstanding from my side.
Rancher compare certificate from ssl connection and kubeconfig certificate.

As kubeconfig file was generated by Rancher, certificate-authority-data field is the base64 of certificate used by Rancher located at (https://rancher.xxxxx/v3/settings/cacerts).

To make kubectl works with a reverseproxy doing ssl offloading:
You need to download certificate (.pem) used by the revere proxy.
Then convert it in base64 and copy past it in certificate-authority-data field of the kubeconfig file.

Now kubectl shouls work with issue.

PS Be Careful when you compare kubeconfig certificate in base64 and your certificate in base64. Both begin and finish by same sequence.

LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0t … LS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==

I didn’t saw difference. I thinked my certificate was already inside and I didn’t go futher.

Regards,
Foreman21