Help: SSL Certs to external sites replaced with custom domain cert

Rancher 2.x
1

I have a weird issue, just managed to install rancher on k3s using following this guide https://rancher.com/docs/rancher/v2.5/en/installation/install-rancher-on-k8s/ I went with the “Certificate From Files” route, creating a secret using my pem/key files, my rancher dashboard is now serving the correct cert, but rancher itself can’t make external https calls because something is serving my cert to the internal pods as well (including rancher itself)

Summary of install:

helm install rancher rancher-latest/rancher  --namespace cattle-system  --set hostname=rancher.mydomain.com --set ingress.tls.source=secret
kubectl -n cattle-system create secret tls tls-rancher-ingress  --cert=/var/ssl/fullchain.pem --key=/var/ssl/privkey.pem

This is most obvious when adding app catalogues. The certficate subject name shown is my own domain

image
image1161×101 10.9 KB

I tried curl on google from inside of one of the rancher pods and had the same result:

root@rancher-6dc45b4c86-crtz2:/var/lib/rancher# curl https://google.com
curl: (51) SSL: no alternative certificate subject name matches target host name 'google.com'
root@rancher-6dc45b4c86-crtz2:/var/lib/rancher# curl --insecure -vvI https://www.google.com 2>&1 | awk 'BEGIN { cert=0 } /^\* SSL connection/ { cert=1 } /^\*/ { if (cert) print }'
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=mydomain.com
*  start date: Mar 23 09:26:31 2021 GMT
*  expire date: Jun 21 09:26:31 2021 GMT
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* Connection #0 to host www.google.com left intact

I’m assuming traefik is modifying the incoming response cert and replacing it with my domain’s cert? Not sure why or how to stop that happening

2

In case anyone encounters this issue, I found the solution here. It was the “search” parameter in my /etc/resolv.conf. I found a thread discussing the issue on github here: