Rancher PrivateCA

Hi all,

Rancher v2.3.5

currently I am struggeling with my Rancher setup. I am running a 3 node HA setup using Helm 2 (for Gitlab Integration support). The cluster is behind a L7 Loadbalancer which manages SSL.

I spin up Rancher via:

helm install rancher-latest/rancher   --name rancher   --namespace cattle-system   --set hostname=[redacted]   --set tls=external --set privateCA=true

Then I added the cacerts.pem file like this:

kubectl -n cattle-system create secret generic tls-ca   --from-file=cacerts.pem

After this I was going to the Rancher Shell and tried to curl https://charts.gitlab.io
The result:

curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

I was also checking if the CA structure is correct which is the case. So it should be running.

I am behind a private CA which replaces all certs with a selfsigned cert.

Am I missing something? :slight_smile:

Thank you.

Minor Update:
/v3/settings/cacerts returns me all CA Certificates

Also I dont don’t get those errors on my servers itself because I already added the certificates.

The configuration setting is for accessing Rancher (making sure correct CA and cert is configured), and Rancher can be accessed. The curl https://charts.gitlab.io in the shell is an outgoing connection, if that returns invalid, there is something in between tampering with the connection. Using curl -vk https://charts.gitlab.io will show you the certificate information presented which should lead you to a root cause.

If you have any proxy configuration, please share that too as it helps investigate issues.

Hi @superseb

Thank you for your fast response.

It seems not to accept the selfsigned cert signed by our CA. I both tried to add it to ca-certificates.crt and ca-additional.pem.

Normally I would just add the certificate to the trused CAs right? Am I missing something?

rancher@redacted:~$ helm get values rancher
additionalTrustedCAs: true
hostname: redacted
privateCA: true
tls: external

These are the logs for the curl -vk https://charts.gitlab.io command.

root@rancher-6666fd4b95-xndj2:/var/lib/rancher# curl -vk https://charts.gitlab.io 
* Rebuilt URL to: https://charts.gitlab.io/
*   Trying 35.185.44.232...
* TCP_NODELAY set
* Connected to charts.gitlab.io (35.185.44.232) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: OU=Domain Control Validated; CN=*.gitlab.io
*  start date: Dec 19 16:46:20 2019 GMT
*  expire date: Jan 19 07:59:59 2021 GMT
*  issuer: C=DE; ST=Some-State [redacted]
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> GET / HTTP/1.1
> Host: charts.gitlab.io
> User-Agent: curl/7.58.0
> Accept: */*
> 

tls-ca-additional is the one we’re after here, as described on https://rancher.com/docs/rancher/v2.x/en/installation/options/chart-options/#additional-trusted-cas. That is for validating outgoing connections.

Can you verify the certificate is placed correctly in the container? (https://github.com/rancher/rancher/blob/master/chart/templates/deployment.yaml#L125-L133)

What is the error when you try to add the catalog in Rancher and Rancher tries to refresh it?

The tls-ca-additional field is added as described and the certificate should be placed corretly in the container (mappings are correct).

However if I add the catalog to Rancher I get no error but when I try to install the Gitlab Runner via Gitlab (https://rancher.com/blog/2019/connecting-gitlab-autodevops-authorized-cluster-endpoints/) I get this error in Rancher:

+ helm init --upgrade
Creating /root/.helm
Creating /root/.helm/repository
Creating /root/.helm/repository/cache
Creating /root/.helm/repository/local
Creating /root/.helm/plugins
Creating /root/.helm/starters
Creating /root/.helm/cache/archive
Creating /root/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /root/.helm.
Tiller (the Helm server-side component) has been updated to gcr.io/kubernetes-helm/tiller:v2.16.1 .
+ seq 1 30
+ helm version --tls --tls-ca-cert /data/helm/runner/config/ca.pem --tls-cert /data/helm/runner/config/cert.pem --tls-key /data/helm/runner/config/key.pem
Client: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
+ s=0
+ break
+ exit 0
+ helm repo add runner https://charts.gitlab.io
Error: Looks like "https://charts.gitlab.io" is not a valid chart repository or cannot be reached: Get https://charts.gitlab.io/index.yaml: x509: certificate signed by unknown authority

Is there any way to debug it more detailed?

Thank you!

Edit:

Do I have to redeploy some images or other additional steps after adding ca-additional.pem?

I also checked both configured paths /etc/ssl/certs/ca-additional.pem and /etc/rancher/ssl/ca-additional.pem the certificates are mounted.

Ok, that makes sense. The certificate is configured in Rancher, so Rancher will have no problem reaching the catalog. The procedure you are showing is from GitLab, and they have this described in the docs regarding the use of a proxy: https://docs.gitlab.com/ee/topics/autodevops/#installing-helm-behind-a-proxy

If you are using helm locally to install the runner, you should configure the local environment with a proxy so Helm can use that.

Let me know if that doesn’t help.

Hey now I tried to install the runner directly from the catalog. Now I am running into:

Registration attempt 13 of 30
Runtime platform arch=amd64 os=linux pid=302 revision=003fe500 version=12.7.1
WARNING: Running in user-mode.
WARNING: The user-mode requires you to manually start builds processing:
WARNING: $ gitlab-runner run
WARNING: Use sudo for system-mode:
WARNING: $ sudo gitlab-runner...
ERROR: Registering runner... failed runner=dnxi4dXg status=couldn't execute POST against https://ti-git.mah.roche.com/api/v4/runners: Post https://[redacted].com/api/v4/runners: x509: certificate signed by unknown authority
PANIC: Failed to register this runner. Perhaps you are having network problems

For my understanding:
Am I correct in my assumption that the ca-additional.pem file does not affect other pods? It’s only relevant for the Rancher system itself.

So in this case I would need to add the certificates again like this:
https://docs.gitlab.com/runner/install/kubernetes.html#providing-a-custom-certificate-for-accessing-gitlab

Thank you.

Correct.

If Rancher’s own certificates are wrongly configured, the agent won’t connect and log that. If there is a proxy in between here, there is another issue that will be fixed in the upcoming release (https://github.com/rancher/rancher/issues/24876)

If Rancher itself needs to connect to some external service, it will use ca-additional.pem to validate the certificate chain (the external service still needs to provide all the certificates needed up, that is the server certificate obviously and any intermediates). This will not be propagated or injected into any cluster resource (not the Rancher HA cluster or any managed cluster), this is where the both GitLab docs come in with options to make sure the connection can be made successfully. (I see that my link only provides an option to provide a proxy, not a certificate to validate the connection. Your link seems to show a method to provide a certificate which will then be used in the connection)

1 Like

Thank you for the explanation! :slightly_smiling_face: