Let's encrypt certificate on rancher-server: unable to get local issuer certificate

Rancher 2.x
1

Hello,

I try to register a rancher-agent, running on RancherOS, against a rancher-server with a certificate from let’s encrypt but the agent fails with the error:
“Failed to connect to proxy” error=“x509: certificate has expired or is not yet valid”

The problem occurs behind traefik as reverse proxy and beeing directly exposed.

My docker-compose file for rancher-server:

version: '3.3'

services:
  rancher-server:
  hostname: rancher
  domainname: example.tld
  container_name: rancher-server
   restart: always
   ports:
     - "192.168.1.42:80:80"
     - "192.168.1.42:443:443"
   image: rancher/rancher:latest
   command:
     - --no-cacerts
     - --log-format=simple
   volumes:
     - /vol_raidz1/docker/persistend/rancher:/var/lib/rancher:rw
      - /vol_raidz1/docker/persistend/traefik/certs/rancher.example.tld.key:/etc/rancher/ssl/key.pem:ro
      - /vol_raidz1/docker/persistend/traefik/certs/rancher.example.tld.crt:/etc/rancher/ssl/cert.pem:ro
   labels:
     - "traefik.enable=true"
     - "traefik.backend=rancher"
     - "traefik.port=443"
     - "traefik.frontend.rule=Host:rancher.example.tld"
     - "traefik.passHostHeader=true"
     - "traefik.protocol=https"
     - "com.centurylinklabs.watchtower.enable=true"
   networks:
     traefik_proxy:
       aliases:
         - rancher
       ipv4_address: 172.19.0.34
 
  networks:
    traefik_proxy:
      external: true

Output from the rancher-agent container:

openssl s_client -connect rancher.example.tld:443
[...]
Certificate chain
 0 s:CN = rancher.example.tld
   i:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
 1 s:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
[...]
    Verify return code: 20 (unable to get local issuer certificate)
[...]

curl -v https://rancher.example.tld/
[...]
* Server certificate:
*  subject: CN=rancher.example.tld
*  start date: Jul 21 12:57:33 2019 GMT
*  expire date: Oct 19 12:57:33 2019 GMT
*  subjectAltName: host "rancher.example.tld" matched cert's "rancher.example.tld"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
[...]

Output from the rancher-agent node:

openssl s_client -connect rancher.example.tld:443
[...]
Certificate chain
 0 s:CN = rancher.example.tld
   i:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
 1 s:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
[...]
    Verify return code: 0 (ok)
[...]

curl -v https://rancher.example.tld/
[...]
* Server certificate:
*  subject: CN=rancher.example.tld
*  start date: Jul 21 12:57:33 2019 GMT
*  expire date: Oct 19 12:57:33 2019 GMT
*  subjectAltName: host "rancher.example.tld" matched cert's "rancher.example.tld"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
[...]

When using an official certificate bought from a ca, it’s working.

2

I’m having a similar issue (although I haven’t tried with a store bought certificate). In my case I’m using the Traefik generated LetsEncrypt certificate and just the defaults on the Rancher server.

3

I think I’ve reduced the problem to RancherOS. So no idea why there is a difference. It’s working with an ubuntu server 1804 installation…

4

Ok, well…not really. I’ve installed docker during the installation of ubuntu server. And docker was installed via snapd bullshit. Removed snapd and installed docker from the official repository. Same problem.

5

Did you manage to resolve this? I haven’t tried again to run Rancher behind Traefik with an LE cert.