Rancher 2 Private Docker Registry?

Rancher 2.x
1

I’ve been evaluating Rancher 2 for use in my organization. On my Rancher server, pulling images from the Docker Hub registry works fine,
but I am having an issue using a private Docker Registry.

My private Docker registry…

  • vhdocker.hosp.domain.com
  • Valid CA-signed certificate for HTTPS
  • Listening on TCP 443
  • No user authentication for pushing images to, or pulling images from my private registry
  • Using Docker I am able to push images to, and pull images from my private registry

On my Rancher server I have…

  • Ubuntu 16.04.5 LTS
  • Docker version 18.06.1-ce, build e68fc7a
  • A Docker container running the latest version of Rancher 2, no other containers
  • Added the CA-signed certificate for my private Docker Registry to the Rancher server host, but not the Rancher server container.
    sudo cp vhdocker.hosp.domain.com.crt /usr/local/share/ca-certificates
    sudo update-ca-certificates

Steps

  1. I add my private Docker Registry to Rancher server
  2. Attempt to deploy a workload from an image in my private registry
  3. A pod is created on a node
    At his point I receive a minimum availability error and Rancher server appears unable to pull the image from my private Docker Registry. The following cert related errors are displayed:
    ImagePullBackOff: Back-off pulling image "vhdocker.hosp.domain.com/myapp"

ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get https://vhdocker.hosp.domain.com/v2/: x509: certificate signed by unknown authority

When running the Rancher server container with Docker is there a procedure I need to perform to pass the cert for my private Docker Registry to Rancher? Something like:
--env REGISTRY_CERTIFICATE=/mnt/certs/vhdocker.hosp.domain.com

Thanks.

Screenshots:

reg1
reg1967×483 33.6 KB
reg2
reg2837×214 22.3 KB
reg3
reg3969×247 24.6 KB
rancher1
rancher1968×435 62.5 KB
rancher2
rancher2966×676 92.2 KB

2

After a week of trial-and-error and some Googling I figured it out.

ImagePullBackOff:

To me the above indicated the issue was with Rancher-Kubernetes. Turns out this was actually Docker related.

On the host that will be running the Rancher agent container, prior to pulling the rancher agent image and running it as a container, add the certificate for your private Docker Registry to the trusted certificates on the host; then, this is the crucial part —restart the Docker daemon.

cp /mnt/certs/vhdocker.hosp.domain.com.crt /usr/local/share/ca-certificates/
update-ca-certificates
systemctl restart docker.service
ls /etc/ssl/certs | awk /vhdocker.hosp/

(Not sure if it’s necessary, but I also did this on the host that will be running my Rancher server container)

Steps

  1. Install private Docker registry cert on host that will run the Rancher server container
  2. Restart Docker daemon
  3. Run Rancher server image as a container
  4. Install private Docker registry cert on host that will run the Rancher client container
  5. Restart Docker daemon
  6. Run Rancher client image as a container

After this I was able to deploy a workload from an image in my private Docker Registry.

3 Likes
3

We encountered the same issue. Unfortunately, I think this also means anytime you have Rancher create a new node in a cluster, you’ll need to perform those same steps on the newly-created host before you can actually pull from your registry. I think the best practice is to avoid using any sort of self-signed certs so that Docker can recognize the cert on the registry automatically.

1 Like