Register an imported cluster using a private registry

Rancher
1

Hello,

I’m in an half-air-gapped environment (I can access the docker registries but only through a proxy) and I have installed the rancher control-plane using our private registry just fine.
Now I’m trying to import an existing cluster and I also need to use our private registry for that.
So i’m creating the cluster in rancher and getting the manifest to apply.
Then, in this manifest I need to modify the image of the containers to use our private registry.
Once done I can apply the manifest using kubectl apply.
This is half working.
As a matter of fact the deployment and the daemon set are created in the first place using the private registry but after that it seems they are updated (by the agent itself?) and reverted to the public image name, so the pods can’t be created.

$ kubectl get all -n cattle-system
NAME                                        READY   STATUS             RESTARTS   AGE
pod/cattle-cluster-agent-657ff69f4f-vkmjk   0/1     ImagePullBackOff   0          157m
pod/cattle-cluster-agent-67ddfc445-7222s    1/1     Running            0          157m 
pod/cattle-node-agent-4j4p8                 1/1     Running            0          157m
pod/cattle-node-agent-ptkzx                 0/1     ImagePullBackOff   0          157m

NAME                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/cattle-node-agent   2         2         1       1            1           <none>          157m

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cattle-cluster-agent   1/1     1            1           157m

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/cattle-cluster-agent-657ff69f4f   1         1         0       157m
replicaset.apps/cattle-cluster-agent-67ddfc445    1         1         1       157m

If I check the the deployment at this point, it will have the standard image name, and not the one in my private registry:

$ kubectl  -n cattle-system describe deployment.apps/cattle-cluster-agent
Name:                   cattle-cluster-agent
Namespace:              cattle-system
CreationTimestamp:      Tue, 23 Jun 2020 16:47:15 -0400
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 2
Selector:               app=cattle-cluster-agent
Replicas:               1 desired | 1 updated | 2 total | 1 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app=cattle-cluster-agent
  Service Account:  cattle
  Containers:
    cluster-register:
      Image:      rancher/rancher-agent:v2.4.3 <= Here I should have myprivateregistry.com/rancher/rancher-agent:2.4.3
     Port:       <none>
     Host Port:  <none>
[...]

Any idea on how I can fix that? Is there any way to configure the ranger-agent image to look for the image at another place?

For sure I can edit again the deployment and deamonsets with the correct image name but I need to do that in a automated way and more straightforward way.
Thanks for you help.

Regards,

Olivier

2

Hi @oliverf1

I also had this issue, because the images are pulled unqualified, e.g. without registry prefix at rancher/rancher-agent. You can edit the deployment with

kubectl edit deployments cattle-cluster-agent -n cattle-system

to pull from for example: docker.io/rancher/rancher-agent or your-docker-registry/rancher/rancher-agent, but for me that should be the preferred way, because at updates etc this can be changed.

I did a little research, but I dont see any other options, as adding the docker.io registry to the unqualified-search-registries at /etc/crio/crio.conf for a K8S Cluster.

Do anyone know another way, how to add the registry prefix for the k8s cluster registration? For a RKE2 cluster the registrys can be set at the provisioning setup, but for K8S at Cluster Registration/Import, I see no option for that.

Kind Regards
Torsten

3

Hi @oliverf1

I had the same problem as you by adding a K8S Cluster to Rancher.

The main problem is, that the rancher-system agent image is pulled unqualified. So, this can be changed by editing the deployment:

kubectl edit deployments cattle-cluster-agent -n cattle-system

But for me thats not the main solution, because that can be changed later by running updates etc.

So, I prefer to add docker registry or your private docker registry, to the unqualified registry at /etc/crio/crio.conf:

unqualified-search-registries = ["private-docker-registry"]

Then the image can be pulled from docker registry. There is also the way of editing the kubectl apply yaml, but I dont prefer this.

For the RKE2 Cluster I’am able to set to own registrys, but for a cluster import I am not be able to choose a setting, that the image will be pulled e.g. by a prefix.

Do anyone know, how to import the cluster with a prefix for the rancher system agent, so I dont have to configure a unqualified registry?