I have three server:
1 master: 192.168.1.131
1 node: 192.168.1.132
1 rancher: 192.168.1.133
I have create docker image (private registry docker) on node used 192.168.1.132:5000/test.
Both master and node pushed and pulled to image. But used rancher deploy set image 192.168.1.132:5000/test then error:
Failed to pull image “192.168.1.132:5000/test-demo”: rpc error: code = Unknown desc = failed to pull and unpack image “192.168.1.132:5000/test-demo:latest”: failed to resolve reference “192.168.1.132:5000/test-demo:latest”: failed to do request: Head “https://192.168.1.132:5000/v2/test-demo/manifests/latest”: http: server gave HTTP response to HTTPS client.
My image used
http not https. But rancher send https.
How can I resolve “server gave HTTP response to HTTPS client”
HTTPS Client Authentication is a more secure method of authentication than either basic or form-based authentication . It uses HTTP over SSL (HTTPS), in which the server authenticates the client using the client’s Public Key Certificate (PKC).
Hopefully this will help anyone having issues getting the insecure-registries fix to work.
Try below code.
Adding
{ "insecure-registries":["host:port"] }
to
/etc/docker/daemon.json
did not work for me until I created the file
/etc/default/docker
and put the line
DOCKER_OPTS="--config-file=/etc/docker/daemon.json"
in it and then restarted the docker daemon with
sudo systemctl stop docker and sudo systemctl start docker.
For some reason just doing a sudo systemctl restart docker did not work. It threw an error about trying to restart the service to quickly.
Also for ["host:port"] I used the IP of my Docker registry as opposed to the hostname as I did not have DNS or a hosts file setup to be able to find the registry by hostname.
This drove me absolutely nuts until I stumbled upon the /etc/default/docker bit here
I am new to Docker and so I don’t know if this is new requirement since this initial post was answered or if there was something else I missed when I first setup my registry. Though all I did was to follow the current docs on the Docker site itself.
I tried your solution by adding " DOCKER_OPTS="--config-file=/etc/docker/daemon.json"" into /etc/default/docker. I still get the same error.
Is this Rancher specific issue or I should see with vanilla Kubernetes too?
We setup Harbour Container registry without enabling https.
Any help is appreciated!
Hi, Can you please give some more information:
Cluster Type where you are deploying that image (RKE/RKE2.K3s)?
How is it provisioned? (Using Rancher, or standalone and imported on rancher)?
The error points to some incorrect registries configuration. Looks like you have installed docker on the servers. Kubernetes uses containerd as it’s CRI. So adding insecure registries in your docker daemon.json will not help here. You have to tell the containerd that the registry is insecure(http) and not https. The way to do this, is configuring the registry correctly.
If this is a Cluster provisioned from Rancher UI (Not imported), then in the cluster config you’ll find a registries tab. Please configure the registry correctly there.
If this is a standalone cluster and imported in Rancher then please create a registries.yaml file with the registry config.
@vaishnav we installed RKE 2.8 using bootstrap UI on local VMs.
As you suggested we edited cluster config to add private registries as one of the mirrors.
The private registry Harbour is configured only for http.
We are still seeing same issue with ImageBackOff with message " gave HTTP response to HTTPS client".
Is http supported by Rancher/Kubernetes for private registries?
@vaishnav As Govind mentioned in config registries we have added our local registry ip in mirror section and also created regstriy secrets im attaching few images from config.yaml, error image from pods and also deployment yaml file images.
this is yaml file where we are trying to fetch image
this config cluster registries
Hi @Govind_Avireddi , Is this rke2 cluster? If yes after updating the registries, can you paste the output of the registries.yaml content from one the cluster nodes? you can find it in /etc/rancher/rke2/registries.yaml. Please feel free to mask the actual IPs. My guess is the registries might not be configured correctly.
Also in the snapshot you provided I can see that the top container registry is left blank and you have selected a secret for authentication? Can you please explain what it is? Because the registry is empty and you are providing authentication for the same. Can you put your rancher system-default-registry IP(if provided while installing rancher) there and check?
file consist:
{“configs”:{“172.22.100.185”:{“auth”:{“username”:“user”,“password”:“Pass”,“auth”:“”,“identity_token”:“”},“tls”:{“ca_file”:“”,“cert_file”:“”,“key_file”:“”,“insecure_skip_verify”:true}}},“mirrors”:{“…100.185”:{“endpoint”:[“http://…100.185:8080”]}}}
Hi @sohil344 , Do you have just one repository in your harbour registry where all your images are present? Also is this airgapped setup?
Can you please add the registry that you used while installing rancher(system-default-registry) in the container registry field(The first field in the registry config snapshot in Rancher UI you sent above) which is currently empty? Also add it in the mirrors as well just like you added the other registry. Please add the registry hostname as IP:PORT. I can see in the snapshot you have added only the IP. In the config section please add both the registry hostnames and passwords and save the config.
If this does not work, request you to share the output of /var/lib/rancher/rke2/agent/etc/containerd/config/toml masking all the sensitive information.
1 Like
We edited /etc/rancher/rke2/registries.yaml with following:
configs:
"xx.xx.100.185:8080":
auth:
username: sabc.yxz
password: letmein@123
mirrors:
xx.xx.100.185:8080:
endpoint:
- "http://xx.xx.100.185:8080"
After rebooting master and worker node, these changes were overwritten.
We went to Rancher UI and edited cluster config in YAML format under “registries” section to add the same contents. After saving YAML changes in the UI, they get reverted back to old settings.
We also referred this link but our changes according to this page are getting overwritten: Containerd Registry Configuration | RKE2
After we made changes to mirror section using Rancher UI config form, we are able to create pods pulling images from private registry.
Thank you very much @vaishnav for the help.
1 Like
Glad to know it worked @Govind_Avireddi . The reason the changes are getting overwritten is because this a rancher managed cluster. When you restart the service or reboot the node, all the configurations will be pulled from the Rancher Manager where the registry configuration might be different. Manually editing the registry.yaml should only be done if it is a standalone RKE2 cluster and not a Rancher Managed cluster since upon service restart it will the configurations which are configured from the UI.
1 Like
