"read: connection reset by peer" when pulling from insecure registry

Hi Rancher community!
Thanks for such a great product! We are happily using K3s on lightweight hardware to provide integrated open source medical applications in developing countries.

However, there a technical blocker we can’t figure out.

Cluster is running 1 master + 2 workers. k3s v1.19.7+k3s1
Master is running a insecure registry.

See the K8s files: deployment.yaml · GitHub (with the appropriate /etc/rancher/k3s/registries.yaml)

Now for some reason pulling the “big” images fails with the following error:": rpc error: code = Unknown desc = failed to pull and unpack image "": failed to copy: read tcp> read: connection reset by peer

Note that this image is present on the registry:
curl -L


and surprisingly, pulling the image manually just works perfectly fine:
sudo k3s ctr images pull --plain-http                                           resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:31a2c7d53422b3b8751c2412c920aaad44df7251627d9d80677bd7b3d3889f5f: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:23db86e92518ba6b474e62ff908b51051ac12f2f2bc99c7df9b454ae2a39a44c:   downloading    |++++++++++++++++++++++++++++++++++++++| 13.6 KiB/13.6 KiB 
layer-sha256:d080dad46627d4103daddf54c89f2ef0a0b72ba8270066c50e95ffbf4a79362e:    downloading    |--------------------------------------|  1.0 MiB/46.9 MiB 
layer-sha256:6215d7c7c20cc7c1e9ed699f00b5b2187052e3bcedd158a75e1ca1d4400e14a8:    downloading    |++++++++++++++++++++++++++++++++++++++|  555.0 B/555.0 

The registry pod does not show any error message. Only 200 codes. I would suspect some kind of network issue. I don’t know how to troubleshoot further.


  • Pulling the image “eventually” works. May take hours/days though.
  • The bigger the image, the longer it gets to finally pull the image.

Any help would be appreciated! Thanks a bunch!


Actually, little update:
sudo k3s ctr images pull --plain-http
did return the same message once (see log)

ctr: failed to copy: read tcp> read: connection reset by peer

But then 1 retry was enough to get it working…