"read: connection reset by peer" when pulling from insecure registry

k3s, k3OS, and k3d
1

Hi Rancher community!
Thanks for such a great product! We are happily using K3s on lightweight hardware to provide integrated open source medical applications in developing countries.

However, there a technical blocker we can’t figure out.

Setup:
Cluster is running 1 master + 2 workers. k3s v1.19.7+k3s1
Master is running a insecure registry.

See the K8s files: deployment.yaml · GitHub (with the appropriate /etc/rancher/k3s/registries.yaml)

Issue
Now for some reason pulling the “big” images fails with the following error:

10.0.90.99/mekomsolutions/odoo:145a9e3": rpc error: code = Unknown desc = failed to pull and unpack image "10.0.90.99/mekomsolutions/odoo:145a9e3": failed to copy: read tcp 10.0.90.12:35870->10.0.90.99:80: read: connection reset by peer

Note that this image is present on the registry:
curl -L http://10.0.90.99/v2/mekomsolutions/odoo/tags/list

{"name":"mekomsolutions/odoo","tags":["145a9e3"]}

and surprisingly, pulling the image manually just works perfectly fine:
sudo k3s ctr images pull --plain-http 10.0.90.99/mekomsolutions/odoo:145a9e

10.0.90.99/mekomsolutions/odoo:145a9e3:                                           resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:31a2c7d53422b3b8751c2412c920aaad44df7251627d9d80677bd7b3d3889f5f: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:23db86e92518ba6b474e62ff908b51051ac12f2f2bc99c7df9b454ae2a39a44c:   downloading    |++++++++++++++++++++++++++++++++++++++| 13.6 KiB/13.6 KiB 
layer-sha256:d080dad46627d4103daddf54c89f2ef0a0b72ba8270066c50e95ffbf4a79362e:    downloading    |--------------------------------------|  1.0 MiB/46.9 MiB 
layer-sha256:6215d7c7c20cc7c1e9ed699f00b5b2187052e3bcedd158a75e1ca1d4400e14a8:    downloading    |++++++++++++++++++++++++++++++++++++++|  555.0 B/555.0 
...

The registry pod does not show any error message. Only 200 codes. I would suspect some kind of network issue. I don’t know how to troubleshoot further.

Remarks:

  • Pulling the image “eventually” works. May take hours/days though.
  • The bigger the image, the longer it gets to finally pull the image.

Any help would be appreciated! Thanks a bunch!

Romain.

2

Actually, little update:
sudo k3s ctr images pull --plain-http 10.0.90.99/mekomsolutions/odoo:145a9e3
did return the same message once (see log)

ctr: failed to copy: read tcp 10.0.90.12:37968->10.0.90.99:80: read: connection reset by peer

But then 1 retry was enough to get it working…

:thinking: