Unable to Pull Docker Images (v.1.5.6)

Rancher 1.x
1

I’ve been running RancherOS for about 18 months now and it has been flawless. However, earlier this week I ran into a problem building new stacks. The specific error is thrown by service.upgradewait.exception:

Container should have been running but is in error state. Check logs for more information.: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

In doing some research, I’ve tried confirming that the DNS services look right:

[rancher@ArmstrongLabs ~]$ cat /etc/resolv.conf
nameserver 8.8.8.8
nameserver 4.2.2.2
nameserver 192.168.2.1

and I’m able to ping 8.8.8.8

[rancher@ArmstrongLabs ~]$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=119 time=21.476 ms
64 bytes from 8.8.8.8: seq=1 ttl=119 time=21.956 ms
64 bytes from 8.8.8.8: seq=2 ttl=119 time=21.755 ms

I did note that I’m able to ping 4.2.2.2 and 192.168.2.1. The the docker index looks correct (?):

[rancher@ArmstrongLabs ~]$ nslookup index.docker.io
Server:    8.8.8.8
Address 1: 8.8.8.8 dns.google

Name:      index.docker.io
Address 1: 50.16.172.3 ec2-50-16-172-3.compute-1.amazonaws.com
Address 2: 52.3.104.67 ec2-52-3-104-67.compute-1.amazonaws.com
Address 3: 34.193.164.221 ec2-34-193-164-221.compute-1.amazonaws.com
Address 4: 3.220.75.233 ec2-3-220-75-233.compute-1.amazonaws.com
Address 5: 54.236.224.26 ec2-54-236-224-26.compute-1.amazonaws.com
Address 6: 54.81.26.184 ec2-54-81-26-184.compute-1.amazonaws.com

To ensure I’ve isolated what I think the problem is – I’ve built a very simple MQTT using docker-compose.yml:

version: '2'
services:
  MQTT:
    image: eclipse-mosquitto:1.6.8
    #volumes:
    #- mqttBrokerStore:/mosquitto
    ports:
    - 1883:1883/tcp
    - 9001:9001/tcp

Version 1.6.8 sits on my server. When I change the image to :latest so that it would force a new pull, the error above is generated.

To double check that it isn’t a router issue – I performed all the above from a docker instance running on my PC without issues or errors.

Out of curiosity, I followed the https://registry-1.docker.io/v2/ link and it does return an error that says "authentication required". I don’t think this is the issue - the error I’m getting in RancherOS isn’t a 401/403 error type (?).

System Info:

RancherOS was a bare metal install on a Rizen 9.

[rancher@ArmstrongLabs ~]$ sudo ros os version
v1.5.6
  • As part of my trouble shooting I upgraded from v.1.5.4 to v.1.5.6 – no effect.

Any thoughts or directions to troubleshoot further would be greatly appreciated.