Air-gapped rke provisioning still uses docker.io

Rancher
1

I have setup a private registry at 10.208.129.99 and pushed all rancher system images into it.

Here is the cluster.yml file I used (created by ‘rke config’):

nodes:

  • address: 10.208.129.100
    port: “22”
    internal_address: “”
    role:
    • controlplane
    • worker
    • etcd
      hostname_override: “”
      user: ywb
      docker_socket: /var/run/docker.sock
      ssh_key: “”
      ssh_key_path: ~/.ssh/id_rsa
      ssh_cert: “”
      ssh_cert_path: “”
      labels: {}
      taints: []
  • address: 10.208.129.101
    port: “22”
    internal_address: “”
    role:
    • controlplane
    • worker
    • etcd
      hostname_override: “”
      user: ywb
      docker_socket: /var/run/docker.sock
      ssh_key: “”
      ssh_key_path: ~/.ssh/id_rsa
      ssh_cert: “”
      ssh_cert_path: “”
      labels: {}
      taints: []
  • address: 10.208.129.102
    port: “22”
    internal_address: “”
    role:
    • controlplane
    • worker
    • etcd
      hostname_override: “”
      user: ywb
      docker_socket: /var/run/docker.sock
      ssh_key: “”
      ssh_key_path: ~/.ssh/id_rsa
      ssh_cert: “”
      ssh_cert_path: “”
      labels: {}
      taints: []
      services:
      etcd:
      image: “”
      extra_args: {}
      extra_binds: []
      extra_env: []
      win_extra_args: {}
      win_extra_binds: []
      win_extra_env: []
      external_urls: []
      ca_cert: “”
      cert: “”
      key: “”
      path: “”
      uid: 0
      gid: 0
      snapshot: null
      retention: “”
      creation: “”
      backup_config: null
      kube-api:
      image: “”
      extra_args: {}
      extra_binds: []
      extra_env: []
      win_extra_args: {}
      win_extra_binds: []
      win_extra_env: []
      service_cluster_ip_range: 10.43.0.0/16
      service_node_port_range: “”
      pod_security_policy: false
      always_pull_images: false
      secrets_encryption_config: null
      audit_log: null
      admission_configuration: null
      event_rate_limit: null
      kube-controller:
      image: “”
      extra_args: {}
      extra_binds: []
      extra_env: []
      win_extra_args: {}
      win_extra_binds: []
      win_extra_env: []
      cluster_cidr: 10.42.0.0/16
      service_cluster_ip_range: 10.43.0.0/16
      scheduler:
      image: “”
      extra_args: {}
      extra_binds: []
      extra_env: []
      win_extra_args: {}
      win_extra_binds: []
      win_extra_env: []
      kubelet:
      image: “”
      extra_args: {}
      extra_binds: []
      extra_env: []
      win_extra_args: {}
      win_extra_binds: []
      win_extra_env: []
      cluster_domain: cluster.local
      infra_container_image: “”
      cluster_dns_server: 10.43.0.10
      fail_swap_on: false
      generate_serving_certificate: false
      kubeproxy:
      image: “”
      extra_args: {}
      extra_binds: []
      extra_env: []
      win_extra_args: {}
      win_extra_binds: []
      win_extra_env: []
      network:
      plugin: canal
      options: {}
      mtu: 0
      node_selector: {}
      update_strategy: null
      tolerations: []
      authentication:
      strategy: x509
      sans: []
      webhook: null
      addons: “”
      addons_include: []
      system_images:
      etcd: rancher/coreos-etcd:v3.4.13-rancher1
      alpine: rancher/rke-tools:v0.1.68
      nginx_proxy: rancher/rke-tools:v0.1.68
      cert_downloader: rancher/rke-tools:v0.1.68
      kubernetes_services_sidecar: rancher/rke-tools:v0.1.68
      kubedns: rancher/k8s-dns-kube-dns:1.15.10
      dnsmasq: rancher/k8s-dns-dnsmasq-nanny:1.15.10
      kubedns_sidecar: rancher/k8s-dns-sidecar:1.15.10
      kubedns_autoscaler: rancher/cluster-proportional-autoscaler:1.8.1
      coredns: rancher/coredns-coredns:1.7.0
      coredns_autoscaler: rancher/cluster-proportional-autoscaler:1.8.1
      nodelocal: rancher/k8s-dns-node-cache:1.15.13
      kubernetes: rancher/hyperkube:v1.19.6-rancher1
      flannel: rancher/coreos-flannel:v0.13.0-rancher1
      flannel_cni: rancher/flannel-cni:v0.3.0-rancher6
      calico_node: rancher/calico-node:v3.16.5
      calico_cni: rancher/calico-cni:v3.16.5
      calico_controllers: rancher/calico-kube-controllers:v3.16.5
      calico_ctl: rancher/calico-ctl:v3.16.5
      calico_flexvol: rancher/calico-pod2daemon-flexvol:v3.16.5
      canal_node: rancher/calico-node:v3.16.5
      canal_cni: rancher/calico-cni:v3.16.5
      canal_controllers: rancher/calico-kube-controllers:v3.16.5
      canal_flannel: rancher/coreos-flannel:v0.13.0-rancher1
      canal_flexvol: rancher/calico-pod2daemon-flexvol:v3.16.5
      weave_node: weaveworks/weave-kube:2.7.0
      weave_cni: weaveworks/weave-npc:2.7.0
      pod_infra_container: rancher/pause:3.2
      ingress: rancher/nginx-ingress-controller:nginx-0.35.0-rancher2
      ingress_backend: rancher/nginx-ingress-controller-defaultbackend:1.5-rancher1
      metrics_server: rancher/metrics-server:v0.3.6
      windows_pod_infra_container: rancher/kubelet-pause:v0.1.4
      aci_cni_deploy_container: noiro/cnideploy:5.1.1.0.1ae238a
      aci_host_container: noiro/aci-containers-host:5.1.1.0.1ae238a
      aci_opflex_container: noiro/opflex:5.1.1.0.1ae238a
      aci_mcast_container: noiro/opflex:5.1.1.0.1ae238a
      aci_ovs_container: noiro/openvswitch:5.1.1.0.1ae238a
      aci_controller_container: noiro/aci-containers-controller:5.1.1.0.1ae238a
      aci_gbp_server_container: noiro/gbp-server:5.1.1.0.1ae238a
      aci_opflex_server_container: noiro/opflex-server:5.1.1.0.1ae238a
      ssh_key_path: ~/.ssh/id_rsa
      ssh_cert_path: “”
      ssh_agent_auth: false
      authorization:
      mode: rbac
      options: {}
      ignore_docker_version: null
      kubernetes_version: “”
      private_registries:
  • url: 10.208.129.99
    user: admin
    password: ‘Harbor12345’
    is_default: true
    ingress:
    provider: “”
    options: {}
    node_selector: {}
    extra_args: {}
    dns_policy: “”
    extra_envs: []
    extra_volumes: []
    extra_volume_mounts: []
    update_strategy: null
    http_port: 0
    https_port: 0
    network_mode: “”
    tolerations: []
    default_backend: null
    cluster_name: “”
    cloud_provider:
    name: “”
    prefix_path: “”
    win_prefix_path: “”
    addon_job_timeout: 0
    bastion_host:
    address: “”
    port: “”
    user: “”
    ssh_key: “”
    ssh_key_path: “”
    ssh_cert: “”
    ssh_cert_path: “”
    monitoring:
    provider: “”
    options: {}
    node_selector: {}
    update_strategy: null
    replicas: null
    tolerations: []
    restore:
    restore: false
    snapshot_name: “”
    rotate_encryption_key: false
    dns: null

Here is the output of ‘rke up’:

INFO[0000] Running RKE version: v1.2.4
INFO[0000] Initiating Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [10.208.129.101]
INFO[0000] [dialer] Setup tunnel for host [10.208.129.102]
INFO[0000] [dialer] Setup tunnel for host [10.208.129.100]
INFO[0000] Checking if container [cluster-state-deployer] is running on host [10.208.129.102], try #1
INFO[0000] Pulling image [rancher/rke-tools:v0.1.68] on host [10.208.129.102], try #1
INFO[0015] Pulling image [rancher/rke-tools:v0.1.68] on host [10.208.129.102], try #1
INFO[0033] Pulling image [rancher/rke-tools:v0.1.68] on host [10.208.129.102], try #1
WARN[0048] Failed to create Docker container [cluster-state-deployer] on host [10.208.129.102]: Error response from daemon: No such image: rancher/rke-tools:v0.1.68
WARN[0048] Failed to create Docker container [cluster-state-deployer] on host [10.208.129.102]: Error response from daemon: No such image: rancher/rke-tools:v0.1.68
WARN[0048] Failed to create Docker container [cluster-state-deployer] on host [10.208.129.102]: Error response from daemon: No such image: rancher/rke-tools:v0.1.68
INFO[0048] Checking if container [cluster-state-deployer] is running on host [10.208.129.100], try #1
INFO[0048] Pulling image [rancher/rke-tools:v0.1.68] on host [10.208.129.100], try #1
INFO[0063] Pulling image [rancher/rke-tools:v0.1.68] on host [10.208.129.100], try #1
INFO[0081] Pulling image [rancher/rke-tools:v0.1.68] on host [10.208.129.100], try #1
WARN[0096] Failed to create Docker container [cluster-state-deployer] on host [10.208.129.100]: Error response from daemon: No such image: rancher/rke-tools:v0.1.68
WARN[0096] Failed to create Docker container [cluster-state-deployer] on host [10.208.129.100]: Error response from daemon: No such image: rancher/rke-tools:v0.1.68
WARN[0096] Failed to create Docker container [cluster-state-deployer] on host [10.208.129.100]: Error response from daemon: No such image: rancher/rke-tools:v0.1.68
INFO[0096] Checking if container [cluster-state-deployer] is running on host [10.208.129.101], try #1
INFO[0096] Pulling image [rancher/rke-tools:v0.1.68] on host [10.208.129.101], try #1
INFO[0111] Pulling image [rancher/rke-tools:v0.1.68] on host [10.208.129.101], try #1
INFO[0129] Pulling image [rancher/rke-tools:v0.1.68] on host [10.208.129.101], try #1
WARN[0144] Failed to create Docker container [cluster-state-deployer] on host [10.208.129.101]: Error response from daemon: No such image: rancher/rke-tools:v0.1.68
WARN[0144] Failed to create Docker container [cluster-state-deployer] on host [10.208.129.101]: Error response from daemon: No such image: rancher/rke-tools:v0.1.68
WARN[0144] Failed to create Docker container [cluster-state-deployer] on host [10.208.129.101]: Error response from daemon: No such image: rancher/rke-tools:v0.1.68
INFO[0144] [certificates] Generating CA kubernetes certificates
INFO[0144] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates
INFO[0144] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates
INFO[0144] [certificates] Generating Kubernetes API server certificates
INFO[0144] [certificates] Generating Service account token key
INFO[0144] [certificates] Generating Kube Controller certificates
INFO[0145] [certificates] Generating Kube Scheduler certificates
INFO[0145] [certificates] Generating Kube Proxy certificates
INFO[0145] [certificates] Generating Node certificate
INFO[0145] [certificates] Generating admin certificates and kubeconfig
INFO[0145] [certificates] Generating Kubernetes API server proxy client certificates
INFO[0146] [certificates] Generating kube-etcd-10-208-129-100 certificate and key
INFO[0146] [certificates] Generating kube-etcd-10-208-129-101 certificate and key
INFO[0146] [certificates] Generating kube-etcd-10-208-129-102 certificate and key
INFO[0147] Successfully Deployed state file at [./cluster.rkestate]
INFO[0147] Building Kubernetes cluster
INFO[0147] [dialer] Setup tunnel for host [10.208.129.102]
INFO[0147] [dialer] Setup tunnel for host [10.208.129.100]
INFO[0147] [dialer] Setup tunnel for host [10.208.129.101]
INFO[0147] [network] Deploying port listener containers
INFO[0147] Pulling image [rancher/rke-tools:v0.1.68] on host [10.208.129.100], try #1
INFO[0147] Pulling image [rancher/rke-tools:v0.1.68] on host [10.208.129.101], try #1
INFO[0147] Pulling image [rancher/rke-tools:v0.1.68] on host [10.208.129.102], try #1
INFO[0162] Pulling image [rancher/rke-tools:v0.1.68] on host [10.208.129.100], try #1
INFO[0162] Pulling image [rancher/rke-tools:v0.1.68] on host [10.208.129.101], try #1
INFO[0162] Pulling image [rancher/rke-tools:v0.1.68] on host [10.208.129.102], try #1
INFO[0180] Pulling image [rancher/rke-tools:v0.1.68] on host [10.208.129.100], try #1
INFO[0180] Pulling image [rancher/rke-tools:v0.1.68] on host [10.208.129.101], try #1
INFO[0180] Pulling image [rancher/rke-tools:v0.1.68] on host [10.208.129.102], try #1
WARN[0195] Failed to create Docker container [rke-etcd-port-listener] on host [10.208.129.100]: Error response from daemon: No such image: rancher/rke-tools:v0.1.68
WARN[0195] Failed to create Docker container [rke-etcd-port-listener] on host [10.208.129.100]: Error response from daemon: No such image: rancher/rke-tools:v0.1.68
WARN[0195] Failed to create Docker container [rke-etcd-port-listener] on host [10.208.129.101]: Error response from daemon: No such image: rancher/rke-tools:v0.1.68
WARN[0195] Failed to create Docker container [rke-etcd-port-listener] on host [10.208.129.100]: Error response from daemon: No such image: rancher/rke-tools:v0.1.68
WARN[0195] Failed to create Docker container [rke-etcd-port-listener] on host [10.208.129.102]: Error response from daemon: No such image: rancher/rke-tools:v0.1.68
WARN[0195] Failed to create Docker container [rke-etcd-port-listener] on host [10.208.129.101]: Error response from daemon: No such image: rancher/rke-tools:v0.1.68
WARN[0195] Failed to create Docker container [rke-etcd-port-listener] on host [10.208.129.101]: Error response from daemon: No such image: rancher/rke-tools:v0.1.68
WARN[0195] Failed to create Docker container [rke-etcd-port-listener] on host [10.208.129.102]: Error response from daemon: No such image: rancher/rke-tools:v0.1.68
WARN[0195] Failed to create Docker container [rke-etcd-port-listener] on host [10.208.129.102]: Error response from daemon: No such image: rancher/rke-tools:v0.1.68
FATA[0195] [Failed to create [rke-etcd-port-listener] container on host [10.208.129.100]: Failed to create Docker container [rke-etcd-port-listener] on host [10.208.129.100]: Error response from daemon: No such image: rancher/rke-tools:v0.1.68]

I have no idea what’s going wrong…Any help is appreciated, thx!

2

One solution is to prepend each system image with the registry url, like this:
system_images:
etcd: 10.208.129.99/rancher/coreos-etcd:v3.4.13-rancher1
…

But the rke document says that:

As of v0.1.10, you have to configure your private registry credentials, but you can specify this registry as a default registry so that all system images are pulled from the designated private registry. You can use the command rke config --system-images to get the list of default system images to populate your private registry.

Before v0.1.10, you had to configure your private registry credentials and update the names of all the system images in the cluster.yml so that the image names would have the private registry URL appended before each image name.

3

Remove system_images: key and underlying values from your cluster.yml. If you need to select a k8s version, use kubernetes_version key in cluster.yml.

4

Solved. Thanks a lot!