Hi folks,
I am trying to setup Rancher on 3 kvms. rke up --config ./rancher.yml fails with
INFO[0000] Running RKE version: v1.1.3
INFO[0000] Initiating Kubernetes cluster
INFO[0000] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates
INFO[0000] [certificates] Generating admin certificates and kubeconfig
INFO[0000] Successfully Deployed state file at [./rancher.rkestate]
INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [rr03.ac.example.de]
INFO[0000] [dialer] Setup tunnel for host [rr01.ac.example.de]
INFO[0000] [dialer] Setup tunnel for host [rr02.ac.example.de]
INFO[0000] [network] No hosts added existing cluster, skipping port check
INFO[0000] [certificates] kube-apiserver certificate changed, force deploying certs
INFO[0000] [certificates] Deploying kubernetes certificates to Cluster nodes
INFO[0000] Checking if container [cert-deployer] is running on host [rr03.ac.example.de], try #1
INFO[0000] Checking if container [cert-deployer] is running on host [rr02.ac.example.de], try #1
INFO[0000] Checking if container [cert-deployer] is running on host [rr01.ac.example.de], try #1
INFO[0000] Pulling image [rancher/rke-tools:v0.1.58] on host [rr03.ac.example.de], try #1
INFO[0000] Pulling image [rancher/rke-tools:v0.1.58] on host [rr01.ac.example.de], try #1
INFO[0000] Pulling image [rancher/rke-tools:v0.1.58] on host [rr02.ac.example.de], try #1
INFO[0009] Image [rancher/rke-tools:v0.1.58] exists on host [rr03.ac.example.de]
INFO[0009] Image [rancher/rke-tools:v0.1.58] exists on host [rr01.ac.example.de]
INFO[0009] Starting container [cert-deployer] on host [rr03.ac.example.de], try #1
INFO[0009] Starting container [cert-deployer] on host [rr01.ac.example.de], try #1
INFO[0009] Checking if container [cert-deployer] is running on host [rr03.ac.example.de], try #1
INFO[0010] Checking if container [cert-deployer] is running on host [rr01.ac.example.de], try #1
INFO[0010] Image [rancher/rke-tools:v0.1.58] exists on host [rr02.ac.example.de]
INFO[0012] Starting container [cert-deployer] on host [rr02.ac.example.de], try #1
INFO[0012] Checking if container [cert-deployer] is running on host [rr02.ac.example.de], try #1
INFO[0014] Checking if container [cert-deployer] is running on host [rr03.ac.example.de], try #1
INFO[0014] Removing container [cert-deployer] on host [rr03.ac.example.de], try #1
INFO[0015] Checking if container [cert-deployer] is running on host [rr01.ac.example.de], try #1
INFO[0015] Removing container [cert-deployer] on host [rr01.ac.example.de], try #1
INFO[0017] Checking if container [cert-deployer] is running on host [rr02.ac.example.de], try #1
INFO[0017] Removing container [cert-deployer] on host [rr02.ac.example.de], try #1
INFO[0017] [reconcile] Rebuilding and updating local kube config
INFO[0017] Successfully Deployed local admin kubeconfig at [./kube_config_rancher.yml]
INFO[0017] Successfully Deployed local admin kubeconfig at [./kube_config_rancher.yml]
INFO[0017] Successfully Deployed local admin kubeconfig at [./kube_config_rancher.yml]
INFO[0017] [certificates] Successfully deployed kubernetes certificates to Cluster nodes
INFO[0017] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [rr01.ac.example.de]
INFO[0017] Image [rancher/rke-tools:v0.1.58] exists on host [rr01.ac.example.de]
INFO[0017] Starting container [file-deployer] on host [rr01.ac.example.de], try #1
INFO[0018] Successfully started [file-deployer] container on host [rr01.ac.example.de]
INFO[0018] Waiting for [file-deployer] container to exit on host [rr01.ac.example.de]
INFO[0018] Waiting for [file-deployer] container to exit on host [rr01.ac.example.de]
INFO[0018] Container [file-deployer] is still running on host [rr01.ac.example.de]
INFO[0019] Waiting for [file-deployer] container to exit on host [rr01.ac.example.de]
INFO[0019] Removing container [file-deployer] on host [rr01.ac.example.de], try #1
INFO[0019] [remove/file-deployer] Successfully removed container on host [rr01.ac.example.de]
INFO[0019] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [rr02.ac.example.de]
INFO[0019] Image [rancher/rke-tools:v0.1.58] exists on host [rr02.ac.example.de]
INFO[0019] Starting container [file-deployer] on host [rr02.ac.example.de], try #1
INFO[0020] Successfully started [file-deployer] container on host [rr02.ac.example.de]
INFO[0020] Waiting for [file-deployer] container to exit on host [rr02.ac.example.de]
INFO[0020] Waiting for [file-deployer] container to exit on host [rr02.ac.example.de]
INFO[0020] Container [file-deployer] is still running on host [rr02.ac.example.de]
INFO[0021] Waiting for [file-deployer] container to exit on host [rr02.ac.example.de]
INFO[0021] Removing container [file-deployer] on host [rr02.ac.example.de], try #1
INFO[0021] [remove/file-deployer] Successfully removed container on host [rr02.ac.example.de]
INFO[0021] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [rr03.ac.example.de]
INFO[0021] Image [rancher/rke-tools:v0.1.58] exists on host [rr03.ac.example.de]
INFO[0021] Starting container [file-deployer] on host [rr03.ac.example.de], try #1
INFO[0021] Successfully started [file-deployer] container on host [rr03.ac.example.de]
INFO[0021] Waiting for [file-deployer] container to exit on host [rr03.ac.example.de]
INFO[0021] Waiting for [file-deployer] container to exit on host [rr03.ac.example.de]
INFO[0021] Container [file-deployer] is still running on host [rr03.ac.example.de]
INFO[0022] Waiting for [file-deployer] container to exit on host [rr03.ac.example.de]
INFO[0022] Removing container [file-deployer] on host [rr03.ac.example.de], try #1
INFO[0022] [remove/file-deployer] Successfully removed container on host [rr03.ac.example.de]
INFO[0022] [/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes
INFO[0022] [reconcile] Reconciling cluster state
INFO[0022] [reconcile] Check etcd hosts to be deleted
INFO[0022] [reconcile] Check etcd hosts to be added
INFO[0022] [reconcile] Rebuilding and updating local kube config
INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_rancher.yml]
INFO[0023] Successfully Deployed local admin kubeconfig at [./kube_config_rancher.yml]
INFO[0023] Successfully Deployed local admin kubeconfig at [./kube_config_rancher.yml]
INFO[0023] [reconcile] Reconciled cluster state successfully
INFO[0023] max_unavailable_worker got rounded down to 0, resetting to 1
INFO[0023] Setting maxUnavailable for worker nodes to: 1
INFO[0023] Setting maxUnavailable for controlplane nodes to: 1
INFO[0023] Pre-pulling kubernetes images
INFO[0023] Pulling image [rancher/hyperkube:v1.18.3-rancher2] on host [rr03.ac.example.de], try #1
INFO[0023] Pulling image [rancher/hyperkube:v1.18.3-rancher2] on host [rr02.ac.example.de], try #1
INFO[0023] Pulling image [rancher/hyperkube:v1.18.3-rancher2] on host [rr01.ac.example.de], try #1
INFO[0084] Image [rancher/hyperkube:v1.18.3-rancher2] exists on host [rr03.ac.example.de]
INFO[0086] Image [rancher/hyperkube:v1.18.3-rancher2] exists on host [rr01.ac.example.de]
INFO[0091] Image [rancher/hyperkube:v1.18.3-rancher2] exists on host [rr02.ac.example.de]
INFO[0091] Kubernetes images pulled successfully
INFO[0091] [etcd] Building up etcd plane..
INFO[0091] Image [rancher/rke-tools:v0.1.58] exists on host [rr01.ac.example.de]
INFO[0093] Starting container [etcd-fix-perm] on host [rr01.ac.example.de], try #1
INFO[0093] Successfully started [etcd-fix-perm] container on host [rr01.ac.example.de]
INFO[0093] Waiting for [etcd-fix-perm] container to exit on host [rr01.ac.example.de]
INFO[0093] Waiting for [etcd-fix-perm] container to exit on host [rr01.ac.example.de]
INFO[0093] Container [etcd-fix-perm] is still running on host [rr01.ac.example.de]
INFO[0094] Waiting for [etcd-fix-perm] container to exit on host [rr01.ac.example.de]
INFO[0094] Removing container [etcd-fix-perm] on host [rr01.ac.example.de], try #1
INFO[0094] [remove/etcd-fix-perm] Successfully removed container on host [rr01.ac.example.de]
INFO[0094] Pulling image [rancher/coreos-etcd:v3.4.3-rancher1] on host [rr01.ac.example.de], try #1
INFO[0101] Image [rancher/coreos-etcd:v3.4.3-rancher1] exists on host [rr01.ac.example.de]
INFO[0101] Starting container [etcd] on host [rr01.ac.example.de], try #1
INFO[0102] [etcd] Successfully started [etcd] container on host [rr01.ac.example.de]
INFO[0102] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [rr01.ac.example.de]
INFO[0102] Image [rancher/rke-tools:v0.1.58] exists on host [rr01.ac.example.de]
INFO[0102] Starting container [etcd-rolling-snapshots] on host [rr01.ac.example.de], try #1
INFO[0102] [etcd] Successfully started [etcd-rolling-snapshots] container on host [rr01.ac.example.de]
INFO[0107] Image [rancher/rke-tools:v0.1.58] exists on host [rr01.ac.example.de]
INFO[0108] Starting container [rke-bundle-cert] on host [rr01.ac.example.de], try #1
INFO[0108] [certificates] Successfully started [rke-bundle-cert] container on host [rr01.ac.example.de]
INFO[0108] Waiting for [rke-bundle-cert] container to exit on host [rr01.ac.example.de]
INFO[0108] Container [rke-bundle-cert] is still running on host [rr01.ac.example.de]
INFO[0109] Waiting for [rke-bundle-cert] container to exit on host [rr01.ac.example.de]
INFO[0109] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [rr01.ac.example.de]
INFO[0109] Removing container [rke-bundle-cert] on host [rr01.ac.example.de], try #1
INFO[0109] Image [rancher/rke-tools:v0.1.58] exists on host [rr01.ac.example.de]
INFO[0109] Starting container [rke-log-linker] on host [rr01.ac.example.de], try #1
INFO[0110] [etcd] Successfully started [rke-log-linker] container on host [rr01.ac.example.de]
INFO[0110] Removing container [rke-log-linker] on host [rr01.ac.example.de], try #1
INFO[0110] [remove/rke-log-linker] Successfully removed container on host [rr01.ac.example.de]
INFO[0110] Image [rancher/rke-tools:v0.1.58] exists on host [rr02.ac.example.de]
INFO[0112] Starting container [etcd-fix-perm] on host [rr02.ac.example.de], try #1
INFO[0112] Successfully started [etcd-fix-perm] container on host [rr02.ac.example.de]
INFO[0112] Waiting for [etcd-fix-perm] container to exit on host [rr02.ac.example.de]
INFO[0112] Waiting for [etcd-fix-perm] container to exit on host [rr02.ac.example.de]
INFO[0112] Container [etcd-fix-perm] is still running on host [rr02.ac.example.de]
INFO[0113] Waiting for [etcd-fix-perm] container to exit on host [rr02.ac.example.de]
INFO[0113] Removing container [etcd-fix-perm] on host [rr02.ac.example.de], try #1
INFO[0113] [remove/etcd-fix-perm] Successfully removed container on host [rr02.ac.example.de]
INFO[0113] Pulling image [rancher/coreos-etcd:v3.4.3-rancher1] on host [rr02.ac.example.de], try #1
INFO[0120] Image [rancher/coreos-etcd:v3.4.3-rancher1] exists on host [rr02.ac.example.de]
INFO[0121] Starting container [etcd] on host [rr02.ac.example.de], try #1
INFO[0121] [etcd] Successfully started [etcd] container on host [rr02.ac.example.de]
INFO[0121] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [rr02.ac.example.de]
INFO[0121] Image [rancher/rke-tools:v0.1.58] exists on host [rr02.ac.example.de]
INFO[0121] Starting container [etcd-rolling-snapshots] on host [rr02.ac.example.de], try #1
INFO[0122] [etcd] Successfully started [etcd-rolling-snapshots] container on host [rr02.ac.example.de]
INFO[0127] Image [rancher/rke-tools:v0.1.58] exists on host [rr02.ac.example.de]
INFO[0127] Starting container [rke-bundle-cert] on host [rr02.ac.example.de], try #1
INFO[0128] [certificates] Successfully started [rke-bundle-cert] container on host [rr02.ac.example.de]
INFO[0128] Waiting for [rke-bundle-cert] container to exit on host [rr02.ac.example.de]
INFO[0128] Container [rke-bundle-cert] is still running on host [rr02.ac.example.de]
INFO[0129] Waiting for [rke-bundle-cert] container to exit on host [rr02.ac.example.de]
INFO[0129] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [rr02.ac.example.de]
INFO[0129] Removing container [rke-bundle-cert] on host [rr02.ac.example.de], try #1
INFO[0129] Image [rancher/rke-tools:v0.1.58] exists on host [rr02.ac.example.de]
INFO[0129] Starting container [rke-log-linker] on host [rr02.ac.example.de], try #1
INFO[0130] [etcd] Successfully started [rke-log-linker] container on host [rr02.ac.example.de]
INFO[0130] Removing container [rke-log-linker] on host [rr02.ac.example.de], try #1
INFO[0130] [remove/rke-log-linker] Successfully removed container on host [rr02.ac.example.de]
INFO[0130] Image [rancher/rke-tools:v0.1.58] exists on host [rr03.ac.example.de]
INFO[0130] Starting container [etcd-fix-perm] on host [rr03.ac.example.de], try #1
INFO[0131] Successfully started [etcd-fix-perm] container on host [rr03.ac.example.de]
INFO[0131] Waiting for [etcd-fix-perm] container to exit on host [rr03.ac.example.de]
INFO[0131] Waiting for [etcd-fix-perm] container to exit on host [rr03.ac.example.de]
INFO[0131] Container [etcd-fix-perm] is still running on host [rr03.ac.example.de]
INFO[0132] Waiting for [etcd-fix-perm] container to exit on host [rr03.ac.example.de]
INFO[0132] Removing container [etcd-fix-perm] on host [rr03.ac.example.de], try #1
INFO[0132] [remove/etcd-fix-perm] Successfully removed container on host [rr03.ac.example.de]
INFO[0132] Pulling image [rancher/coreos-etcd:v3.4.3-rancher1] on host [rr03.ac.example.de], try #1
INFO[0138] Image [rancher/coreos-etcd:v3.4.3-rancher1] exists on host [rr03.ac.example.de]
INFO[0138] Starting container [etcd] on host [rr03.ac.example.de], try #1
INFO[0138] [etcd] Successfully started [etcd] container on host [rr03.ac.example.de]
INFO[0138] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [rr03.ac.example.de]
INFO[0138] Image [rancher/rke-tools:v0.1.58] exists on host [rr03.ac.example.de]
INFO[0138] Starting container [etcd-rolling-snapshots] on host [rr03.ac.example.de], try #1
INFO[0139] [etcd] Successfully started [etcd-rolling-snapshots] container on host [rr03.ac.example.de]
INFO[0144] Image [rancher/rke-tools:v0.1.58] exists on host [rr03.ac.example.de]
INFO[0144] Starting container [rke-bundle-cert] on host [rr03.ac.example.de], try #1
INFO[0145] [certificates] Successfully started [rke-bundle-cert] container on host [rr03.ac.example.de]
INFO[0145] Waiting for [rke-bundle-cert] container to exit on host [rr03.ac.example.de]
INFO[0145] Container [rke-bundle-cert] is still running on host [rr03.ac.example.de]
INFO[0146] Waiting for [rke-bundle-cert] container to exit on host [rr03.ac.example.de]
INFO[0146] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [rr03.ac.example.de]
INFO[0146] Removing container [rke-bundle-cert] on host [rr03.ac.example.de], try #1
INFO[0146] Image [rancher/rke-tools:v0.1.58] exists on host [rr03.ac.example.de]
INFO[0146] Starting container [rke-log-linker] on host [rr03.ac.example.de], try #1
INFO[0146] [etcd] Successfully started [rke-log-linker] container on host [rr03.ac.example.de]
INFO[0146] Removing container [rke-log-linker] on host [rr03.ac.example.de], try #1
INFO[0147] [remove/rke-log-linker] Successfully removed container on host [rr03.ac.example.de]
INFO[0147] [etcd] Successfully started etcd plane.. Checking etcd cluster health
INFO[0147] [controlplane] Now checking status of node rr01.ac.example.de, try #1
INFO[0172] [controlplane] Now checking status of node rr02.ac.example.de, try #1
INFO[0197] [controlplane] Now checking status of node rr03.ac.example.de, try #1
INFO[0222] Attempting upgrade of controlplane components on following hosts in NotReady status: rr01.ac.example.de,rr02.ac.example.de,rr03.ac.example.de
INFO[0222] [controlplane] Building up Controller Plane..
INFO[0222] Checking if container [service-sidekick] is running on host [rr03.ac.example.de], try #1
INFO[0222] Checking if container [service-sidekick] is running on host [rr01.ac.example.de], try #1
INFO[0222] Image [rancher/rke-tools:v0.1.58] exists on host [rr03.ac.example.de]
INFO[0222] Image [rancher/rke-tools:v0.1.58] exists on host [rr01.ac.example.de]
INFO[0222] Checking if container [service-sidekick] is running on host [rr02.ac.example.de], try #1
INFO[0222] Image [rancher/rke-tools:v0.1.58] exists on host [rr02.ac.example.de]
INFO[0222] Image [rancher/hyperkube:v1.18.3-rancher2] exists on host [rr01.ac.example.de]
INFO[0222] Image [rancher/hyperkube:v1.18.3-rancher2] exists on host [rr03.ac.example.de]
INFO[0222] Starting container [kube-apiserver] on host [rr01.ac.example.de], try #1
INFO[0222] Starting container [kube-apiserver] on host [rr03.ac.example.de], try #1
INFO[0222] Image [rancher/hyperkube:v1.18.3-rancher2] exists on host [rr02.ac.example.de]
INFO[0222] [controlplane] Successfully started [kube-apiserver] container on host [rr01.ac.example.de]
INFO[0222] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [rr01.ac.example.de]
INFO[0222] [controlplane] Successfully started [kube-apiserver] container on host [rr03.ac.example.de]
INFO[0222] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [rr03.ac.example.de]
INFO[0222] Starting container [kube-apiserver] on host [rr02.ac.example.de], try #1
INFO[0223] [controlplane] Successfully started [kube-apiserver] container on host [rr02.ac.example.de]
INFO[0223] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [rr02.ac.example.de]
INFO[0234] [healthcheck] service [kube-apiserver] on host [rr03.ac.example.de] is healthy
INFO[0234] Image [rancher/rke-tools:v0.1.58] exists on host [rr03.ac.example.de]
INFO[0234] Starting container [rke-log-linker] on host [rr03.ac.example.de], try #1
INFO[0234] [healthcheck] service [kube-apiserver] on host [rr02.ac.example.de] is healthy
INFO[0234] Image [rancher/rke-tools:v0.1.58] exists on host [rr02.ac.example.de]
INFO[0234] [controlplane] Successfully started [rke-log-linker] container on host [rr03.ac.example.de]
INFO[0234] Removing container [rke-log-linker] on host [rr03.ac.example.de], try #1
INFO[0235] Starting container [rke-log-linker] on host [rr02.ac.example.de], try #1
INFO[0235] [remove/rke-log-linker] Successfully removed container on host [rr03.ac.example.de]
INFO[0235] Image [rancher/hyperkube:v1.18.3-rancher2] exists on host [rr03.ac.example.de]
INFO[0235] Starting container [kube-controller-manager] on host [rr03.ac.example.de], try #1
INFO[0235] [controlplane] Successfully started [kube-controller-manager] container on host [rr03.ac.example.de]
INFO[0235] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [rr03.ac.example.de]
INFO[0235] [healthcheck] service [kube-apiserver] on host [rr01.ac.example.de] is healthy
INFO[0235] Image [rancher/rke-tools:v0.1.58] exists on host [rr01.ac.example.de]
INFO[0235] Starting container [rke-log-linker] on host [rr01.ac.example.de], try #1
INFO[0235] [controlplane] Successfully started [rke-log-linker] container on host [rr02.ac.example.de]
INFO[0235] Removing container [rke-log-linker] on host [rr02.ac.example.de], try #1
INFO[0235] [controlplane] Successfully started [rke-log-linker] container on host [rr01.ac.example.de]
INFO[0235] Removing container [rke-log-linker] on host [rr01.ac.example.de], try #1
INFO[0235] [remove/rke-log-linker] Successfully removed container on host [rr02.ac.example.de]
INFO[0235] Image [rancher/hyperkube:v1.18.3-rancher2] exists on host [rr02.ac.example.de]
INFO[0236] Starting container [kube-controller-manager] on host [rr02.ac.example.de], try #1
INFO[0236] [remove/rke-log-linker] Successfully removed container on host [rr01.ac.example.de]
INFO[0236] Image [rancher/hyperkube:v1.18.3-rancher2] exists on host [rr01.ac.example.de]
INFO[0236] Starting container [kube-controller-manager] on host [rr01.ac.example.de], try #1
INFO[0236] [controlplane] Successfully started [kube-controller-manager] container on host [rr02.ac.example.de]
INFO[0236] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [rr02.ac.example.de]
INFO[0236] [controlplane] Successfully started [kube-controller-manager] container on host [rr01.ac.example.de]
INFO[0236] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [rr01.ac.example.de]
INFO[0240] [healthcheck] service [kube-controller-manager] on host [rr03.ac.example.de] is healthy
INFO[0240] Image [rancher/rke-tools:v0.1.58] exists on host [rr03.ac.example.de]
INFO[0240] Starting container [rke-log-linker] on host [rr03.ac.example.de], try #1
INFO[0241] [controlplane] Successfully started [rke-log-linker] container on host [rr03.ac.example.de]
INFO[0241] Removing container [rke-log-linker] on host [rr03.ac.example.de], try #1
INFO[0241] [remove/rke-log-linker] Successfully removed container on host [rr03.ac.example.de]
INFO[0241] Image [rancher/hyperkube:v1.18.3-rancher2] exists on host [rr03.ac.example.de]
INFO[0241] Starting container [kube-scheduler] on host [rr03.ac.example.de], try #1
INFO[0241] [healthcheck] service [kube-controller-manager] on host [rr02.ac.example.de] is healthy
INFO[0241] Image [rancher/rke-tools:v0.1.58] exists on host [rr02.ac.example.de]
INFO[0241] [controlplane] Successfully started [kube-scheduler] container on host [rr03.ac.example.de]
INFO[0241] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [rr03.ac.example.de]
INFO[0241] [healthcheck] service [kube-controller-manager] on host [rr01.ac.example.de] is healthy
INFO[0241] Image [rancher/rke-tools:v0.1.58] exists on host [rr01.ac.example.de]
INFO[0241] Starting container [rke-log-linker] on host [rr01.ac.example.de], try #1
INFO[0242] Starting container [rke-log-linker] on host [rr02.ac.example.de], try #1
INFO[0242] [controlplane] Successfully started [rke-log-linker] container on host [rr01.ac.example.de]
INFO[0242] Removing container [rke-log-linker] on host [rr01.ac.example.de], try #1
INFO[0242] [controlplane] Successfully started [rke-log-linker] container on host [rr02.ac.example.de]
INFO[0242] Removing container [rke-log-linker] on host [rr02.ac.example.de], try #1
INFO[0242] [remove/rke-log-linker] Successfully removed container on host [rr01.ac.example.de]
INFO[0242] Image [rancher/hyperkube:v1.18.3-rancher2] exists on host [rr01.ac.example.de]
INFO[0242] Starting container [kube-scheduler] on host [rr01.ac.example.de], try #1
INFO[0242] [controlplane] Successfully started [kube-scheduler] container on host [rr01.ac.example.de]
INFO[0242] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [rr01.ac.example.de]
INFO[0242] [remove/rke-log-linker] Successfully removed container on host [rr02.ac.example.de]
INFO[0242] Image [rancher/hyperkube:v1.18.3-rancher2] exists on host [rr02.ac.example.de]
INFO[0243] Starting container [kube-scheduler] on host [rr02.ac.example.de], try #1
INFO[0243] [controlplane] Successfully started [kube-scheduler] container on host [rr02.ac.example.de]
INFO[0243] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [rr02.ac.example.de]
INFO[0246] [healthcheck] service [kube-scheduler] on host [rr03.ac.example.de] is healthy
INFO[0246] Image [rancher/rke-tools:v0.1.58] exists on host [rr03.ac.example.de]
INFO[0247] Starting container [rke-log-linker] on host [rr03.ac.example.de], try #1
INFO[0247] [controlplane] Successfully started [rke-log-linker] container on host [rr03.ac.example.de]
INFO[0247] Removing container [rke-log-linker] on host [rr03.ac.example.de], try #1
INFO[0247] [remove/rke-log-linker] Successfully removed container on host [rr03.ac.example.de]
INFO[0248] [healthcheck] service [kube-scheduler] on host [rr01.ac.example.de] is healthy
INFO[0248] Image [rancher/rke-tools:v0.1.58] exists on host [rr01.ac.example.de]
INFO[0248] Starting container [rke-log-linker] on host [rr01.ac.example.de], try #1
INFO[0248] [healthcheck] service [kube-scheduler] on host [rr02.ac.example.de] is healthy
INFO[0248] Image [rancher/rke-tools:v0.1.58] exists on host [rr02.ac.example.de]
INFO[0248] [controlplane] Successfully started [rke-log-linker] container on host [rr01.ac.example.de]
INFO[0248] Removing container [rke-log-linker] on host [rr01.ac.example.de], try #1
INFO[0249] [remove/rke-log-linker] Successfully removed container on host [rr01.ac.example.de]
INFO[0249] Starting container [rke-log-linker] on host [rr02.ac.example.de], try #1
INFO[0249] [controlplane] Successfully started [rke-log-linker] container on host [rr02.ac.example.de]
INFO[0249] Removing container [rke-log-linker] on host [rr02.ac.example.de], try #1
INFO[0249] [remove/rke-log-linker] Successfully removed container on host [rr02.ac.example.de]
INFO[0249] [controlplane] Successfully started Controller Plane..
INFO[0249] [worker] Building up Worker Plane..
INFO[0249] Checking if container [service-sidekick] is running on host [rr01.ac.example.de], try #1
INFO[0249] Checking if container [service-sidekick] is running on host [rr02.ac.example.de], try #1
INFO[0249] Checking if container [service-sidekick] is running on host [rr03.ac.example.de], try #1
INFO[0249] [sidekick] Sidekick container already created on host [rr03.ac.example.de]
INFO[0249] [sidekick] Sidekick container already created on host [rr01.ac.example.de]
INFO[0249] [sidekick] Sidekick container already created on host [rr02.ac.example.de]
INFO[0250] Image [rancher/hyperkube:v1.18.3-rancher2] exists on host [rr03.ac.example.de]
INFO[0250] Image [rancher/hyperkube:v1.18.3-rancher2] exists on host [rr01.ac.example.de]
INFO[0250] Image [rancher/hyperkube:v1.18.3-rancher2] exists on host [rr02.ac.example.de]
INFO[0250] Starting container [kubelet] on host [rr01.ac.example.de], try #1
INFO[0250] Starting container [kubelet] on host [rr03.ac.example.de], try #1
INFO[0250] Starting container [kubelet] on host [rr02.ac.example.de], try #1
INFO[0250] [worker] Successfully started [kubelet] container on host [rr01.ac.example.de]
INFO[0250] [healthcheck] Start Healthcheck on service [kubelet] on host [rr01.ac.example.de]
INFO[0250] [worker] Successfully started [kubelet] container on host [rr03.ac.example.de]
INFO[0250] [healthcheck] Start Healthcheck on service [kubelet] on host [rr03.ac.example.de]
INFO[0250] [worker] Successfully started [kubelet] container on host [rr02.ac.example.de]
INFO[0250] [healthcheck] Start Healthcheck on service [kubelet] on host [rr02.ac.example.de]
INFO[0260] [healthcheck] service [kubelet] on host [rr03.ac.example.de] is healthy
INFO[0260] Image [rancher/rke-tools:v0.1.58] exists on host [rr03.ac.example.de]
INFO[0260] [healthcheck] service [kubelet] on host [rr01.ac.example.de] is healthy
INFO[0260] Image [rancher/rke-tools:v0.1.58] exists on host [rr01.ac.example.de]
INFO[0260] Starting container [rke-log-linker] on host [rr03.ac.example.de], try #1
INFO[0260] Starting container [rke-log-linker] on host [rr01.ac.example.de], try #1
INFO[0260] [healthcheck] service [kubelet] on host [rr02.ac.example.de] is healthy
INFO[0260] Image [rancher/rke-tools:v0.1.58] exists on host [rr02.ac.example.de]
INFO[0261] [worker] Successfully started [rke-log-linker] container on host [rr03.ac.example.de]
INFO[0261] Removing container [rke-log-linker] on host [rr03.ac.example.de], try #1
INFO[0261] Starting container [rke-log-linker] on host [rr02.ac.example.de], try #1
INFO[0261] [worker] Successfully started [rke-log-linker] container on host [rr01.ac.example.de]
INFO[0261] Removing container [rke-log-linker] on host [rr01.ac.example.de], try #1
INFO[0261] [remove/rke-log-linker] Successfully removed container on host [rr03.ac.example.de]
INFO[0261] Image [rancher/hyperkube:v1.18.3-rancher2] exists on host [rr03.ac.example.de]
INFO[0261] Starting container [kube-proxy] on host [rr03.ac.example.de], try #1
INFO[0261] [remove/rke-log-linker] Successfully removed container on host [rr01.ac.example.de]
INFO[0261] Image [rancher/hyperkube:v1.18.3-rancher2] exists on host [rr01.ac.example.de]
INFO[0261] Starting container [kube-proxy] on host [rr01.ac.example.de], try #1
INFO[0261] [worker] Successfully started [kube-proxy] container on host [rr03.ac.example.de]
INFO[0261] [healthcheck] Start Healthcheck on service [kube-proxy] on host [rr03.ac.example.de]
INFO[0261] [worker] Successfully started [rke-log-linker] container on host [rr02.ac.example.de]
INFO[0261] Removing container [rke-log-linker] on host [rr02.ac.example.de], try #1
INFO[0261] [worker] Successfully started [kube-proxy] container on host [rr01.ac.example.de]
INFO[0261] [healthcheck] Start Healthcheck on service [kube-proxy] on host [rr01.ac.example.de]
INFO[0262] [remove/rke-log-linker] Successfully removed container on host [rr02.ac.example.de]
INFO[0262] Image [rancher/hyperkube:v1.18.3-rancher2] exists on host [rr02.ac.example.de]
INFO[0262] Starting container [kube-proxy] on host [rr02.ac.example.de], try #1
INFO[0262] [worker] Successfully started [kube-proxy] container on host [rr02.ac.example.de]
INFO[0262] [healthcheck] Start Healthcheck on service [kube-proxy] on host [rr02.ac.example.de]
INFO[0297] [healthcheck] service [kube-proxy] on host [rr03.ac.example.de] is healthy
INFO[0297] Image [rancher/rke-tools:v0.1.58] exists on host [rr03.ac.example.de]
INFO[0297] Starting container [rke-log-linker] on host [rr03.ac.example.de], try #1
INFO[0297] [healthcheck] service [kube-proxy] on host [rr01.ac.example.de] is healthy
INFO[0297] Image [rancher/rke-tools:v0.1.58] exists on host [rr01.ac.example.de]
INFO[0298] Starting container [rke-log-linker] on host [rr01.ac.example.de], try #1
INFO[0298] [worker] Successfully started [rke-log-linker] container on host [rr03.ac.example.de]
INFO[0298] Removing container [rke-log-linker] on host [rr03.ac.example.de], try #1
INFO[0298] [worker] Successfully started [rke-log-linker] container on host [rr01.ac.example.de]
INFO[0298] Removing container [rke-log-linker] on host [rr01.ac.example.de], try #1
INFO[0298] [remove/rke-log-linker] Successfully removed container on host [rr03.ac.example.de]
INFO[0298] [healthcheck] service [kube-proxy] on host [rr02.ac.example.de] is healthy
INFO[0298] Image [rancher/rke-tools:v0.1.58] exists on host [rr02.ac.example.de]
INFO[0298] [remove/rke-log-linker] Successfully removed container on host [rr01.ac.example.de]
INFO[0299] Starting container [rke-log-linker] on host [rr02.ac.example.de], try #1
INFO[0299] [worker] Successfully started [rke-log-linker] container on host [rr02.ac.example.de]
INFO[0299] Removing container [rke-log-linker] on host [rr02.ac.example.de], try #1
INFO[0299] [remove/rke-log-linker] Successfully removed container on host [rr02.ac.example.de]
INFO[0299] [worker] Successfully started Worker Plane..
INFO[0299] [controlplane] Now checking status of node rr01.ac.example.de, try #1
ERRO[0325] Host rr01.ac.example.de failed to report Ready status with error: [controlplane] Error getting node rr01.ac.example.de: "rr01.ac.example.de" not found
INFO[0325] [controlplane] Now checking status of node rr02.ac.example.de, try #1
ERRO[0350] Host rr02.ac.example.de failed to report Ready status with error: [controlplane] Error getting node rr02.ac.example.de: "rr02.ac.example.de" not found
INFO[0350] [controlplane] Now checking status of node rr03.ac.example.de, try #1
ERRO[0375] Host rr03.ac.example.de failed to report Ready status with error: [controlplane] Error getting node rr03.ac.example.de: "rr03.ac.example.de" not found
INFO[0375] [controlplane] Processing controlplane hosts for upgrade 1 at a time
INFO[0375] Processing controlplane host rr01.ac.example.de
INFO[0375] [controlplane] Now checking status of node rr01.ac.example.de, try #1
ERRO[0400] Failed to upgrade hosts: rr01.ac.example.de with error [[controlplane] Error getting node rr01.ac.example.de: "rr01.ac.example.de" not found]
FATA[0400] [controlPlane] Failed to upgrade Control Plane: [[[controlplane] Error getting node rr01.ac.example.de: "rr01.ac.example.de" not found]]
I had added these host names to /etc/hosts on the nodes, they can be resolved via DNS, too, so I wonder WTH?
rancher.yml is
ignore_docker_version: false
ssh_key_path: ~/.ssh/id_ecdsa
network:
# plugin: canal
plugin: flannel
# flannel on lxc has problems
nodes:
- address: rr01.ac.example.com
user: debian
role: [controlplane,worker,etcd]
- address: rr02.ac.example.com
user: debian
role: [controlplane,worker,etcd]
- address: rr03.ac.example.com
user: debian
role: [controlplane,worker,etcd]
services:
etcd:
snapshot: true
creation: 6h
retention: 24h
Every helpful comment is highly appreciated
Harri