Hey, I am following rancher course “Certified Rancher Operator: Level 1” and I am at step 1.3.5, I’ve managed to create a single node cluster but when I try to add two more clusters I just get an error message that says
[etcd] failed to check health for etcd host [10.10.10.6]: failed to get /health for host [10.10.10.6]: Get “-ht-tps://10.10.10.6:2379/health”: net/http: TLS handshake timeout
I should mention that I am using VMware, I have 3 ubuntu nodes, using rke from my own macOS laptop, the ip addresses are all set up, my nodes and host can ping and ssh into each other without any problem.
I opened all the necessary ports as well on all of my nodes
sudo ufw status
Status: active
To Action From
22 ALLOW Anywhere
64 ALLOW Anywhere
6443 ALLOW Anywhere
22/tcp ALLOW Anywhere
Anywhere on weave ALLOW 10.32.0.0/12
6783/udp ALLOW Anywhere
6784/udp ALLOW Anywhere
6783/tcp ALLOW Anywhere
2379 ALLOW Anywhere
2380 ALLOW Anywhere
22 (v6) ALLOW Anywhere (v6)
64 (v6) ALLOW Anywhere (v6)
6443 (v6) ALLOW Anywhere (v6)
22/tcp (v6) ALLOW Anywhere (v6)
6783/udp (v6) ALLOW Anywhere (v6)
6784/udp (v6) ALLOW Anywhere (v6)
6783/tcp (v6) ALLOW Anywhere (v6)
2379 (v6) ALLOW Anywhere (v6)
2380 (v6) ALLOW Anywhere (v6)
10.32.0.0/12 ALLOW OUT Anywhere on weave
and here is my log from when I try to create the cluster,
DEBU[0000] Loglevel set to [debug]
INFO[0000] Running RKE version: v1.2.8
DEBU[0000] audit log policy found in cluster.yml
INFO[0000] Initiating Kubernetes cluster
DEBU[0000] metadataInitialized: [False] []
DEBU[0000] Loading data.json from local source
DEBU[0000] data.json SHA256 checksum: bc3a458d02c4b5a658a894140e1745ba2d90a5518854aa8c8f75a2f43a8fbed7
DEBU[0000] metadata initialized successfully
DEBU[0000] metadataInitialized: [true] []
DEBU[0000] No DNS provider configured, setting default based on cluster version [1.20.6-rancher1-1]
DEBU[0000] DNS provider set to [coredns]
DEBU[0000] Checking if cluster version [1.20.6-rancher1-1] needs to have kube-api audit log enabled
DEBU[0000] Cluster version [1.20.6-rancher1-1] needs to have kube-api audit log enabled
DEBU[0000] Enabling kube-api audit log for cluster version [v1.20.6-rancher1-1]
DEBU[0000] No input provided for maxUnavailableWorker, setting it to default value of 10 percent
DEBU[0000] No input provided for maxUnavailableControlplane, setting it to default value of 1
DEBU[0000] Host: 10.10.10.6 has role: controlplane
DEBU[0000] Host: 10.10.10.6 has role: worker
DEBU[0000] Host: 10.10.10.6 has role: etcd
DEBU[0000] Host: 10.10.10.4 has role: controlplane
DEBU[0000] Host: 10.10.10.4 has role: worker
DEBU[0000] Host: 10.10.10.4 has role: etcd
DEBU[0000] Host: 10.10.10.5 has role: controlplane
DEBU[0000] Host: 10.10.10.5 has role: worker
DEBU[0000] Host: 10.10.10.5 has role: etcd
DEBU[0000] [state] previous state found, this is not a legacy cluster
INFO[0000] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates
INFO[0000] [certificates] Generating admin certificates and kubeconfig
INFO[0000] Successfully Deployed state file at [./cluster.rkestate]
DEBU[0000] Checking if cluster version [1.20.6-rancher1-1] needs to have kube-api audit log enabled
DEBU[0000] Cluster version [1.20.6-rancher1-1] needs to have kube-api audit log enabled
DEBU[0000] Enabling kube-api audit log for cluster version [v1.20.6-rancher1-1]
DEBU[0000] Host: 10.10.10.6 has role: controlplane
DEBU[0000] Host: 10.10.10.6 has role: worker
DEBU[0000] Host: 10.10.10.6 has role: etcd
DEBU[0000] Host: 10.10.10.4 has role: controlplane
DEBU[0000] Host: 10.10.10.4 has role: worker
DEBU[0000] Host: 10.10.10.4 has role: etcd
DEBU[0000] Host: 10.10.10.5 has role: controlplane
DEBU[0000] Host: 10.10.10.5 has role: worker
DEBU[0000] Host: 10.10.10.5 has role: etcd
INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [10.10.10.4]
INFO[0000] [dialer] Setup tunnel for host [10.10.10.5]
INFO[0000] [dialer] Setup tunnel for host [10.10.10.6]
DEBU[0000] Connecting to Docker API for host [10.10.10.6]
DEBU[0000] Connecting to Docker API for host [10.10.10.4]
DEBU[0000] Connecting to Docker API for host [10.10.10.5]
DEBU[0000] Docker Info found for host [10.10.10.5]: types.Info{ID:“TN3K:CHPK:MOIH:IOGY:I5YH:55ZZ:6QD5:76IY:5N4V:SSVY:2NU6:WTV3”, Containers:3, ContainersRunning:2, ContainersPaused:0, ContainersStopped:1, Images:3, Driver:“overlay2”, DriverStatus:[][2]string{[2]string{“Backing Filesystem”, “extfs”}, [2]string{“Supports d_type”, “true”}, [2]string{“Native Overlay Diff”, “true”}}, SystemStatus:[][2]string(nil), Plugins:types.PluginsInfo{Volume:[]string{“local”}, Network:[]string{“bridge”, “host”, “ipvlan”, “macvlan”, “null”, “overlay”}, Authorization:[]string(nil), Log:[]string{“awslogs”, “fluentd”, “gcplogs”, “gelf”, “journald”, “json-file”, “local”, “logentries”, “splunk”, “syslog”}}, MemoryLimit:true, SwapLimit:true, KernelMemory:true, KernelMemoryTCP:true, CPUCfsPeriod:true, CPUCfsQuota:true, CPUShares:true, CPUSet:true, PidsLimit:true, IPv4Forwarding:true, BridgeNfIptables:true, BridgeNfIP6tables:true, Debug:false, NFd:33, OomKillDisable:true, NGoroutines:46, SystemTime:“2021-05-13T12:32:35.907278356-07:00”, LoggingDriver:“json-file”, CgroupDriver:“cgroupfs”, NEventsListener:0, KernelVersion:“5.8.0-50-generic”, OperatingSystem:“Ubuntu 20.04.2 LTS”, OSVersion:"", OSType:“linux”, Architecture:“x86_64”, IndexServerAddress:“ht-tps://index.docker.io/v1/”, RegistryConfig:(*registry.ServiceConfig)(0x140002de0e0), NCPU:2, MemTotal:4096712704, GenericResources:[]swarm.GenericResource(nil), DockerRootDir:"/var/lib/docker", HT-TPProxy:"", HTT-PSProxy:"", NoProxy:"", Name:“ubuntu”, Labels:[]string{}, ExperimentalBuild:false, ServerVersion:“19.03.15”, ClusterStore:"", ClusterAdvertise:"", Runtimes:map[string]types.Runtime{“runc”:types.Runtime{Path:“runc”, Args:[]string(nil)}}, DefaultRuntime:“runc”, Swarm:swarm.Info{NodeID:"", NodeAddr:"", LocalNodeState:“inactive”, ControlAvailable:false, Error:"", RemoteManagers:[]swarm.Peer(nil), Nodes:0, Managers:0, Cluster:(*swarm.ClusterInfo)(nil), Warnings:[]string(nil)}, LiveRestoreEnabled:false, Isolation:"", InitBinary:“docker-init”, ContainerdCommit:types.Commit{ID:“05f951a3781f4f2c1911b05e61c160e9c30eaa8e”, Expected:“05f951a3781f4f2c1911b05e61c160e9c30eaa8e”}, RuncCommit:types.Commit{ID:“12644e614e25b05da6fd08a38ffa0cfe1903fdec”, Expected:“12644e614e25b05da6fd08a38ffa0cfe1903fdec”}, InitCommit:types.Commit{ID:“fec3683”, Expected:“fec3683”}, SecurityOptions:[]string{“apparmor”, “seccomp”}, ProductLicense:"", Warnings:[]string(nil)}
DEBU[0000] Docker Info found for host [10.10.10.4]: types.Info{ID:“IYCH:GOCY:EJYR:WXKM:PL56:GSDD:WMM7:AUB4:YCN3:OVSI:L4BU:PKMX”, Containers:3, ContainersRunning:2, ContainersPaused:0, ContainersStopped:1, Images:3, Driver:“overlay2”, DriverStatus:[][2]string{[2]string{“Backing Filesystem”, “extfs”}, [2]string{“Supports d_type”, “true”}, [2]string{“Native Overlay Diff”, “true”}}, SystemStatus:[][2]string(nil), Plugins:types.PluginsInfo{Volume:[]string{“local”}, Network:[]string{“bridge”, “host”, “ipvlan”, “macvlan”, “null”, “overlay”}, Authorization:[]string(nil), Log:[]string{“awslogs”, “fluentd”, “gcplogs”, “gelf”, “journald”, “json-file”, “local”, “logentries”, “splunk”, “syslog”}}, MemoryLimit:true, SwapLimit:true, KernelMemory:true, KernelMemoryTCP:true, CPUCfsPeriod:true, CPUCfsQuota:true, CPUShares:true, CPUSet:true, PidsLimit:true, IPv4Forwarding:true, BridgeNfIptables:true, BridgeNfIP6tables:true, Debug:false, NFd:33, OomKillDisable:true, NGoroutines:46, SystemTime:“2021-05-13T12:32:35.915590091-07:00”, LoggingDriver:“json-file”, CgroupDriver:“cgroupfs”, NEventsListener:0, KernelVersion:“5.8.0-50-generic”, OperatingSystem:“Ubuntu 20.04.2 LTS”, OSVersion:"", OSType:“linux”, Architecture:“x86_64”, IndexServerAddress:“ht–tps://index.docker.io/v1/”, RegistryConfig:(*registry.ServiceConfig)(0x140002de2a0), NCPU:2, MemTotal:4096712704, GenericResources:[]swarm.GenericResource(nil), DockerRootDir:"/var/lib/docker", HTTPProxy:"", HTTP-SProxy:"", NoProxy:"", Name:“ubuntu”, Labels:[]string{}, ExperimentalBuild:false, ServerVersion:“19.03.15”, ClusterStore:"", ClusterAdvertise:"", Runtimes:map[string]types.Runtime{“runc”:types.Runtime{Path:“runc”, Args:[]string(nil)}}, DefaultRuntime:“runc”, Swarm:swarm.Info{NodeID:"", NodeAddr:"", LocalNodeState:“inactive”, ControlAvailable:false, Error:"", RemoteManagers:[]swarm.Peer(nil), Nodes:0, Managers:0, Cluster:(*swarm.ClusterInfo)(nil), Warnings:[]string(nil)}, LiveRestoreEnabled:false, Isolation:"", InitBinary:“docker-init”, ContainerdCommit:types.Commit{ID:“05f951a3781f4f2c1911b05e61c160e9c30eaa8e”, Expected:“05f951a3781f4f2c1911b05e61c160e9c30eaa8e”}, RuncCommit:types.Commit{ID:“12644e614e25b05da6fd08a38ffa0cfe1903fdec”, Expected:“12644e614e25b05da6fd08a38ffa0cfe1903fdec”}, InitCommit:types.Commit{ID:“fec3683”, Expected:“fec3683”}, SecurityOptions:[]string{“apparmor”, “seccomp”}, ProductLicense:"", Warnings:[]string(nil)}
DEBU[0000] Docker Info found for host [10.10.10.6]: types.Info{ID:“4HZV:5RMB:PLXD:D3BR:3DS7:J3TM:6ZMK:GBIE:6IWC:TUCG:J45D:IPYM”, Containers:3, ContainersRunning:2, ContainersPaused:0, ContainersStopped:1, Images:3, Driver:“overlay2”, DriverStatus:[][2]string{[2]string{“Backing Filesystem”, “extfs”}, [2]string{“Supports d_type”, “true”}, [2]string{“Native Overlay Diff”, “true”}, [2]string{“userxattr”, “false”}}, SystemStatus:[][2]string(nil), Plugins:types.PluginsInfo{Volume:[]string{“local”}, Network:[]string{“bridge”, “host”, “ipvlan”, “macvlan”, “null”, “overlay”}, Authorization:[]string(nil), Log:[]string{“awslogs”, “fluentd”, “gcplogs”, “gelf”, “journald”, “json-file”, “local”, “logentries”, “splunk”, “syslog”}}, MemoryLimit:true, SwapLimit:true, KernelMemory:true, KernelMemoryTCP:true, CPUCfsPeriod:true, CPUCfsQuota:true, CPUShares:true, CPUSet:true, PidsLimit:true, IPv4Forwarding:true, BridgeNfIptables:true, BridgeNfIP6tables:true, Debug:false, NFd:34, OomKillDisable:true, NGoroutines:45, SystemTime:“2021-05-13T12:32:35.925137865-07:00”, LoggingDriver:“json-file”, CgroupDriver:“cgroupfs”, NEventsListener:0, KernelVersion:“5.8.0-50-generic”, OperatingSystem:“Ubuntu 20.04.2 LTS”, OSVersion:“20.04”, OSType:“linux”, Architecture:“x86_64”, IndexServerAddress:“http-s://index.docker.io/v1/”, RegistryConfig:(*registry.ServiceConfig)(0x140002de460), NCPU:2, MemTotal:4096712704, GenericResources:[]swarm.GenericResource(nil), DockerRootDir:"/var/lib/docker", HTTPProxy:"", H-TTPSProxy:"", NoProxy:"", Name:“ubuntu”, Labels:[]string{}, ExperimentalBuild:false, ServerVersion:“20.10.6”, ClusterStore:"", ClusterAdvertise:"", Runtimes:map[string]types.Runtime{“io.containerd.runc.v2”:types.Runtime{Path:“runc”, Args:[]string(nil)}, “io.containerd.runtime.v1.linux”:types.Runtime{Path:“runc”, Args:[]string(nil)}, “runc”:types.Runtime{Path:“runc”, Args:[]string(nil)}}, DefaultRuntime:“runc”, Swarm:swarm.Info{NodeID:"", NodeAddr:"", LocalNodeState:“inactive”, ControlAvailable:false, Error:"", RemoteManagers:[]swarm.Peer(nil), Nodes:0, Managers:0, Cluster:(*swarm.ClusterInfo)(nil), Warnings:[]string(nil)}, LiveRestoreEnabled:false, Isolation:"", InitBinary:“docker-init”, ContainerdCommit:types.Commit{ID:“05f951a3781f4f2c1911b05e61c160e9c30eaa8e”, Expected:“05f951a3781f4f2c1911b05e61c160e9c30eaa8e”}, RuncCommit:types.Commit{ID:“12644e614e25b05da6fd08a38ffa0cfe1903fdec”, Expected:“12644e614e25b05da6fd08a38ffa0cfe1903fdec”}, InitCommit:types.Commit{ID:“de40ad0”, Expected:“de40ad0”}, SecurityOptions:[]string{“apparmor”, “seccomp”}, ProductLicense:"", Warnings:[]string(nil)}
INFO[0000] [network] Deploying port listener containers
DEBU[0000] [network] Starting deployListener [rke-etcd-port-listener] on host [10.10.10.5]
DEBU[0000] [network] Starting deployListener [rke-etcd-port-listener] on host [10.10.10.6]
DEBU[0000] [network] Starting deployListener [rke-etcd-port-listener] on host [10.10.10.4]
DEBU[0000] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4], try #1
DEBU[0000] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5], try #1
DEBU[0000] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6], try #1
INFO[0000] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5]
INFO[0000] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4]
INFO[0000] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6]
INFO[0000] Starting container [rke-etcd-port-listener] on host [10.10.10.5], try #1
INFO[0000] Starting container [rke-etcd-port-listener] on host [10.10.10.4], try #1
INFO[0000] Starting container [rke-etcd-port-listener] on host [10.10.10.6], try #1
DEBU[0001] [network] Service is already up on host [10.10.10.6]
DEBU[0001] [network] Service is already up on host [10.10.10.4]
DEBU[0001] [network] Service is already up on host [10.10.10.5]
DEBU[0001] [network] Starting deployListener [rke-cp-port-listener] on host [10.10.10.6]
DEBU[0001] [network] Starting deployListener [rke-cp-port-listener] on host [10.10.10.5]
DEBU[0001] [network] Starting deployListener [rke-cp-port-listener] on host [10.10.10.4]
DEBU[0001] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6], try #1
DEBU[0001] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4], try #1
DEBU[0001] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5], try #1
INFO[0001] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6]
INFO[0001] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4]
INFO[0001] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5]
INFO[0001] Starting container [rke-cp-port-listener] on host [10.10.10.4], try #1
INFO[0006] Starting container [rke-cp-port-listener] on host [10.10.10.6], try #1
INFO[0006] Starting container [rke-cp-port-listener] on host [10.10.10.5], try #1
INFO[0006] [network] Successfully started [rke-cp-port-listener] container on host [10.10.10.4]
INFO[0007] [network] Successfully started [rke-cp-port-listener] container on host [10.10.10.5]
INFO[0007] [network] Successfully started [rke-cp-port-listener] container on host [10.10.10.6]
DEBU[0007] [network] Starting deployListener [rke-worker-port-listener] on host [10.10.10.6]
DEBU[0007] [network] Starting deployListener [rke-worker-port-listener] on host [10.10.10.5]
DEBU[0007] [network] Starting deployListener [rke-worker-port-listener] on host [10.10.10.4]
DEBU[0007] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6], try #1
DEBU[0007] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5], try #1
DEBU[0007] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4], try #1
INFO[0007] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6]
INFO[0007] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4]
INFO[0007] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5]
INFO[0007] Starting container [rke-worker-port-listener] on host [10.10.10.6], try #1
INFO[0007] Starting container [rke-worker-port-listener] on host [10.10.10.4], try #1
INFO[0007] Starting container [rke-worker-port-listener] on host [10.10.10.5], try #1
INFO[0013] [network] Successfully started [rke-worker-port-listener] container on host [10.10.10.5]
INFO[0013] [network] Successfully started [rke-worker-port-listener] container on host [10.10.10.4]
INFO[0014] [network] Successfully started [rke-worker-port-listener] container on host [10.10.10.6]
INFO[0014] [network] Port listener containers deployed successfully
INFO[0014] [network] Running etcd <-> etcd port checks
INFO[0014] [network] Checking if host [10.10.10.4] can connect to host(s) [10.10.10.6 10.10.10.4 10.10.10.5] on port(s) [2379 2380], try #1
DEBU[0014] [remove/rke-port-checker] Checking if container is running on host [10.10.10.4]
INFO[0014] [network] Checking if host [10.10.10.5] can connect to host(s) [10.10.10.6 10.10.10.4 10.10.10.5] on port(s) [2379 2380], try #1
DEBU[0014] [remove/rke-port-checker] Checking if container is running on host [10.10.10.5]
INFO[0014] [network] Checking if host [10.10.10.6] can connect to host(s) [10.10.10.6 10.10.10.4 10.10.10.5] on port(s) [2379 2380], try #1
DEBU[0014] [remove/rke-port-checker] Checking if container is running on host [10.10.10.6]
DEBU[0014] [remove/rke-port-checker] Container doesn’t exist on host [10.10.10.4]
DEBU[0014] [remove/rke-port-checker] Container doesn’t exist on host [10.10.10.5]
DEBU[0014] [remove/rke-port-checker] Container doesn’t exist on host [10.10.10.6]
DEBU[0014] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4], try #1
DEBU[0014] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5], try #1
DEBU[0014] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6], try #1
INFO[0014] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4]
INFO[0014] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5]
INFO[0014] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6]
INFO[0014] Starting container [rke-port-checker] on host [10.10.10.4], try #1
INFO[0015] Starting container [rke-port-checker] on host [10.10.10.5], try #1
INFO[0015] Starting container [rke-port-checker] on host [10.10.10.6], try #1
INFO[0015] [network] Successfully started [rke-port-checker] container on host [10.10.10.4]
INFO[0015] [network] Successfully started [rke-port-checker] container on host [10.10.10.5]
DEBU[0015] [network] containerLog [] on host: 10.10.10.4
INFO[0015] Removing container [rke-port-checker] on host [10.10.10.4], try #1
DEBU[0015] [network] Length of containerLog is [0] on host: 10.10.10.4
DEBU[0015] [network] containerLog [] on host: 10.10.10.5
INFO[0015] Removing container [rke-port-checker] on host [10.10.10.5], try #1
INFO[0015] [network] Successfully started [rke-port-checker] container on host [10.10.10.6]
DEBU[0015] [network] containerLog [] on host: 10.10.10.6
INFO[0015] Removing container [rke-port-checker] on host [10.10.10.6], try #1
DEBU[0015] [network] Length of containerLog is [0] on host: 10.10.10.5
DEBU[0015] [network] Length of containerLog is [0] on host: 10.10.10.6
INFO[0015] [network] Running control plane → etcd port checks
INFO[0015] [network] Checking if host [10.10.10.6] can connect to host(s) [10.10.10.6 10.10.10.4 10.10.10.5] on port(s) [2379], try #1
DEBU[0015] [remove/rke-port-checker] Checking if container is running on host [10.10.10.6]
INFO[0015] [network] Checking if host [10.10.10.4] can connect to host(s) [10.10.10.6 10.10.10.4 10.10.10.5] on port(s) [2379], try #1
DEBU[0015] [remove/rke-port-checker] Checking if container is running on host [10.10.10.4]
INFO[0015] [network] Checking if host [10.10.10.5] can connect to host(s) [10.10.10.6 10.10.10.4 10.10.10.5] on port(s) [2379], try #1
DEBU[0015] [remove/rke-port-checker] Checking if container is running on host [10.10.10.5]
DEBU[0015] [remove/rke-port-checker] Container doesn’t exist on host [10.10.10.5]
DEBU[0015] [remove/rke-port-checker] Container doesn’t exist on host [10.10.10.4]
DEBU[0015] [remove/rke-port-checker] Container doesn’t exist on host [10.10.10.6]
DEBU[0015] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6], try #1
DEBU[0015] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4], try #1
DEBU[0015] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5], try #1
INFO[0015] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6]
INFO[0015] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4]
INFO[0015] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5]
INFO[0015] Starting container [rke-port-checker] on host [10.10.10.6], try #1
INFO[0015] Starting container [rke-port-checker] on host [10.10.10.4], try #1
INFO[0015] Starting container [rke-port-checker] on host [10.10.10.5], try #1
INFO[0015] [network] Successfully started [rke-port-checker] container on host [10.10.10.4]
INFO[0015] [network] Successfully started [rke-port-checker] container on host [10.10.10.5]
DEBU[0016] [network] containerLog [] on host: 10.10.10.4
INFO[0016] Removing container [rke-port-checker] on host [10.10.10.4], try #1
DEBU[0016] [network] containerLog [] on host: 10.10.10.5
INFO[0016] Removing container [rke-port-checker] on host [10.10.10.5], try #1
DEBU[0016] [network] Length of containerLog is [0] on host: 10.10.10.5
DEBU[0016] [network] Length of containerLog is [0] on host: 10.10.10.4
INFO[0017] [network] Successfully started [rke-port-checker] container on host [10.10.10.6]
DEBU[0017] [network] containerLog [] on host: 10.10.10.6
INFO[0017] Removing container [rke-port-checker] on host [10.10.10.6], try #1
DEBU[0017] [network] Length of containerLog is [0] on host: 10.10.10.6
INFO[0017] [network] Running control plane → worker port checks
INFO[0017] [network] Checking if host [10.10.10.6] can connect to host(s) [10.10.10.6 10.10.10.4 10.10.10.5] on port(s) [10250], try #1
DEBU[0017] [remove/rke-port-checker] Checking if container is running on host [10.10.10.6]
INFO[0017] [network] Checking if host [10.10.10.4] can connect to host(s) [10.10.10.6 10.10.10.4 10.10.10.5] on port(s) [10250], try #1
DEBU[0017] [remove/rke-port-checker] Checking if container is running on host [10.10.10.4]
INFO[0017] [network] Checking if host [10.10.10.5] can connect to host(s) [10.10.10.6 10.10.10.4 10.10.10.5] on port(s) [10250], try #1
DEBU[0017] [remove/rke-port-checker] Checking if container is running on host [10.10.10.5]
DEBU[0017] [remove/rke-port-checker] Container doesn’t exist on host [10.10.10.4]
DEBU[0017] [remove/rke-port-checker] Container doesn’t exist on host [10.10.10.6]
DEBU[0017] [remove/rke-port-checker] Container doesn’t exist on host [10.10.10.5]
DEBU[0017] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4], try #1
DEBU[0017] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6], try #1
DEBU[0017] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5], try #1
INFO[0017] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4]
INFO[0017] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6]
INFO[0017] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5]
INFO[0018] Starting container [rke-port-checker] on host [10.10.10.5], try #1
INFO[0018] Starting container [rke-port-checker] on host [10.10.10.4], try #1
INFO[0018] Starting container [rke-port-checker] on host [10.10.10.6], try #1
INFO[0018] [network] Successfully started [rke-port-checker] container on host [10.10.10.5]
INFO[0018] [network] Successfully started [rke-port-checker] container on host [10.10.10.6]
INFO[0018] [network] Successfully started [rke-port-checker] container on host [10.10.10.4]
DEBU[0018] [network] containerLog [] on host: 10.10.10.5
INFO[0018] Removing container [rke-port-checker] on host [10.10.10.5], try #1
DEBU[0018] [network] containerLog [] on host: 10.10.10.6
INFO[0018] Removing container [rke-port-checker] on host [10.10.10.6], try #1
DEBU[0018] [network] Length of containerLog is [0] on host: 10.10.10.6
DEBU[0018] [network] Length of containerLog is [0] on host: 10.10.10.5
DEBU[0018] [network] containerLog [] on host: 10.10.10.4
INFO[0018] Removing container [rke-port-checker] on host [10.10.10.4], try #1
DEBU[0018] [network] Length of containerLog is [0] on host: 10.10.10.4
INFO[0018] [network] Running workers → control plane port checks
INFO[0018] [network] Checking if host [10.10.10.6] can connect to host(s) [10.10.10.6 10.10.10.4 10.10.10.5] on port(s) [6443], try #1
DEBU[0018] [remove/rke-port-checker] Checking if container is running on host [10.10.10.6]
INFO[0018] [network] Checking if host [10.10.10.4] can connect to host(s) [10.10.10.6 10.10.10.4 10.10.10.5] on port(s) [6443], try #1
DEBU[0018] [remove/rke-port-checker] Checking if container is running on host [10.10.10.4]
INFO[0018] [network] Checking if host [10.10.10.5] can connect to host(s) [10.10.10.6 10.10.10.4 10.10.10.5] on port(s) [6443], try #1
DEBU[0018] [remove/rke-port-checker] Checking if container is running on host [10.10.10.5]
DEBU[0018] [remove/rke-port-checker] Container doesn’t exist on host [10.10.10.5]
DEBU[0018] [remove/rke-port-checker] Container doesn’t exist on host [10.10.10.4]
DEBU[0018] [remove/rke-port-checker] Container doesn’t exist on host [10.10.10.6]
DEBU[0018] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5], try #1
DEBU[0018] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6], try #1
DEBU[0018] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4], try #1
INFO[0018] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4]
INFO[0018] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6]
INFO[0018] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5]
INFO[0018] Starting container [rke-port-checker] on host [10.10.10.4], try #1
INFO[0018] Starting container [rke-port-checker] on host [10.10.10.5], try #1
INFO[0018] Starting container [rke-port-checker] on host [10.10.10.6], try #1
INFO[0018] [network] Successfully started [rke-port-checker] container on host [10.10.10.5]
INFO[0018] [network] Successfully started [rke-port-checker] container on host [10.10.10.4]
DEBU[0018] [network] containerLog [] on host: 10.10.10.4
INFO[0018] Removing container [rke-port-checker] on host [10.10.10.4], try #1
DEBU[0018] [network] containerLog [] on host: 10.10.10.5
INFO[0018] Removing container [rke-port-checker] on host [10.10.10.5], try #1
DEBU[0018] [network] Length of containerLog is [0] on host: 10.10.10.4
DEBU[0018] [network] Length of containerLog is [0] on host: 10.10.10.5
INFO[0018] [network] Successfully started [rke-port-checker] container on host [10.10.10.6]
DEBU[0019] [network] containerLog [] on host: 10.10.10.6
INFO[0019] Removing container [rke-port-checker] on host [10.10.10.6], try #1
DEBU[0019] [network] Length of containerLog is [0] on host: 10.10.10.6
INFO[0019] [network] Checking KubeAPI port Control Plane hosts
DEBU[0019] [network] Checking KubeAPI port [6443] on host: 10.10.10.6
DEBU[0019] [network] Checking KubeAPI port [6443] on host: 10.10.10.4
DEBU[0019] [network] Checking KubeAPI port [6443] on host: 10.10.10.5
INFO[0019] [network] Removing port listener containers
DEBU[0019] [remove/rke-etcd-port-listener] Checking if container is running on host [10.10.10.6]
DEBU[0019] [remove/rke-etcd-port-listener] Checking if container is running on host [10.10.10.5]
DEBU[0019] [remove/rke-etcd-port-listener] Checking if container is running on host [10.10.10.4]
DEBU[0019] [remove/rke-etcd-port-listener] Removing container on host [10.10.10.6]
INFO[0019] Removing container [rke-etcd-port-listener] on host [10.10.10.6], try #1
DEBU[0019] [remove/rke-etcd-port-listener] Removing container on host [10.10.10.4]
INFO[0019] Removing container [rke-etcd-port-listener] on host [10.10.10.4], try #1
DEBU[0019] [remove/rke-etcd-port-listener] Removing container on host [10.10.10.5]
INFO[0019] Removing container [rke-etcd-port-listener] on host [10.10.10.5], try #1
INFO[0019] [remove/rke-etcd-port-listener] Successfully removed container on host [10.10.10.6]
INFO[0019] [remove/rke-etcd-port-listener] Successfully removed container on host [10.10.10.4]
INFO[0019] [remove/rke-etcd-port-listener] Successfully removed container on host [10.10.10.5]
DEBU[0019] [remove/rke-cp-port-listener] Checking if container is running on host [10.10.10.6]
DEBU[0019] [remove/rke-cp-port-listener] Checking if container is running on host [10.10.10.5]
DEBU[0019] [remove/rke-cp-port-listener] Checking if container is running on host [10.10.10.4]
DEBU[0019] [remove/rke-cp-port-listener] Removing container on host [10.10.10.4]
INFO[0019] Removing container [rke-cp-port-listener] on host [10.10.10.4], try #1
DEBU[0019] [remove/rke-cp-port-listener] Removing container on host [10.10.10.6]
INFO[0019] Removing container [rke-cp-port-listener] on host [10.10.10.6], try #1
DEBU[0019] [remove/rke-cp-port-listener] Removing container on host [10.10.10.5]
INFO[0019] Removing container [rke-cp-port-listener] on host [10.10.10.5], try #1
INFO[0019] [remove/rke-cp-port-listener] Successfully removed container on host [10.10.10.6]
INFO[0019] [remove/rke-cp-port-listener] Successfully removed container on host [10.10.10.5]
INFO[0019] [remove/rke-cp-port-listener] Successfully removed container on host [10.10.10.4]
DEBU[0019] [remove/rke-worker-port-listener] Checking if container is running on host [10.10.10.6]
DEBU[0019] [remove/rke-worker-port-listener] Checking if container is running on host [10.10.10.4]
DEBU[0019] [remove/rke-worker-port-listener] Checking if container is running on host [10.10.10.5]
DEBU[0019] [remove/rke-worker-port-listener] Removing container on host [10.10.10.4]
INFO[0019] Removing container [rke-worker-port-listener] on host [10.10.10.4], try #1
DEBU[0019] [remove/rke-worker-port-listener] Removing container on host [10.10.10.6]
INFO[0019] Removing container [rke-worker-port-listener] on host [10.10.10.6], try #1
DEBU[0019] [remove/rke-worker-port-listener] Removing container on host [10.10.10.5]
INFO[0019] Removing container [rke-worker-port-listener] on host [10.10.10.5], try #1
INFO[0020] [remove/rke-worker-port-listener] Successfully removed container on host [10.10.10.4]
INFO[0020] [remove/rke-worker-port-listener] Successfully removed container on host [10.10.10.5]
INFO[0020] [remove/rke-worker-port-listener] Successfully removed container on host [10.10.10.6]
INFO[0020] [network] Port listener containers removed successfully
INFO[0020] [certificates] Deploying kubernetes certificates to Cluster nodes
INFO[0020] Checking if container [cert-deployer] is running on host [10.10.10.4], try #1
INFO[0020] Checking if container [cert-deployer] is running on host [10.10.10.6], try #1
INFO[0020] Checking if container [cert-deployer] is running on host [10.10.10.5], try #1
DEBU[0020] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4], try #1
DEBU[0020] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5], try #1
DEBU[0020] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6], try #1
INFO[0020] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5]
INFO[0020] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4]
INFO[0020] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6]
INFO[0020] Starting container [cert-deployer] on host [10.10.10.4], try #1
INFO[0020] Starting container [cert-deployer] on host [10.10.10.6], try #1
INFO[0020] Starting container [cert-deployer] on host [10.10.10.5], try #1
DEBU[0021] [certificates] Successfully started Certificate deployer container: cert-deployer
INFO[0021] Checking if container [cert-deployer] is running on host [10.10.10.4], try #1
DEBU[0021] [certificates] Successfully started Certificate deployer container: cert-deployer
INFO[0021] Checking if container [cert-deployer] is running on host [10.10.10.6], try #1
DEBU[0021] [certificates] Successfully started Certificate deployer container: cert-deployer
INFO[0021] Checking if container [cert-deployer] is running on host [10.10.10.5], try #1
INFO[0026] Checking if container [cert-deployer] is running on host [10.10.10.6], try #1
INFO[0026] Checking if container [cert-deployer] is running on host [10.10.10.4], try #1
INFO[0026] Removing container [cert-deployer] on host [10.10.10.6], try #1
INFO[0026] Checking if container [cert-deployer] is running on host [10.10.10.5], try #1
INFO[0026] Removing container [cert-deployer] on host [10.10.10.4], try #1
INFO[0026] Removing container [cert-deployer] on host [10.10.10.5], try #1
INFO[0026] [reconcile] Rebuilding and updating local kube config
DEBU[0026] [reconcile] Rebuilding and updating local kube config, creating new kubeconfig
DEBU[0026] Deploying admin Kubeconfig locally at [./kube_config_cluster.yml]
INFO[0026] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
DEBU[0026] [version] Using ./kube_config_cluster.yml to connect to Kubernetes cluster…
DEBU[0026] [version] Getting Kubernetes server version…
WARN[0026] [reconcile] host [10.10.10.6] is a control plane node without reachable Kubernetes API endpoint in the cluster
DEBU[0026] [reconcile] Rebuilding and updating local kube config, creating new kubeconfig
DEBU[0026] Deploying admin Kubeconfig locally at [./kube_config_cluster.yml]
INFO[0026] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
DEBU[0026] [version] Using ./kube_config_cluster.yml to connect to Kubernetes cluster…
DEBU[0026] [version] Getting Kubernetes server version…
WARN[0026] [reconcile] host [10.10.10.4] is a control plane node without reachable Kubernetes API endpoint in the cluster
DEBU[0026] [reconcile] Rebuilding and updating local kube config, creating new kubeconfig
DEBU[0026] Deploying admin Kubeconfig locally at [./kube_config_cluster.yml]
INFO[0026] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
DEBU[0026] [version] Using ./kube_config_cluster.yml to connect to Kubernetes cluster…
DEBU[0026] [version] Getting Kubernetes server version…
WARN[0026] [reconcile] host [10.10.10.5] is a control plane node without reachable Kubernetes API endpoint in the cluster
WARN[0026] [reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found
INFO[0026] [certificates] Successfully deployed kubernetes certificates to Cluster nodes
INFO[0026] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.10.10.4]
DEBU[0026] [remove/file-deployer] Checking if container is running on host [10.10.10.4]
DEBU[0026] [remove/file-deployer] Container doesn’t exist on host [10.10.10.4]
DEBU[0026] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4], try #1
INFO[0026] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4]
INFO[0026] Starting container [file-deployer] on host [10.10.10.4], try #1
INFO[0026] Successfully started [file-deployer] container on host [10.10.10.4]
INFO[0026] Waiting for [file-deployer] container to exit on host [10.10.10.4]
INFO[0026] Waiting for [file-deployer] container to exit on host [10.10.10.4]
INFO[0027] Container [file-deployer] is still running on host [10.10.10.4]: stderr: [], stdout: []
INFO[0028] Waiting for [file-deployer] container to exit on host [10.10.10.4]
DEBU[0028] Exit code for [file-deployer] container on host [10.10.10.4] is [0]
DEBU[0028] [remove/file-deployer] Checking if container is running on host [10.10.10.4]
DEBU[0028] [remove/file-deployer] Removing container on host [10.10.10.4]
INFO[0028] Removing container [file-deployer] on host [10.10.10.4], try #1
INFO[0028] [remove/file-deployer] Successfully removed container on host [10.10.10.4]
DEBU[0028] [file-deploy] Successfully deployed file [/etc/kubernetes/audit-policy.yaml] on node [10.10.10.4]
INFO[0028] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.10.10.5]
DEBU[0028] [remove/file-deployer] Checking if container is running on host [10.10.10.5]
DEBU[0028] [remove/file-deployer] Container doesn’t exist on host [10.10.10.5]
DEBU[0028] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5], try #1
INFO[0028] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5]
INFO[0028] Starting container [file-deployer] on host [10.10.10.5], try #1
INFO[0028] Successfully started [file-deployer] container on host [10.10.10.5]
INFO[0028] Waiting for [file-deployer] container to exit on host [10.10.10.5]
INFO[0028] Waiting for [file-deployer] container to exit on host [10.10.10.5]
INFO[0028] Container [file-deployer] is still running on host [10.10.10.5]: stderr: [], stdout: []
INFO[0029] Waiting for [file-deployer] container to exit on host [10.10.10.5]
DEBU[0030] Exit code for [file-deployer] container on host [10.10.10.5] is [0]
DEBU[0030] [remove/file-deployer] Checking if container is running on host [10.10.10.5]
DEBU[0030] [remove/file-deployer] Removing container on host [10.10.10.5]
INFO[0030] Removing container [file-deployer] on host [10.10.10.5], try #1
INFO[0030] [remove/file-deployer] Successfully removed container on host [10.10.10.5]
DEBU[0030] [file-deploy] Successfully deployed file [/etc/kubernetes/audit-policy.yaml] on node [10.10.10.5]
INFO[0030] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.10.10.6]
DEBU[0030] [remove/file-deployer] Checking if container is running on host [10.10.10.6]
DEBU[0030] [remove/file-deployer] Container doesn’t exist on host [10.10.10.6]
DEBU[0030] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6], try #1
INFO[0030] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6]
INFO[0030] Starting container [file-deployer] on host [10.10.10.6], try #1
INFO[0030] Successfully started [file-deployer] container on host [10.10.10.6]
INFO[0030] Waiting for [file-deployer] container to exit on host [10.10.10.6]
INFO[0030] Waiting for [file-deployer] container to exit on host [10.10.10.6]
DEBU[0031] Exit code for [file-deployer] container on host [10.10.10.6] is [0]
DEBU[0031] [remove/file-deployer] Checking if container is running on host [10.10.10.6]
DEBU[0031] [remove/file-deployer] Removing container on host [10.10.10.6]
INFO[0031] Removing container [file-deployer] on host [10.10.10.6], try #1
INFO[0031] [remove/file-deployer] Successfully removed container on host [10.10.10.6]
DEBU[0031] [file-deploy] Successfully deployed file [/etc/kubernetes/audit-policy.yaml] on node [10.10.10.6]
INFO[0031] [/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes
INFO[0031] [reconcile] Reconciling cluster state
INFO[0031] [reconcile] This is newly generated cluster
DEBU[0031] Encryption is disabled in both current and new spec; no action is required
INFO[0031] Pre-pulling kubernetes images
DEBU[0031] Checking if image [rancher/hyperkube:v1.20.6-rancher1] exists on host [10.10.10.6], try #1
DEBU[0031] Checking if image [rancher/hyperkube:v1.20.6-rancher1] exists on host [10.10.10.5], try #1
DEBU[0031] Checking if image [rancher/hyperkube:v1.20.6-rancher1] exists on host [10.10.10.4], try #1
INFO[0031] Image [rancher/hyperkube:v1.20.6-rancher1] exists on host [10.10.10.6]
INFO[0031] Image [rancher/hyperkube:v1.20.6-rancher1] exists on host [10.10.10.5]
INFO[0031] Image [rancher/hyperkube:v1.20.6-rancher1] exists on host [10.10.10.4]
INFO[0031] Kubernetes images pulled successfully
DEBU[0031] getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.6-rancher1-1]
DEBU[0031] Extracted version [v1.20.6-rancher1] from image [rancher/hyperkube:v1.20.6-rancher1]
DEBU[0031] getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]
DEBU[0031] Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]
DEBU[0031] etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001
DEBU[0031] etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd
DEBU[0031] Version [3.4.15-rancher1] is equal or higher than version [3.2.99]
DEBU[0031] getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.6-rancher1-1]
DEBU[0031] Extracted version [v1.20.6-rancher1] from image [rancher/hyperkube:v1.20.6-rancher1]
DEBU[0031] getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]
DEBU[0031] Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]
DEBU[0031] etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001
DEBU[0031] etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd
DEBU[0031] Version [3.4.15-rancher1] is equal or higher than version [3.2.99]
DEBU[0031] getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.6-rancher1-1]
DEBU[0031] Extracted version [v1.20.6-rancher1] from image [rancher/hyperkube:v1.20.6-rancher1]
DEBU[0031] getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]
DEBU[0031] Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]
DEBU[0031] etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001
DEBU[0031] etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd
DEBU[0031] Version [3.4.15-rancher1] is equal or higher than version [3.2.99]
INFO[0031] [etcd] Building up etcd plane…
DEBU[0031] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6], try #1
INFO[0031] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6]
INFO[0031] Starting container [etcd-fix-perm] on host [10.10.10.6], try #1
INFO[0031] Successfully started [etcd-fix-perm] container on host [10.10.10.6]
INFO[0031] Waiting for [etcd-fix-perm] container to exit on host [10.10.10.6]
INFO[0031] Waiting for [etcd-fix-perm] container to exit on host [10.10.10.6]
DEBU[0032] Exit code for [etcd-fix-perm] container on host [10.10.10.6] is [0]
DEBU[0032] [remove/etcd-fix-perm] Checking if container is running on host [10.10.10.6]
DEBU[0032] [remove/etcd-fix-perm] Removing container on host [10.10.10.6]
INFO[0032] Removing container [etcd-fix-perm] on host [10.10.10.6], try #1
INFO[0032] [remove/etcd-fix-perm] Successfully removed container on host [10.10.10.6]
DEBU[0032] [etcd] Container [etcd] is already running on host [10.10.10.6]
DEBU[0032] [etcd] Checking if container [etcd] is eligible for upgrade on host [10.10.10.6]
DEBU[0032] [etcd] Container [etcd] is not eligible for upgrade on host [10.10.10.6]
DEBU[0032] Extracted version [v0.1.74] from image [rancher/rke-tools:v0.1.74]
DEBU[0032] Extracted version [v0.1.74] from image [rancher/rke-tools:v0.1.74]
INFO[0032] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.10.10.6]
DEBU[0032] [etcd] Using command [/opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=10.10.10.6:2379 --retention=72h --creation=12h] for rolling snapshot container [etcd-rolling-snapshots] on host [10.10.10.6]
DEBU[0032] [remove/etcd-rolling-snapshots] Checking if container is running on host [10.10.10.6]
DEBU[0032] [remove/etcd-rolling-snapshots] Removing container on host [10.10.10.6]
INFO[0032] Removing container [etcd-rolling-snapshots] on host [10.10.10.6], try #1
INFO[0032] [remove/etcd-rolling-snapshots] Successfully removed container on host [10.10.10.6]
DEBU[0032] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6], try #1
INFO[0032] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6]
INFO[0032] Starting container [etcd-rolling-snapshots] on host [10.10.10.6], try #1
INFO[0032] [etcd] Successfully started [etcd-rolling-snapshots] container on host [10.10.10.6]
DEBU[0037] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6], try #1
INFO[0037] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6]
INFO[0038] Starting container [rke-bundle-cert] on host [10.10.10.6], try #1
INFO[0038] [certificates] Successfully started [rke-bundle-cert] container on host [10.10.10.6]
INFO[0038] Waiting for [rke-bundle-cert] container to exit on host [10.10.10.6]
DEBU[0038] Exit code for [rke-bundle-cert] container on host [10.10.10.6] is [0]
INFO[0038] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.10.10.6]
INFO[0038] Removing container [rke-bundle-cert] on host [10.10.10.6], try #1
DEBU[0038] [etcd] Creating log link for Container [etcd-rolling-snapshots] on host [10.10.10.6]
DEBU[0038] [remove/rke-log-linker] Checking if container is running on host [10.10.10.6]
DEBU[0038] [remove/rke-log-linker] Container doesn’t exist on host [10.10.10.6]
DEBU[0038] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6], try #1
INFO[0038] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6]
INFO[0038] Starting container [rke-log-linker] on host [10.10.10.6], try #1
INFO[0039] [etcd] Successfully started [rke-log-linker] container on host [10.10.10.6]
DEBU[0039] [remove/rke-log-linker] Checking if container is running on host [10.10.10.6]
DEBU[0039] [remove/rke-log-linker] Removing container on host [10.10.10.6]
INFO[0039] Removing container [rke-log-linker] on host [10.10.10.6], try #1
INFO[0039] [remove/rke-log-linker] Successfully removed container on host [10.10.10.6]
DEBU[0039] [etcd] Successfully created log link for Container [etcd-rolling-snapshots] on host [10.10.10.6]
DEBU[0039] [etcd] Creating log link for Container [etcd] on host [10.10.10.6]
DEBU[0039] [remove/rke-log-linker] Checking if container is running on host [10.10.10.6]
DEBU[0039] [remove/rke-log-linker] Container doesn’t exist on host [10.10.10.6]
DEBU[0039] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6], try #1
INFO[0039] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.6]
INFO[0039] Starting container [rke-log-linker] on host [10.10.10.6], try #1
INFO[0040] [etcd] Successfully started [rke-log-linker] container on host [10.10.10.6]
DEBU[0040] [remove/rke-log-linker] Checking if container is running on host [10.10.10.6]
DEBU[0040] [remove/rke-log-linker] Removing container on host [10.10.10.6]
INFO[0040] Removing container [rke-log-linker] on host [10.10.10.6], try #1
INFO[0040] [remove/rke-log-linker] Successfully removed container on host [10.10.10.6]
DEBU[0040] [etcd] Successfully created log link for Container [etcd] on host [10.10.10.6]
DEBU[0040] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4], try #1
INFO[0040] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4]
INFO[0040] Starting container [etcd-fix-perm] on host [10.10.10.4], try #1
INFO[0041] Successfully started [etcd-fix-perm] container on host [10.10.10.4]
INFO[0041] Waiting for [etcd-fix-perm] container to exit on host [10.10.10.4]
INFO[0041] Waiting for [etcd-fix-perm] container to exit on host [10.10.10.4]
INFO[0041] Container [etcd-fix-perm] is still running on host [10.10.10.4]: stderr: [], stdout: []
INFO[0042] Waiting for [etcd-fix-perm] container to exit on host [10.10.10.4]
DEBU[0042] Exit code for [etcd-fix-perm] container on host [10.10.10.4] is [0]
DEBU[0042] [remove/etcd-fix-perm] Checking if container is running on host [10.10.10.4]
DEBU[0042] [remove/etcd-fix-perm] Removing container on host [10.10.10.4]
INFO[0042] Removing container [etcd-fix-perm] on host [10.10.10.4], try #1
INFO[0042] [remove/etcd-fix-perm] Successfully removed container on host [10.10.10.4]
DEBU[0042] [etcd] Container [etcd] is already running on host [10.10.10.4]
DEBU[0042] [etcd] Checking if container [etcd] is eligible for upgrade on host [10.10.10.4]
DEBU[0042] [etcd] Container [etcd] is not eligible for upgrade on host [10.10.10.4]
DEBU[0042] Extracted version [v0.1.74] from image [rancher/rke-tools:v0.1.74]
DEBU[0042] Extracted version [v0.1.74] from image [rancher/rke-tools:v0.1.74]
INFO[0042] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.10.10.4]
DEBU[0042] [etcd] Using command [/opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=10.10.10.4:2379 --retention=72h --creation=12h] for rolling snapshot container [etcd-rolling-snapshots] on host [10.10.10.4]
DEBU[0042] [remove/etcd-rolling-snapshots] Checking if container is running on host [10.10.10.4]
DEBU[0042] [remove/etcd-rolling-snapshots] Removing container on host [10.10.10.4]
INFO[0042] Removing container [etcd-rolling-snapshots] on host [10.10.10.4], try #1
INFO[0042] [remove/etcd-rolling-snapshots] Successfully removed container on host [10.10.10.4]
DEBU[0042] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4], try #1
INFO[0042] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4]
INFO[0042] Starting container [etcd-rolling-snapshots] on host [10.10.10.4], try #1
INFO[0042] [etcd] Successfully started [etcd-rolling-snapshots] container on host [10.10.10.4]
DEBU[0047] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4], try #1
INFO[0048] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4]
INFO[0048] Starting container [rke-bundle-cert] on host [10.10.10.4], try #1
INFO[0048] [certificates] Successfully started [rke-bundle-cert] container on host [10.10.10.4]
INFO[0048] Waiting for [rke-bundle-cert] container to exit on host [10.10.10.4]
DEBU[0048] Exit code for [rke-bundle-cert] container on host [10.10.10.4] is [0]
INFO[0048] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.10.10.4]
INFO[0048] Removing container [rke-bundle-cert] on host [10.10.10.4], try #1
DEBU[0048] [etcd] Creating log link for Container [etcd-rolling-snapshots] on host [10.10.10.4]
DEBU[0049] [remove/rke-log-linker] Checking if container is running on host [10.10.10.4]
DEBU[0049] [remove/rke-log-linker] Container doesn’t exist on host [10.10.10.4]
DEBU[0049] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4], try #1
INFO[0049] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4]
INFO[0049] Starting container [rke-log-linker] on host [10.10.10.4], try #1
INFO[0049] [etcd] Successfully started [rke-log-linker] container on host [10.10.10.4]
DEBU[0049] [remove/rke-log-linker] Checking if container is running on host [10.10.10.4]
DEBU[0049] [remove/rke-log-linker] Removing container on host [10.10.10.4]
INFO[0049] Removing container [rke-log-linker] on host [10.10.10.4], try #1
INFO[0049] [remove/rke-log-linker] Successfully removed container on host [10.10.10.4]
DEBU[0049] [etcd] Successfully created log link for Container [etcd-rolling-snapshots] on host [10.10.10.4]
DEBU[0049] [etcd] Creating log link for Container [etcd] on host [10.10.10.4]
DEBU[0049] [remove/rke-log-linker] Checking if container is running on host [10.10.10.4]
DEBU[0049] [remove/rke-log-linker] Container doesn’t exist on host [10.10.10.4]
DEBU[0049] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4], try #1
INFO[0049] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.4]
INFO[0049] Starting container [rke-log-linker] on host [10.10.10.4], try #1
INFO[0050] [etcd] Successfully started [rke-log-linker] container on host [10.10.10.4]
DEBU[0050] [remove/rke-log-linker] Checking if container is running on host [10.10.10.4]
DEBU[0050] [remove/rke-log-linker] Removing container on host [10.10.10.4]
INFO[0050] Removing container [rke-log-linker] on host [10.10.10.4], try #1
INFO[0050] [remove/rke-log-linker] Successfully removed container on host [10.10.10.4]
DEBU[0050] [etcd] Successfully created log link for Container [etcd] on host [10.10.10.4]
DEBU[0050] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5], try #1
INFO[0050] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5]
INFO[0050] Starting container [etcd-fix-perm] on host [10.10.10.5], try #1
INFO[0051] Successfully started [etcd-fix-perm] container on host [10.10.10.5]
INFO[0051] Waiting for [etcd-fix-perm] container to exit on host [10.10.10.5]
INFO[0051] Waiting for [etcd-fix-perm] container to exit on host [10.10.10.5]
INFO[0051] Container [etcd-fix-perm] is still running on host [10.10.10.5]: stderr: [], stdout: []
INFO[0052] Waiting for [etcd-fix-perm] container to exit on host [10.10.10.5]
DEBU[0052] Exit code for [etcd-fix-perm] container on host [10.10.10.5] is [0]
DEBU[0052] [remove/etcd-fix-perm] Checking if container is running on host [10.10.10.5]
DEBU[0052] [remove/etcd-fix-perm] Removing container on host [10.10.10.5]
INFO[0052] Removing container [etcd-fix-perm] on host [10.10.10.5], try #1
INFO[0052] [remove/etcd-fix-perm] Successfully removed container on host [10.10.10.5]
DEBU[0052] [etcd] Container [etcd] is already running on host [10.10.10.5]
DEBU[0052] [etcd] Checking if container [etcd] is eligible for upgrade on host [10.10.10.5]
DEBU[0052] [etcd] Container [etcd] is not eligible for upgrade on host [10.10.10.5]
DEBU[0052] Extracted version [v0.1.74] from image [rancher/rke-tools:v0.1.74]
DEBU[0052] Extracted version [v0.1.74] from image [rancher/rke-tools:v0.1.74]
INFO[0052] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.10.10.5]
DEBU[0052] [etcd] Using command [/opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=10.10.10.5:2379 --retention=72h --creation=12h] for rolling snapshot container [etcd-rolling-snapshots] on host [10.10.10.5]
DEBU[0052] [remove/etcd-rolling-snapshots] Checking if container is running on host [10.10.10.5]
DEBU[0052] [remove/etcd-rolling-snapshots] Removing container on host [10.10.10.5]
INFO[0052] Removing container [etcd-rolling-snapshots] on host [10.10.10.5], try #1
INFO[0052] [remove/etcd-rolling-snapshots] Successfully removed container on host [10.10.10.5]
DEBU[0052] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5], try #1
INFO[0052] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5]
INFO[0052] Starting container [etcd-rolling-snapshots] on host [10.10.10.5], try #1
INFO[0053] [etcd] Successfully started [etcd-rolling-snapshots] container on host [10.10.10.5]
DEBU[0058] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5], try #1
INFO[0058] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5]
INFO[0058] Starting container [rke-bundle-cert] on host [10.10.10.5], try #1
INFO[0058] [certificates] Successfully started [rke-bundle-cert] container on host [10.10.10.5]
INFO[0058] Waiting for [rke-bundle-cert] container to exit on host [10.10.10.5]
DEBU[0059] Exit code for [rke-bundle-cert] container on host [10.10.10.5] is [0]
INFO[0059] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.10.10.5]
INFO[0059] Removing container [rke-bundle-cert] on host [10.10.10.5], try #1
DEBU[0059] [etcd] Creating log link for Container [etcd-rolling-snapshots] on host [10.10.10.5]
DEBU[0059] [remove/rke-log-linker] Checking if container is running on host [10.10.10.5]
DEBU[0059] [remove/rke-log-linker] Container doesn’t exist on host [10.10.10.5]
DEBU[0059] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5], try #1
INFO[0059] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5]
INFO[0059] Starting container [rke-log-linker] on host [10.10.10.5], try #1
INFO[0059] [etcd] Successfully started [rke-log-linker] container on host [10.10.10.5]
DEBU[0059] [remove/rke-log-linker] Checking if container is running on host [10.10.10.5]
DEBU[0059] [remove/rke-log-linker] Removing container on host [10.10.10.5]
INFO[0059] Removing container [rke-log-linker] on host [10.10.10.5], try #1
INFO[0059] [remove/rke-log-linker] Successfully removed container on host [10.10.10.5]
DEBU[0059] [etcd] Successfully created log link for Container [etcd-rolling-snapshots] on host [10.10.10.5]
DEBU[0059] [etcd] Creating log link for Container [etcd] on host [10.10.10.5]
DEBU[0059] [remove/rke-log-linker] Checking if container is running on host [10.10.10.5]
DEBU[0059] [remove/rke-log-linker] Container doesn’t exist on host [10.10.10.5]
DEBU[0059] Checking if image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5], try #1
INFO[0059] Image [rancher/rke-tools:v0.1.74] exists on host [10.10.10.5]
INFO[0060] Starting container [rke-log-linker] on host [10.10.10.5], try #1
INFO[0060] [etcd] Successfully started [rke-log-linker] container on host [10.10.10.5]
DEBU[0060] [remove/rke-log-linker] Checking if container is running on host [10.10.10.5]
DEBU[0060] [remove/rke-log-linker] Removing container on host [10.10.10.5]
INFO[0060] Removing container [rke-log-linker] on host [10.10.10.5], try #1
INFO[0060] [remove/rke-log-linker] Successfully removed container on host [10.10.10.5]
DEBU[0060] [etcd] Successfully created log link for Container [etcd] on host [10.10.10.5]
INFO[0060] [etcd] Successfully started etcd plane… Checking etcd cluster health
DEBU[0060] [etcd] check etcd cluster health on host [10.10.10.6]
DEBU[0070] [etcd] failed to check health for etcd host [10.10.10.6]: failed to get /health for host [10.10.10.6]: Get “ht-tps://10.10.10.6:2379/health”: net/http: TLS handshake timeout
DEBU[0086] [etcd] failed to check health for etcd host [10.10.10.6]: failed to get /health for host [10.10.10.6]: Get “htt-ps://10.10.10.6:2379/health”: net/http: TLS handshake timeout
it keeps failing till I get the error message and the process stops 
please help.
(I’ve added some dashes between the links since the system won’t allow me to add more than two links to my topic)