Hi,
I have problems bootstraping a rancher user cluster.
Rancher successfully running on three nodes. Now tryining to bootstrap a user cluster with three nodes failing with: Failed to apply the ServiceAccount needed for job execution: Post “https://192.168.164.16:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=30s”: net/http: TLS handshake timeout
Rancher is configured with a certificate from an official ca (Sectigo) und HTTPS-connections are working. The tls.crt contains the cert and the intermediate ones.
Any ideas?
Rancher-HA ist running behind a proxy and works finde. I tried rancher-agent using a private registry or via proxy, same problem. The node is absolutly cleaned according to the docs. rancher-ha ist freshly setup, so there are no problems with old volums, data and so on.
I tried rancher 2.5 and 2.6.
Any ideas? I’m trying since 3 days now, but the problem never changes.
Logs following…
Environment:
SUSE SLES 15 SP2
Docker 20.10.6_ce-6.49.3 (also tried 19.x)
firewalld/selinux/apparmor disabled (enabled doesn’t change anything)
Network-plugin calico
etcd-log on Agent Node:
docker logs -f 8d7942179ff5
2021-09-24 20:54:41.709365 W | pkg/flags: unrecognized environment variable ETCD_UNSUPPORTED_ARCH=x86_64
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-09-24 20:54:41.709519 I | etcdmain: etcd Version: 3.4.16
2021-09-24 20:54:41.709530 I | etcdmain: Git SHA: d19fbe541
2021-09-24 20:54:41.709537 I | etcdmain: Go Version: go1.12.17
2021-09-24 20:54:41.709545 I | etcdmain: Go OS/Arch: linux/amd64
2021-09-24 20:54:41.709557 I | etcdmain: setting maximum number of CPUs to 8, total number of available CPUs is 8
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-09-24 20:54:41.709711 I | embed: peerTLS: cert = /etc/kubernetes/ssl/kube-etcd-192-168-164-15.pem, key = /etc/kubernetes/ssl/kube-etcd-192-168-164-15-key.pem, trusted-ca = /etc/kubernetes/ssl/kube-ca.pem, client-cert-auth = true, crl-file =
2021-09-24 20:54:41.711021 I | embed: name = etcd-rkedev01
2021-09-24 20:54:41.711056 I | embed: data dir = /var/lib/rancher/etcd/
2021-09-24 20:54:41.711066 I | embed: member dir = /var/lib/rancher/etcd/member
2021-09-24 20:54:41.711073 I | embed: heartbeat = 500ms
2021-09-24 20:54:41.711080 I | embed: election = 5000ms
2021-09-24 20:54:41.711087 I | embed: snapshot count = 100000
2021-09-24 20:54:41.711102 I | embed: advertise client URLs = https://192.168.164.15:2379
2021-09-24 20:54:41.727343 I | etcdserver: starting member 911ba85b4da8b655 in cluster c8379e7875d1a66a
raft2021/09/24 20:54:41 INFO: 911ba85b4da8b655 switched to configuration voters=()
raft2021/09/24 20:54:41 INFO: 911ba85b4da8b655 became follower at term 0
raft2021/09/24 20:54:41 INFO: newRaft 911ba85b4da8b655 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
raft2021/09/24 20:54:41 INFO: 911ba85b4da8b655 became follower at term 1
raft2021/09/24 20:54:41 INFO: 911ba85b4da8b655 switched to configuration voters=(10456136069969131093)
2021-09-24 20:54:41.731928 W | auth: simple token is not cryptographically signed
2021-09-24 20:54:41.741870 I | etcdserver: starting server... [version: 3.4.16, cluster version: to_be_decided]
2021-09-24 20:54:41.742168 I | etcdserver: 911ba85b4da8b655 as single-node; fast-forwarding 9 ticks (election ticks 10)
raft2021/09/24 20:54:41 INFO: 911ba85b4da8b655 switched to configuration voters=(10456136069969131093)
2021-09-24 20:54:41.742900 I | etcdserver/membership: added member 911ba85b4da8b655 [https://192.168.164.15:2380] to cluster c8379e7875d1a66a
2021-09-24 20:54:41.745982 I | embed: ClientTLS: cert = /etc/kubernetes/ssl/kube-etcd-192-168-164-15.pem, key = /etc/kubernetes/ssl/kube-etcd-192-168-164-15-key.pem, trusted-ca = /etc/kubernetes/ssl/kube-ca.pem, client-cert-auth = true, crl-file =
2021-09-24 20:54:41.746104 I | embed: listening for peers on [::]:2380
raft2021/09/24 20:54:43 INFO: 911ba85b4da8b655 is starting a new election at term 1
raft2021/09/24 20:54:43 INFO: 911ba85b4da8b655 became candidate at term 2
raft2021/09/24 20:54:43 INFO: 911ba85b4da8b655 received MsgVoteResp from 911ba85b4da8b655 at term 2
raft2021/09/24 20:54:43 INFO: 911ba85b4da8b655 became leader at term 2
raft2021/09/24 20:54:43 INFO: raft.node: 911ba85b4da8b655 elected leader 911ba85b4da8b655 at term 2
2021-09-24 20:54:43.228736 I | etcdserver: published {Name:etcd-rkedev01 ClientURLs:[https://192.168.164.15:2379]} to cluster c8379e7875d1a66a
2021-09-24 20:54:43.228869 I | embed: ready to serve client requests
2021-09-24 20:54:43.228918 I | etcdserver: setting up the initial cluster version to 3.4
2021-09-24 20:54:43.230083 N | etcdserver/membership: set the initial cluster version to 3.4
2021-09-24 20:54:43.230133 I | embed: serving client requests on [::]:2379
2021-09-24 20:54:43.230199 I | etcdserver/api: enabled capabilities for version 3.4
2021-09-24 20:55:46.146670 I | embed: rejected connection from "192.168.164.15:41832" (error "remote error: tls: unknown certificate authority", ServerName "")
2021-09-24 20:55:48.553841 I | embed: rejected connection from "192.168.164.15:41834" (error "remote error: tls: unknown certificate authority", ServerName "")
2021-09-24 20:55:49.866369 I | embed: rejected connection from "192.168.164.15:41836" (error "remote error: tls: unknown certificate authority", ServerName "")
2021-09-24 20:55:52.683701 I | embed: rejected connection from "192.168.164.15:41838" (error "remote error: tls: unknown certificate authority", ServerName "")
2021-09-24 20:55:54.293984 I | embed: rejected connection from "192.168.164.15:41840" (error "remote error: tls: unknown certificate authority", ServerName "")
2021-09-24 20:55:55.401821 I | embed: rejected connection from "192.168.164.15:41844" (error "remote error: tls: unknown certificate authority", ServerName "")
2021-09-24 20:55:56.514356 I | embed: rejected connection from "192.168.164.15:41846" (error "remote error: tls: unknown certificate authority", ServerName "")
2021-09-24 20:55:57.822116 I | embed: rejected connection from "192.168.164.15:41848" (error "remote error: tls: unknown certificate authority", ServerName "")
2021-09-24 20:55:59.346038 I | embed: rejected connection from "192.168.164.15:41850" (error "remote error: tls: unknown certificate authority", ServerName "")
2021-09-24 20:56:00.436602 I | embed: rejected connection from "192.168.164.15:41854" (error "remote error: tls: unknown certificate authority", ServerName "")
2021-09-24 20:56:05.756694 I | embed: rejected connection from "192.168.164.15:41856" (error "tls: client didn't provide a certificate", ServerName "")
2021-09-24 20:56:11.383389 I | embed: rejected connection from "192.168.164.15:41860" (error "tls: client didn't provide a certificate", ServerName "")
2021-09-24 20:56:13.106927 I | embed: rejected connection from "192.168.164.15:41862" (error "tls: client didn't provide a certificate", ServerName "")
2021-09-24 20:56:16.328718 I | embed: rejected connection from "192.168.164.15:41864" (error "tls: client didn't provide a certificate", ServerName "")
2021-09-24 20:57:03.515355 I | embed: rejected connection from "192.168.164.15:36367" (error "EOF", ServerName "")
2021-09-24 20:59:54.006191 I | embed: rejected connection from "192.168.164.15:43115" (error "EOF", ServerName "")
2021-09-24 21:00:41.113906 N | pkg/osutil: received terminated signal, shutting down...
WARNING: 2021/09/24 21:00:41 grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 0.0.0.0:2379: connect: connection refused". Reconnecting...
2021-09-24 21:00:41.121224 I | etcdserver: skipped leadership transfer for single voting member cluster
2021-09-24 21:04:08.000257 W | pkg/flags: unrecognized environment variable ETCD_UNSUPPORTED_ARCH=x86_64
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-09-24 21:04:08.000478 I | etcdmain: etcd Version: 3.4.16
2021-09-24 21:04:08.000499 I | etcdmain: Git SHA: d19fbe541
2021-09-24 21:04:08.000511 I | etcdmain: Go Version: go1.12.17
2021-09-24 21:04:08.000523 I | etcdmain: Go OS/Arch: linux/amd64
2021-09-24 21:04:08.000537 I | etcdmain: setting maximum number of CPUs to 8, total number of available CPUs is 8
2021-09-24 21:04:08.000695 N | etcdmain: the server is already initialized as member before, starting as etcd member...
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-09-24 21:04:08.000795 I | embed: peerTLS: cert = /etc/kubernetes/ssl/kube-etcd-192-168-164-15.pem, key = /etc/kubernetes/ssl/kube-etcd-192-168-164-15-key.pem, trusted-ca = /etc/kubernetes/ssl/kube-ca.pem, client-cert-auth = true, crl-file =
2021-09-24 21:04:08.002717 I | embed: name = etcd-rkedev01
2021-09-24 21:04:08.002751 I | embed: data dir = /var/lib/rancher/etcd/
2021-09-24 21:04:08.002769 I | embed: member dir = /var/lib/rancher/etcd/member
2021-09-24 21:04:08.002780 I | embed: heartbeat = 500ms
2021-09-24 21:04:08.002791 I | embed: election = 5000ms
2021-09-24 21:04:08.002802 I | embed: snapshot count = 100000
2021-09-24 21:04:08.002826 I | embed: advertise client URLs = https://192.168.164.15:2379
2021-09-24 21:04:08.002843 I | embed: initial advertise peer URLs = https://192.168.164.15:2380
2021-09-24 21:04:08.002859 I | embed: initial cluster =
2021-09-24 21:04:08.019815 I | etcdserver: restarting member 911ba85b4da8b655 in cluster c8379e7875d1a66a at commit index 772
raft2021/09/24 21:04:08 INFO: 911ba85b4da8b655 switched to configuration voters=()
raft2021/09/24 21:04:08 INFO: 911ba85b4da8b655 became follower at term 2
raft2021/09/24 21:04:08 INFO: newRaft 911ba85b4da8b655 [peers: [], term: 2, commit: 772, applied: 0, lastindex: 772, lastterm: 2]
2021-09-24 21:04:08.022074 W | auth: simple token is not cryptographically signed
2021-09-24 21:04:08.030593 I | etcdserver: starting server... [version: 3.4.16, cluster version: to_be_decided]
raft2021/09/24 21:04:08 INFO: 911ba85b4da8b655 switched to configuration voters=(10456136069969131093)
2021-09-24 21:04:08.031475 I | etcdserver/membership: added member 911ba85b4da8b655 [https://192.168.164.15:2380] to cluster c8379e7875d1a66a
2021-09-24 21:04:08.031639 N | etcdserver/membership: set the initial cluster version to 3.4
2021-09-24 21:04:08.031703 I | etcdserver/api: enabled capabilities for version 3.4
2021-09-24 21:04:08.035182 I | embed: ClientTLS: cert = /etc/kubernetes/ssl/kube-etcd-192-168-164-15.pem, key = /etc/kubernetes/ssl/kube-etcd-192-168-164-15-key.pem, trusted-ca = /etc/kubernetes/ssl/kube-ca.pem, client-cert-auth = true, crl-file =
2021-09-24 21:04:08.035314 I | embed: listening for peers on [::]:2380
raft2021/09/24 21:04:17 INFO: 911ba85b4da8b655 is starting a new election at term 2
raft2021/09/24 21:04:17 INFO: 911ba85b4da8b655 became candidate at term 3
raft2021/09/24 21:04:17 INFO: 911ba85b4da8b655 received MsgVoteResp from 911ba85b4da8b655 at term 3
raft2021/09/24 21:04:17 INFO: 911ba85b4da8b655 became leader at term 3
raft2021/09/24 21:04:17 INFO: raft.node: 911ba85b4da8b655 elected leader 911ba85b4da8b655 at term 3
2021-09-24 21:04:17.524961 I | embed: ready to serve client requests
2021-09-24 21:04:17.525144 I | etcdserver: published {Name:etcd-rkedev01 ClientURLs:[https://192.168.164.15:2379]} to cluster c8379e7875d1a66a
2021-09-24 21:04:17.528026 I | embed: serving client requests on [::]:2379
2021-09-24 21:05:15.829760 I | mvcc: store.index: compact 635
2021-09-24 21:05:15.831614 I | mvcc: finished scheduled compaction at 635 (took 1.430143ms)
2021-09-24 21:10:15.835386 I | mvcc: store.index: compact 698
2021-09-24 21:10:15.836367 I | mvcc: finished scheduled compaction at 698 (took 632.074µs)
2021-09-24 21:10:47.522537 I | embed: rejected connection from "192.168.164.15:42547" (error "EOF", ServerName "")
2021-09-24 21:15:15.842481 I | mvcc: store.index: compact 984
2021-09-24 21:15:15.844258 I | mvcc: finished scheduled compaction at 984 (took 1.449525ms)
2021-09-24 21:20:15.849190 I | mvcc: store.index: compact 1309
2021-09-24 21:20:15.850949 I | mvcc: finished scheduled compaction at 1309 (took 1.330925ms)
2021-09-24 21:20:37.559299 I | embed: rejected connection from "192.168.164.15:39495" (error "EOF", ServerName "")
Certs:
ls -latr /etc/kubernetes/ssl/
total 108
-rw------- 1 root root 1123 Sep 24 22:52 kube-apiserver-requestheader-ca.pem
-rw------- 1 root root 1302 Sep 24 22:52 kube-apiserver.pem
-rw------- 1 root root 1294 Sep 24 22:52 kube-etcd-192-168-164-15.pem
-rw------- 1 root root 1151 Sep 24 22:52 kube-apiserver-proxy-client.pem
-rw------- 1 root root 1107 Sep 24 22:52 kube-controller-manager.pem
-rw------- 1 root root 1679 Sep 24 22:52 kube-node-key.pem
-rw------- 1 root root 1675 Sep 24 22:52 kube-scheduler-key.pem
-rw------- 1 root root 1679 Sep 24 22:52 kube-apiserver-key.pem
-rw------- 1 root root 1302 Sep 24 22:52 kube-service-account-token.pem
-rw------- 1 root root 1090 Sep 24 22:52 kube-proxy.pem
-rw------- 1 root root 1675 Sep 24 22:52 kube-controller-manager-key.pem
-rw------- 1 root root 1115 Sep 24 22:52 kube-node.pem
-rw------- 1 root root 1675 Sep 24 22:52 kube-etcd-192-168-164-15-key.pem
-rw------- 1 root root 1094 Sep 24 22:52 kube-scheduler.pem
-rw------- 1 root root 1675 Sep 24 22:52 kube-ca-key.pem
-rw------- 1 root root 1058 Sep 24 22:52 kube-ca.pem
-rw------- 1 root root 1679 Sep 24 22:52 kube-proxy-key.pem
-rw------- 1 root root 1675 Sep 24 22:52 kube-apiserver-requestheader-ca-key.pem
-rw------- 1 root root 1679 Sep 24 22:52 kube-apiserver-proxy-client-key.pem
-rw------- 1 root root 1679 Sep 24 22:52 kube-service-account-token-key.pem
-rw------- 1 root root 533 Sep 24 22:52 kubecfg-kube-apiserver-requestheader-ca.yaml
-rw------- 1 root root 501 Sep 24 22:52 kubecfg-kube-controller-manager.yaml
-rw------- 1 root root 445 Sep 24 22:52 kubecfg-kube-node.yaml
-rw------- 1 root root 517 Sep 24 22:52 kubecfg-kube-apiserver-proxy-client.yaml
-rw------- 1 root root 449 Sep 24 22:52 kubecfg-kube-proxy.yaml
-rw------- 1 root root 465 Sep 24 22:52 kubecfg-kube-scheduler.yaml
Rancher agent logs
time="2021-09-24T20:52:27Z" level=info msg="Listening on /tmp/log.sock"
time="2021-09-24T20:52:27Z" level=info msg="Rancher agent version v2.6.0 is starting"
time="2021-09-24T20:52:27Z" level=info msg="Option customConfig=map[address:192.168.164.15 internalAddress: label:map[] roles:[etcd worker controlplane] taints:[]]"
time="2021-09-24T20:52:27Z" level=info msg="Option etcd=true"
time="2021-09-24T20:52:27Z" level=info msg="Option controlPlane=true"
time="2021-09-24T20:52:27Z" level=info msg="Option worker=true"
time="2021-09-24T20:52:27Z" level=info msg="Option requestedHostname=rkedev01"
time="2021-09-24T20:52:27Z" level=info msg="Option dockerInfo={SENJ:ECVU:QIJA:6TUO:NW3S:FATL:WAL6:FX37:Z7UK:WQWI:OJHT:RC23 1 1 0 0 1 overlay2 [[Backing Filesystem xfs] [Supports d_type true] [Native Overlay Diff true] [userxattr false]] [] {[local] [bridge host ipvlan macvlan null overlay] [] [awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} true false true true true true true true true true true true false 33 true 44 2021-09-24T22:52:27.013470837+02:00 json-file cgroupfs 1 0 5.3.18-24.78-default SUSE Linux Enterprise Server 15 SP2 15.2 linux x86_64 https://index.docker.io/v1/ 0xc00181b340 8 33183604736 [] /var/lib/docker lxrkedev01 [] false 20.10.6-ce map[io.containerd.runc.v2:{runc [] <nil>} io.containerd.runtime.v1.linux:{runc [] <nil>} oci:{/usr/sbin/docker-runc [] <nil>} runc:{runc [] <nil>}] runc { inactive false [] 0 0 <nil> []} false docker-init {05f951a3781f4f2c1911b05e61c160e9c30eaa8e 05f951a3781f4f2c1911b05e61c160e9c30eaa8e} {12644e614e25b05da6fd08a38ffa0cfe1903fdec 12644e614e25b05da6fd08a38ffa0cfe1903fdec} { } [name=apparmor name=seccomp,profile=default] [] [WARNING: No swap limit support]}"
time="2021-09-24T20:52:27Z" level=info msg="Connecting to wss://rancher.suedleasing.com/v3/connect/register with token starting with dlblp49gvqgh68cdt6zvb2tzpfl"
time="2021-09-24T20:52:27Z" level=info msg="Connecting to proxy" url="wss://rancher.suedleasing.com/v3/connect/register"
time="2021-09-24T20:52:27Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster is currently provisioning, or etcd, controlplane and worker node have to be registered"
time="2021-09-24T20:52:29Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster is currently provisioning, or etcd, controlplane and worker node have to be registered"
time="2021-09-24T20:52:31Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster is currently provisioning, or etcd, controlplane and worker node have to be registered"
time="2021-09-24T20:52:33Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster is currently provisioning, or etcd, controlplane and worker node have to be registered"
time="2021-09-24T20:52:35Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster is currently provisioning, or etcd, controlplane and worker node have to be registered"
time="2021-09-24T20:52:37Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster is currently provisioning, or etcd, controlplane and worker node have to be registered"
time="2021-09-24T20:52:39Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster is currently provisioning, or etcd, controlplane and worker node have to be registered"
time="2021-09-24T20:52:41Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster is currently provisioning, or etcd, controlplane and worker node have to be registered"
time="2021-09-24T20:52:43Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster is currently provisioning, or etcd, controlplane and worker node have to be registered"
time="2021-09-24T20:52:45Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster is currently provisioning, or etcd, controlplane and worker node have to be registered"
time="2021-09-24T20:52:47Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster is currently provisioning, or etcd, controlplane and worker node have to be registered"
time="2021-09-24T20:52:49Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster is currently provisioning, or etcd, controlplane and worker node have to be registered"
time="2021-09-24T20:52:51Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster is currently provisioning, or etcd, controlplane and worker node have to be registered"
time="2021-09-24T20:52:53Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster is currently provisioning, or etcd, controlplane and worker node have to be registered"
time="2021-09-24T20:52:55Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster is currently provisioning, or etcd, controlplane and worker node have to be registered"
time="2021-09-24T20:52:57Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster is currently provisioning, or etcd, controlplane and worker node have to be registered"
time="2021-09-24T20:52:59Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster is currently provisioning, or etcd, controlplane and worker node have to be registered"
time="2021-09-24T20:53:01Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster is currently provisioning, or etcd, controlplane and worker node have to be registered"
time="2021-09-24T20:53:03Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster is currently provisioning, or etcd, controlplane and worker node have to be registered"
time="2021-09-24T20:53:05Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster is currently provisioning, or etcd, controlplane and worker node have to be registered"
time="2021-09-24T2
(last message repeating every 2 seconds)
rkd-debug.log
time="2021-09-24T20:02:45Z" level=info msg="[rke_provider] rke cluster changed arguments: map[cluster_name:true kubernetes_version:true network:true nodes:true private_registries:true services:true]"
time="2021-09-24T20:03:03Z" level=info msg="[rke_provider] rke cluster changed arguments: map[cluster_name:true kubernetes_version:true network:true nodes:true private_registries:true services:true]"
time="2021-09-24T20:03:25Z" level=info msg="[rke_provider] rke cluster changed arguments: map[cluster_name:true kubernetes_version:true network:true nodes:true private_registries:true services:true]"
time="2021-09-24T20:17:09Z" level=info msg="[rke_provider] rke cluster changed arguments: map[cluster_name:true kubernetes_version:true network:true nodes:true private_registries:true services:true]"
time="2021-09-24T20:34:54Z" level=info msg="[rke_provider] rke cluster changed arguments: map[cluster_name:true kubernetes_version:true network:true nodes:true private_registries:true services:true]"
time="2021-09-24T20:38:52Z" level=info msg="[rke_provider] rke cluster changed arguments: map[cluster_name:true kubernetes_version:true network:true nodes:true private_registries:true services:true]"
time="2021-09-24T20:38:52Z" level=info msg="Creating RKE cluster..."
time="2021-09-24T20:38:52Z" level=info msg="Initiating Kubernetes cluster"
time="2021-09-24T20:38:52Z" level=info msg="[dialer] Setup tunnel for host [192.168.164.13]"
time="2021-09-24T20:38:52Z" level=info msg="[dialer] Setup tunnel for host [192.168.164.12]"
time="2021-09-24T20:38:52Z" level=info msg="[dialer] Setup tunnel for host [192.168.164.14]"
time="2021-09-24T20:38:52Z" level=info msg="Checking if container [cluster-state-deployer] is running on host [192.168.164.12], try #1"
time="2021-09-24T20:38:53Z" level=info msg="Pulling image [rancher/rke-tools:v0.1.75] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:00Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:39:01Z" level=info msg="Starting container [cluster-state-deployer] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:01Z" level=info msg="[state] Successfully started [cluster-state-deployer] container on host [192.168.164.12]"
time="2021-09-24T20:39:01Z" level=info msg="Checking if container [cluster-state-deployer] is running on host [192.168.164.13], try #1"
time="2021-09-24T20:39:01Z" level=info msg="Pulling image [rancher/rke-tools:v0.1.75] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:08Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:39:09Z" level=info msg="Starting container [cluster-state-deployer] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:09Z" level=info msg="[state] Successfully started [cluster-state-deployer] container on host [192.168.164.13]"
time="2021-09-24T20:39:09Z" level=info msg="Checking if container [cluster-state-deployer] is running on host [192.168.164.14], try #1"
time="2021-09-24T20:39:09Z" level=info msg="Pulling image [rancher/rke-tools:v0.1.75] on host [192.168.164.14], try #1"
time="2021-09-24T20:39:17Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:39:18Z" level=info msg="Starting container [cluster-state-deployer] on host [192.168.164.14], try #1"
time="2021-09-24T20:39:18Z" level=info msg="[state] Successfully started [cluster-state-deployer] container on host [192.168.164.14]"
time="2021-09-24T20:39:18Z" level=info msg="[certificates] Generating CA kubernetes certificates"
time="2021-09-24T20:39:18Z" level=info msg="[certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates"
time="2021-09-24T20:39:18Z" level=info msg="[certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates"
time="2021-09-24T20:39:18Z" level=info msg="[certificates] Generating Kubernetes API server certificates"
time="2021-09-24T20:39:19Z" level=info msg="[certificates] Generating Service account token key"
time="2021-09-24T20:39:19Z" level=info msg="[certificates] Generating Kube Controller certificates"
time="2021-09-24T20:39:19Z" level=info msg="[certificates] Generating Kube Scheduler certificates"
time="2021-09-24T20:39:20Z" level=info msg="[certificates] Generating Kube Proxy certificates"
time="2021-09-24T20:39:20Z" level=info msg="[certificates] Generating Node certificate"
time="2021-09-24T20:39:20Z" level=info msg="[certificates] Generating admin certificates and kubeconfig"
time="2021-09-24T20:39:20Z" level=info msg="[certificates] Generating Kubernetes API server proxy client certificates"
time="2021-09-24T20:39:20Z" level=info msg="[certificates] Generating kube-etcd-192-168-164-12 certificate and key"
time="2021-09-24T20:39:20Z" level=info msg="[certificates] Generating kube-etcd-192-168-164-13 certificate and key"
time="2021-09-24T20:39:20Z" level=info msg="[certificates] Generating kube-etcd-192-168-164-14 certificate and key"
time="2021-09-24T20:39:21Z" level=info msg="Successfully Deployed state file at [/rancher-new/rancher-master/terraform-provider-rke-tmp-168971691/cluster.rkestate]"
time="2021-09-24T20:39:21Z" level=info msg="Building Kubernetes cluster"
time="2021-09-24T20:39:21Z" level=info msg="[dialer] Setup tunnel for host [192.168.164.14]"
time="2021-09-24T20:39:21Z" level=info msg="[dialer] Setup tunnel for host [192.168.164.12]"
time="2021-09-24T20:39:21Z" level=info msg="[dialer] Setup tunnel for host [192.168.164.13]"
time="2021-09-24T20:39:21Z" level=info msg="[network] Deploying port listener containers"
time="2021-09-24T20:39:21Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:39:21Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:39:21Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:39:21Z" level=info msg="Starting container [rke-etcd-port-listener] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:21Z" level=info msg="Starting container [rke-etcd-port-listener] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:21Z" level=info msg="Starting container [rke-etcd-port-listener] on host [192.168.164.14], try #1"
time="2021-09-24T20:39:22Z" level=info msg="[network] Successfully started [rke-etcd-port-listener] container on host [192.168.164.13]"
time="2021-09-24T20:39:22Z" level=info msg="[network] Successfully started [rke-etcd-port-listener] container on host [192.168.164.12]"
time="2021-09-24T20:39:22Z" level=info msg="[network] Successfully started [rke-etcd-port-listener] container on host [192.168.164.14]"
time="2021-09-24T20:39:22Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:39:22Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:39:22Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:39:22Z" level=info msg="Starting container [rke-cp-port-listener] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:22Z" level=info msg="Starting container [rke-cp-port-listener] on host [192.168.164.14], try #1"
time="2021-09-24T20:39:22Z" level=info msg="Starting container [rke-cp-port-listener] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:23Z" level=info msg="[network] Successfully started [rke-cp-port-listener] container on host [192.168.164.13]"
time="2021-09-24T20:39:23Z" level=info msg="[network] Successfully started [rke-cp-port-listener] container on host [192.168.164.12]"
time="2021-09-24T20:39:23Z" level=info msg="[network] Successfully started [rke-cp-port-listener] container on host [192.168.164.14]"
time="2021-09-24T20:39:23Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:39:23Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:39:23Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:39:23Z" level=info msg="Starting container [rke-worker-port-listener] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:23Z" level=info msg="Starting container [rke-worker-port-listener] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:23Z" level=info msg="Starting container [rke-worker-port-listener] on host [192.168.164.14], try #1"
time="2021-09-24T20:39:24Z" level=info msg="[network] Successfully started [rke-worker-port-listener] container on host [192.168.164.13]"
time="2021-09-24T20:39:24Z" level=info msg="[network] Successfully started [rke-worker-port-listener] container on host [192.168.164.12]"
time="2021-09-24T20:39:24Z" level=info msg="[network] Successfully started [rke-worker-port-listener] container on host [192.168.164.14]"
time="2021-09-24T20:39:24Z" level=info msg="[network] Port listener containers deployed successfully"
time="2021-09-24T20:39:24Z" level=info msg="[network] Running etcd <-> etcd port checks"
time="2021-09-24T20:39:24Z" level=info msg="[network] Checking if host [192.168.164.12] can connect to host(s) [192.168.164.12 192.168.164.13 192.168.164.14] on port(s) [2379 2380], try #1"
time="2021-09-24T20:39:24Z" level=info msg="[network] Checking if host [192.168.164.13] can connect to host(s) [192.168.164.12 192.168.164.13 192.168.164.14] on port(s) [2379 2380], try #1"
time="2021-09-24T20:39:24Z" level=info msg="[network] Checking if host [192.168.164.14] can connect to host(s) [192.168.164.12 192.168.164.13 192.168.164.14] on port(s) [2379 2380], try #1"
time="2021-09-24T20:39:24Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:39:24Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:39:24Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:39:24Z" level=info msg="Starting container [rke-port-checker] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:24Z" level=info msg="Starting container [rke-port-checker] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:24Z" level=info msg="Starting container [rke-port-checker] on host [192.168.164.14], try #1"
time="2021-09-24T20:39:25Z" level=info msg="[network] Successfully started [rke-port-checker] container on host [192.168.164.13]"
time="2021-09-24T20:39:25Z" level=info msg="[network] Successfully started [rke-port-checker] container on host [192.168.164.12]"
time="2021-09-24T20:39:25Z" level=info msg="Removing container [rke-port-checker] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:25Z" level=info msg="Removing container [rke-port-checker] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:25Z" level=info msg="[network] Successfully started [rke-port-checker] container on host [192.168.164.14]"
time="2021-09-24T20:39:25Z" level=info msg="Removing container [rke-port-checker] on host [192.168.164.14], try #1"
time="2021-09-24T20:39:25Z" level=info msg="[network] Running control plane -> etcd port checks"
time="2021-09-24T20:39:25Z" level=info msg="[network] Checking if host [192.168.164.13] can connect to host(s) [192.168.164.12 192.168.164.13 192.168.164.14] on port(s) [2379], try #1"
time="2021-09-24T20:39:25Z" level=info msg="[network] Checking if host [192.168.164.12] can connect to host(s) [192.168.164.12 192.168.164.13 192.168.164.14] on port(s) [2379], try #1"
time="2021-09-24T20:39:25Z" level=info msg="[network] Checking if host [192.168.164.14] can connect to host(s) [192.168.164.12 192.168.164.13 192.168.164.14] on port(s) [2379], try #1"
time="2021-09-24T20:39:25Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:39:25Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:39:25Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:39:25Z" level=info msg="Starting container [rke-port-checker] on host [192.168.164.14], try #1"
time="2021-09-24T20:39:25Z" level=info msg="Starting container [rke-port-checker] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:25Z" level=info msg="Starting container [rke-port-checker] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:26Z" level=info msg="[network] Successfully started [rke-port-checker] container on host [192.168.164.13]"
time="2021-09-24T20:39:26Z" level=info msg="[network] Successfully started [rke-port-checker] container on host [192.168.164.12]"
time="2021-09-24T20:39:26Z" level=info msg="Removing container [rke-port-checker] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:26Z" level=info msg="Removing container [rke-port-checker] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:26Z" level=info msg="[network] Successfully started [rke-port-checker] container on host [192.168.164.14]"
time="2021-09-24T20:39:26Z" level=info msg="Removing container [rke-port-checker] on host [192.168.164.14], try #1"
time="2021-09-24T20:39:26Z" level=info msg="[network] Running control plane -> worker port checks"
time="2021-09-24T20:39:26Z" level=info msg="[network] Checking if host [192.168.164.13] can connect to host(s) [192.168.164.12 192.168.164.13 192.168.164.14] on port(s) [10250], try #1"
time="2021-09-24T20:39:26Z" level=info msg="[network] Checking if host [192.168.164.12] can connect to host(s) [192.168.164.12 192.168.164.13 192.168.164.14] on port(s) [10250], try #1"
time="2021-09-24T20:39:26Z" level=info msg="[network] Checking if host [192.168.164.14] can connect to host(s) [192.168.164.12 192.168.164.13 192.168.164.14] on port(s) [10250], try #1"
time="2021-09-24T20:39:26Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:39:26Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:39:26Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:39:26Z" level=info msg="Starting container [rke-port-checker] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:26Z" level=info msg="Starting container [rke-port-checker] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:26Z" level=info msg="Starting container [rke-port-checker] on host [192.168.164.14], try #1"
time="2021-09-24T20:39:26Z" level=info msg="[network] Successfully started [rke-port-checker] container on host [192.168.164.13]"
time="2021-09-24T20:39:27Z" level=info msg="[network] Successfully started [rke-port-checker] container on host [192.168.164.12]"
time="2021-09-24T20:39:27Z" level=info msg="Removing container [rke-port-checker] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:27Z" level=info msg="Removing container [rke-port-checker] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:27Z" level=info msg="[network] Successfully started [rke-port-checker] container on host [192.168.164.14]"
time="2021-09-24T20:39:27Z" level=info msg="Removing container [rke-port-checker] on host [192.168.164.14], try #1"
time="2021-09-24T20:39:27Z" level=info msg="[network] Running workers -> control plane port checks"
time="2021-09-24T20:39:27Z" level=info msg="[network] Checking if host [192.168.164.12] can connect to host(s) [192.168.164.12 192.168.164.13 192.168.164.14] on port(s) [6443], try #1"
time="2021-09-24T20:39:27Z" level=info msg="[network] Checking if host [192.168.164.13] can connect to host(s) [192.168.164.12 192.168.164.13 192.168.164.14] on port(s) [6443], try #1"
time="2021-09-24T20:39:27Z" level=info msg="[network] Checking if host [192.168.164.14] can connect to host(s) [192.168.164.12 192.168.164.13 192.168.164.14] on port(s) [6443], try #1"
time="2021-09-24T20:39:27Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:39:27Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:39:27Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:39:27Z" level=info msg="Starting container [rke-port-checker] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:27Z" level=info msg="Starting container [rke-port-checker] on host [192.168.164.14], try #1"
time="2021-09-24T20:39:27Z" level=info msg="Starting container [rke-port-checker] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:27Z" level=info msg="[network] Successfully started [rke-port-checker] container on host [192.168.164.12]"
time="2021-09-24T20:39:27Z" level=info msg="[network] Successfully started [rke-port-checker] container on host [192.168.164.13]"
time="2021-09-24T20:39:27Z" level=info msg="Removing container [rke-port-checker] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:28Z" level=info msg="Removing container [rke-port-checker] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:28Z" level=info msg="[network] Successfully started [rke-port-checker] container on host [192.168.164.14]"
time="2021-09-24T20:39:28Z" level=info msg="Removing container [rke-port-checker] on host [192.168.164.14], try #1"
time="2021-09-24T20:39:28Z" level=info msg="[network] Checking KubeAPI port Control Plane hosts"
time="2021-09-24T20:39:28Z" level=info msg="[network] Removing port listener containers"
time="2021-09-24T20:39:28Z" level=info msg="Removing container [rke-etcd-port-listener] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:28Z" level=info msg="Removing container [rke-etcd-port-listener] on host [192.168.164.14], try #1"
time="2021-09-24T20:39:28Z" level=info msg="Removing container [rke-etcd-port-listener] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:28Z" level=info msg="[remove/rke-etcd-port-listener] Successfully removed container on host [192.168.164.13]"
time="2021-09-24T20:39:28Z" level=info msg="[remove/rke-etcd-port-listener] Successfully removed container on host [192.168.164.12]"
time="2021-09-24T20:39:28Z" level=info msg="[remove/rke-etcd-port-listener] Successfully removed container on host [192.168.164.14]"
time="2021-09-24T20:39:28Z" level=info msg="Removing container [rke-cp-port-listener] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:28Z" level=info msg="Removing container [rke-cp-port-listener] on host [192.168.164.14], try #1"
time="2021-09-24T20:39:28Z" level=info msg="Removing container [rke-cp-port-listener] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:29Z" level=info msg="[remove/rke-cp-port-listener] Successfully removed container on host [192.168.164.12]"
time="2021-09-24T20:39:29Z" level=info msg="[remove/rke-cp-port-listener] Successfully removed container on host [192.168.164.13]"
time="2021-09-24T20:39:29Z" level=info msg="[remove/rke-cp-port-listener] Successfully removed container on host [192.168.164.14]"
time="2021-09-24T20:39:29Z" level=info msg="Removing container [rke-worker-port-listener] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:29Z" level=info msg="Removing container [rke-worker-port-listener] on host [192.168.164.14], try #1"
time="2021-09-24T20:39:29Z" level=info msg="Removing container [rke-worker-port-listener] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:29Z" level=info msg="[remove/rke-worker-port-listener] Successfully removed container on host [192.168.164.13]"
time="2021-09-24T20:39:29Z" level=info msg="[remove/rke-worker-port-listener] Successfully removed container on host [192.168.164.12]"
time="2021-09-24T20:39:29Z" level=info msg="[remove/rke-worker-port-listener] Successfully removed container on host [192.168.164.14]"
time="2021-09-24T20:39:29Z" level=info msg="[network] Port listener containers removed successfully"
time="2021-09-24T20:39:29Z" level=info msg="[certificates] Deploying kubernetes certificates to Cluster nodes"
time="2021-09-24T20:39:29Z" level=info msg="Checking if container [cert-deployer] is running on host [192.168.164.13], try #1"
time="2021-09-24T20:39:29Z" level=info msg="Checking if container [cert-deployer] is running on host [192.168.164.12], try #1"
time="2021-09-24T20:39:29Z" level=info msg="Checking if container [cert-deployer] is running on host [192.168.164.14], try #1"
time="2021-09-24T20:39:29Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:39:29Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:39:29Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:39:29Z" level=info msg="Starting container [cert-deployer] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:29Z" level=info msg="Starting container [cert-deployer] on host [192.168.164.14], try #1"
time="2021-09-24T20:39:29Z" level=info msg="Starting container [cert-deployer] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:29Z" level=info msg="Checking if container [cert-deployer] is running on host [192.168.164.13], try #1"
time="2021-09-24T20:39:30Z" level=info msg="Checking if container [cert-deployer] is running on host [192.168.164.12], try #1"
time="2021-09-24T20:39:30Z" level=info msg="Checking if container [cert-deployer] is running on host [192.168.164.14], try #1"
time="2021-09-24T20:39:34Z" level=info msg="Checking if container [cert-deployer] is running on host [192.168.164.13], try #1"
time="2021-09-24T20:39:34Z" level=info msg="Removing container [cert-deployer] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:35Z" level=info msg="Checking if container [cert-deployer] is running on host [192.168.164.12], try #1"
time="2021-09-24T20:39:35Z" level=info msg="Removing container [cert-deployer] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:35Z" level=info msg="Checking if container [cert-deployer] is running on host [192.168.164.14], try #1"
time="2021-09-24T20:39:35Z" level=info msg="Removing container [cert-deployer] on host [192.168.164.14], try #1"
time="2021-09-24T20:39:35Z" level=info msg="[reconcile] Rebuilding and updating local kube config"
time="2021-09-24T20:39:35Z" level=info msg="Successfully Deployed local admin kubeconfig at [/rancher-new/rancher-master/terraform-provider-rke-tmp-168971691/kube_config_cluster.yml]"
time="2021-09-24T20:39:35Z" level=warning msg="[reconcile] host [192.168.164.12] is a control plane node without reachable Kubernetes API endpoint in the cluster"
time="2021-09-24T20:39:35Z" level=info msg="Successfully Deployed local admin kubeconfig at [/rancher-new/rancher-master/terraform-provider-rke-tmp-168971691/kube_config_cluster.yml]"
time="2021-09-24T20:39:35Z" level=warning msg="[reconcile] host [192.168.164.13] is a control plane node without reachable Kubernetes API endpoint in the cluster"
time="2021-09-24T20:39:35Z" level=info msg="Successfully Deployed local admin kubeconfig at [/rancher-new/rancher-master/terraform-provider-rke-tmp-168971691/kube_config_cluster.yml]"
time="2021-09-24T20:39:35Z" level=warning msg="[reconcile] host [192.168.164.14] is a control plane node without reachable Kubernetes API endpoint in the cluster"
time="2021-09-24T20:39:35Z" level=warning msg="[reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found"
time="2021-09-24T20:39:35Z" level=info msg="[certificates] Successfully deployed kubernetes certificates to Cluster nodes"
time="2021-09-24T20:39:35Z" level=info msg="[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [192.168.164.12]"
time="2021-09-24T20:39:35Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:39:35Z" level=info msg="Starting container [file-deployer] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:36Z" level=info msg="Successfully started [file-deployer] container on host [192.168.164.12]"
time="2021-09-24T20:39:36Z" level=info msg="Waiting for [file-deployer] container to exit on host [192.168.164.12]"
time="2021-09-24T20:39:36Z" level=info msg="Waiting for [file-deployer] container to exit on host [192.168.164.12]"
time="2021-09-24T20:39:36Z" level=info msg="Container [file-deployer] is still running on host [192.168.164.12]: stderr: [], stdout: []"
time="2021-09-24T20:39:37Z" level=info msg="Waiting for [file-deployer] container to exit on host [192.168.164.12]"
time="2021-09-24T20:39:37Z" level=info msg="Removing container [file-deployer] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:37Z" level=info msg="[remove/file-deployer] Successfully removed container on host [192.168.164.12]"
time="2021-09-24T20:39:37Z" level=info msg="[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [192.168.164.13]"
time="2021-09-24T20:39:37Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:39:37Z" level=info msg="Starting container [file-deployer] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:37Z" level=info msg="Successfully started [file-deployer] container on host [192.168.164.13]"
time="2021-09-24T20:39:37Z" level=info msg="Waiting for [file-deployer] container to exit on host [192.168.164.13]"
time="2021-09-24T20:39:37Z" level=info msg="Waiting for [file-deployer] container to exit on host [192.168.164.13]"
time="2021-09-24T20:39:38Z" level=info msg="Removing container [file-deployer] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:38Z" level=info msg="[remove/file-deployer] Successfully removed container on host [192.168.164.13]"
time="2021-09-24T20:39:38Z" level=info msg="[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [192.168.164.14]"
time="2021-09-24T20:39:38Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:39:38Z" level=info msg="Starting container [file-deployer] on host [192.168.164.14], try #1"
time="2021-09-24T20:39:38Z" level=info msg="Successfully started [file-deployer] container on host [192.168.164.14]"
time="2021-09-24T20:39:38Z" level=info msg="Waiting for [file-deployer] container to exit on host [192.168.164.14]"
time="2021-09-24T20:39:38Z" level=info msg="Waiting for [file-deployer] container to exit on host [192.168.164.14]"
time="2021-09-24T20:39:39Z" level=info msg="Removing container [file-deployer] on host [192.168.164.14], try #1"
time="2021-09-24T20:39:39Z" level=info msg="[remove/file-deployer] Successfully removed container on host [192.168.164.14]"
time="2021-09-24T20:39:39Z" level=info msg="[/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes"
time="2021-09-24T20:39:39Z" level=info msg="[reconcile] Reconciling cluster state"
time="2021-09-24T20:39:39Z" level=info msg="[reconcile] This is newly generated cluster"
time="2021-09-24T20:39:39Z" level=info msg="Pre-pulling kubernetes images"
time="2021-09-24T20:39:39Z" level=info msg="Pulling image [rancher/hyperkube:v1.20.8-rancher1] on host [192.168.164.13], try #1"
time="2021-09-24T20:39:39Z" level=info msg="Pulling image [rancher/hyperkube:v1.20.8-rancher1] on host [192.168.164.12], try #1"
time="2021-09-24T20:39:39Z" level=info msg="Pulling image [rancher/hyperkube:v1.20.8-rancher1] on host [192.168.164.14], try #1"
time="2021-09-24T20:40:55Z" level=info msg="Image [rancher/hyperkube:v1.20.8-rancher1] exists on host [192.168.164.13]"
time="2021-09-24T20:40:57Z" level=info msg="Image [rancher/hyperkube:v1.20.8-rancher1] exists on host [192.168.164.12]"
time="2021-09-24T20:41:12Z" level=info msg="Image [rancher/hyperkube:v1.20.8-rancher1] exists on host [192.168.164.14]"
time="2021-09-24T20:41:12Z" level=info msg="Kubernetes images pulled successfully"
time="2021-09-24T20:41:12Z" level=info msg="[etcd] Building up etcd plane.."
time="2021-09-24T20:41:12Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:41:12Z" level=info msg="Starting container [etcd-fix-perm] on host [192.168.164.12], try #1"
time="2021-09-24T20:41:13Z" level=info msg="Successfully started [etcd-fix-perm] container on host [192.168.164.12]"
time="2021-09-24T20:41:13Z" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [192.168.164.12]"
time="2021-09-24T20:41:13Z" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [192.168.164.12]"
time="2021-09-24T20:41:13Z" level=info msg="Removing container [etcd-fix-perm] on host [192.168.164.12], try #1"
time="2021-09-24T20:41:13Z" level=info msg="[remove/etcd-fix-perm] Successfully removed container on host [192.168.164.12]"
time="2021-09-24T20:41:13Z" level=info msg="Pulling image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] on host [192.168.164.12], try #1"
time="2021-09-24T20:41:18Z" level=info msg="Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [192.168.164.12]"
time="2021-09-24T20:41:19Z" level=info msg="Starting container [etcd] on host [192.168.164.12], try #1"
time="2021-09-24T20:41:19Z" level=info msg="[etcd] Successfully started [etcd] container on host [192.168.164.12]"
time="2021-09-24T20:41:19Z" level=info msg="[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [192.168.164.12]"
time="2021-09-24T20:41:19Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:41:19Z" level=info msg="Starting container [etcd-rolling-snapshots] on host [192.168.164.12], try #1"
time="2021-09-24T20:41:19Z" level=info msg="[etcd] Successfully started [etcd-rolling-snapshots] container on host [192.168.164.12]"
time="2021-09-24T20:41:25Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:41:25Z" level=info msg="Starting container [rke-bundle-cert] on host [192.168.164.12], try #1"
time="2021-09-24T20:41:25Z" level=info msg="[certificates] Successfully started [rke-bundle-cert] container on host [192.168.164.12]"
time="2021-09-24T20:41:25Z" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [192.168.164.12]"
time="2021-09-24T20:41:25Z" level=info msg="[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [192.168.164.12]"
time="2021-09-24T20:41:25Z" level=info msg="Removing container [rke-bundle-cert] on host [192.168.164.12], try #1"
time="2021-09-24T20:41:25Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:41:25Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.12], try #1"
time="2021-09-24T20:41:26Z" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [192.168.164.12]"
time="2021-09-24T20:41:26Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.12], try #1"
time="2021-09-24T20:41:26Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.12]"
time="2021-09-24T20:41:26Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:41:26Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.12], try #1"
time="2021-09-24T20:41:26Z" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [192.168.164.12]"
time="2021-09-24T20:41:27Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.12], try #1"
time="2021-09-24T20:41:27Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.12]"
time="2021-09-24T20:41:27Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:41:27Z" level=info msg="Starting container [etcd-fix-perm] on host [192.168.164.13], try #1"
time="2021-09-24T20:41:28Z" level=info msg="Successfully started [etcd-fix-perm] container on host [192.168.164.13]"
time="2021-09-24T20:41:28Z" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [192.168.164.13]"
time="2021-09-24T20:41:28Z" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [192.168.164.13]"
time="2021-09-24T20:41:28Z" level=info msg="Container [etcd-fix-perm] is still running on host [192.168.164.13]: stderr: [], stdout: []"
time="2021-09-24T20:41:29Z" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [192.168.164.13]"
time="2021-09-24T20:41:29Z" level=info msg="Removing container [etcd-fix-perm] on host [192.168.164.13], try #1"
time="2021-09-24T20:41:29Z" level=info msg="[remove/etcd-fix-perm] Successfully removed container on host [192.168.164.13]"
time="2021-09-24T20:41:29Z" level=info msg="Pulling image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] on host [192.168.164.13], try #1"
time="2021-09-24T20:41:34Z" level=info msg="Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [192.168.164.13]"
time="2021-09-24T20:41:34Z" level=info msg="Starting container [etcd] on host [192.168.164.13], try #1"
time="2021-09-24T20:41:34Z" level=info msg="[etcd] Successfully started [etcd] container on host [192.168.164.13]"
time="2021-09-24T20:41:34Z" level=info msg="[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [192.168.164.13]"
time="2021-09-24T20:41:34Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:41:34Z" level=info msg="Starting container [etcd-rolling-snapshots] on host [192.168.164.13], try #1"
time="2021-09-24T20:41:35Z" level=info msg="[etcd] Successfully started [etcd-rolling-snapshots] container on host [192.168.164.13]"
time="2021-09-24T20:41:40Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:41:40Z" level=info msg="Starting container [rke-bundle-cert] on host [192.168.164.13], try #1"
time="2021-09-24T20:41:40Z" level=info msg="[certificates] Successfully started [rke-bundle-cert] container on host [192.168.164.13]"
time="2021-09-24T20:41:40Z" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [192.168.164.13]"
time="2021-09-24T20:41:40Z" level=info msg="[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [192.168.164.13]"
time="2021-09-24T20:41:40Z" level=info msg="Removing container [rke-bundle-cert] on host [192.168.164.13], try #1"
time="2021-09-24T20:41:40Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:41:40Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.13], try #1"
time="2021-09-24T20:41:41Z" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [192.168.164.13]"
time="2021-09-24T20:41:41Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.13], try #1"
time="2021-09-24T20:41:41Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.13]"
time="2021-09-24T20:41:41Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:41:41Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.13], try #1"
time="2021-09-24T20:41:41Z" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [192.168.164.13]"
time="2021-09-24T20:41:41Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.13], try #1"
time="2021-09-24T20:41:42Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.13]"
time="2021-09-24T20:41:42Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:41:42Z" level=info msg="Starting container [etcd-fix-perm] on host [192.168.164.14], try #1"
time="2021-09-24T20:41:43Z" level=info msg="Successfully started [etcd-fix-perm] container on host [192.168.164.14]"
time="2021-09-24T20:41:43Z" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [192.168.164.14]"
time="2021-09-24T20:41:43Z" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [192.168.164.14]"
time="2021-09-24T20:41:43Z" level=info msg="Container [etcd-fix-perm] is still running on host [192.168.164.14]: stderr: [], stdout: []"
time="2021-09-24T20:41:44Z" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [192.168.164.14]"
time="2021-09-24T20:41:44Z" level=info msg="Removing container [etcd-fix-perm] on host [192.168.164.14], try #1"
time="2021-09-24T20:41:44Z" level=info msg="[remove/etcd-fix-perm] Successfully removed container on host [192.168.164.14]"
time="2021-09-24T20:41:44Z" level=info msg="Pulling image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] on host [192.168.164.14], try #1"
time="2021-09-24T20:41:50Z" level=info msg="Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [192.168.164.14]"
time="2021-09-24T20:41:50Z" level=info msg="Starting container [etcd] on host [192.168.164.14], try #1"
time="2021-09-24T20:41:51Z" level=info msg="[etcd] Successfully started [etcd] container on host [192.168.164.14]"
time="2021-09-24T20:41:51Z" level=info msg="[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [192.168.164.14]"
time="2021-09-24T20:41:51Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:41:51Z" level=info msg="Starting container [etcd-rolling-snapshots] on host [192.168.164.14], try #1"
time="2021-09-24T20:41:52Z" level=info msg="[etcd] Successfully started [etcd-rolling-snapshots] container on host [192.168.164.14]"
time="2021-09-24T20:41:57Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:41:57Z" level=info msg="Starting container [rke-bundle-cert] on host [192.168.164.14], try #1"
time="2021-09-24T20:41:57Z" level=info msg="[certificates] Successfully started [rke-bundle-cert] container on host [192.168.164.14]"
time="2021-09-24T20:41:57Z" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [192.168.164.14]"
time="2021-09-24T20:41:58Z" level=info msg="Container [rke-bundle-cert] is still running on host [192.168.164.14]: stderr: [], stdout: []"
time="2021-09-24T20:41:59Z" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [192.168.164.14]"
time="2021-09-24T20:41:59Z" level=info msg="[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [192.168.164.14]"
time="2021-09-24T20:41:59Z" level=info msg="Removing container [rke-bundle-cert] on host [192.168.164.14], try #1"
time="2021-09-24T20:41:59Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:41:59Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.14], try #1"
time="2021-09-24T20:41:59Z" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [192.168.164.14]"
time="2021-09-24T20:41:59Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.14], try #1"
time="2021-09-24T20:42:00Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.14]"
time="2021-09-24T20:42:00Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:42:00Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.14], try #1"
time="2021-09-24T20:42:00Z" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [192.168.164.14]"
time="2021-09-24T20:42:00Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.14], try #1"
time="2021-09-24T20:42:00Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.14]"
time="2021-09-24T20:42:00Z" level=info msg="[etcd] Successfully started etcd plane.. Checking etcd cluster health"
time="2021-09-24T20:42:01Z" level=info msg="[etcd] etcd host [192.168.164.12] reported healthy=true"
time="2021-09-24T20:42:01Z" level=info msg="[controlplane] Building up Controller Plane.."
time="2021-09-24T20:42:01Z" level=info msg="Checking if container [service-sidekick] is running on host [192.168.164.12], try #1"
time="2021-09-24T20:42:01Z" level=info msg="Checking if container [service-sidekick] is running on host [192.168.164.13], try #1"
time="2021-09-24T20:42:01Z" level=info msg="Checking if container [service-sidekick] is running on host [192.168.164.14], try #1"
time="2021-09-24T20:42:01Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:42:01Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:42:01Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:42:01Z" level=info msg="Image [rancher/hyperkube:v1.20.8-rancher1] exists on host [192.168.164.13]"
time="2021-09-24T20:42:01Z" level=info msg="Image [rancher/hyperkube:v1.20.8-rancher1] exists on host [192.168.164.14]"
time="2021-09-24T20:42:01Z" level=info msg="Starting container [kube-apiserver] on host [192.168.164.13], try #1"
time="2021-09-24T20:42:01Z" level=info msg="Image [rancher/hyperkube:v1.20.8-rancher1] exists on host [192.168.164.12]"
time="2021-09-24T20:42:01Z" level=info msg="Starting container [kube-apiserver] on host [192.168.164.14], try #1"
time="2021-09-24T20:42:01Z" level=info msg="Starting container [kube-apiserver] on host [192.168.164.12], try #1"
time="2021-09-24T20:42:01Z" level=info msg="[controlplane] Successfully started [kube-apiserver] container on host [192.168.164.13]"
time="2021-09-24T20:42:01Z" level=info msg="[healthcheck] Start Healthcheck on service [kube-apiserver] on host [192.168.164.13]"
time="2021-09-24T20:42:02Z" level=info msg="[controlplane] Successfully started [kube-apiserver] container on host [192.168.164.12]"
time="2021-09-24T20:42:02Z" level=info msg="[healthcheck] Start Healthcheck on service [kube-apiserver] on host [192.168.164.12]"
time="2021-09-24T20:42:02Z" level=info msg="[controlplane] Successfully started [kube-apiserver] container on host [192.168.164.14]"
time="2021-09-24T20:42:02Z" level=info msg="[healthcheck] Start Healthcheck on service [kube-apiserver] on host [192.168.164.14]"
time="2021-09-24T20:42:10Z" level=info msg="[healthcheck] service [kube-apiserver] on host [192.168.164.13] is healthy"
time="2021-09-24T20:42:10Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:42:10Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.13], try #1"
time="2021-09-24T20:42:10Z" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [192.168.164.13]"
time="2021-09-24T20:42:11Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.13], try #1"
time="2021-09-24T20:42:11Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.13]"
time="2021-09-24T20:42:11Z" level=info msg="Image [rancher/hyperkube:v1.20.8-rancher1] exists on host [192.168.164.13]"
time="2021-09-24T20:42:11Z" level=info msg="[healthcheck] service [kube-apiserver] on host [192.168.164.12] is healthy"
time="2021-09-24T20:42:11Z" level=info msg="Starting container [kube-controller-manager] on host [192.168.164.13], try #1"
time="2021-09-24T20:42:11Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:42:11Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.12], try #1"
time="2021-09-24T20:42:11Z" level=info msg="[controlplane] Successfully started [kube-controller-manager] container on host [192.168.164.13]"
time="2021-09-24T20:42:11Z" level=info msg="[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [192.168.164.13]"
time="2021-09-24T20:42:11Z" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [192.168.164.12]"
time="2021-09-24T20:42:11Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.12], try #1"
time="2021-09-24T20:42:11Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.12]"
time="2021-09-24T20:42:12Z" level=info msg="Image [rancher/hyperkube:v1.20.8-rancher1] exists on host [192.168.164.12]"
time="2021-09-24T20:42:12Z" level=info msg="Starting container [kube-controller-manager] on host [192.168.164.12], try #1"
time="2021-09-24T20:42:12Z" level=info msg="[controlplane] Successfully started [kube-controller-manager] container on host [192.168.164.12]"
time="2021-09-24T20:42:12Z" level=info msg="[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [192.168.164.12]"
time="2021-09-24T20:42:15Z" level=info msg="[healthcheck] service [kube-apiserver] on host [192.168.164.14] is healthy"
time="2021-09-24T20:42:15Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:42:15Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.14], try #1"
time="2021-09-24T20:42:15Z" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [192.168.164.14]"
time="2021-09-24T20:42:15Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.14], try #1"
time="2021-09-24T20:42:15Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.14]"
time="2021-09-24T20:42:16Z" level=info msg="Image [rancher/hyperkube:v1.20.8-rancher1] exists on host [192.168.164.14]"
time="2021-09-24T20:42:16Z" level=info msg="Starting container [kube-controller-manager] on host [192.168.164.14], try #1"
time="2021-09-24T20:42:16Z" level=info msg="[controlplane] Successfully started [kube-controller-manager] container on host [192.168.164.14]"
time="2021-09-24T20:42:16Z" level=info msg="[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [192.168.164.14]"
time="2021-09-24T20:42:17Z" level=info msg="[healthcheck] service [kube-controller-manager] on host [192.168.164.13] is healthy"
time="2021-09-24T20:42:17Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:42:17Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.13], try #1"
time="2021-09-24T20:42:17Z" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [192.168.164.13]"
time="2021-09-24T20:42:17Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.13], try #1"
time="2021-09-24T20:42:17Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.13]"
time="2021-09-24T20:42:17Z" level=info msg="Image [rancher/hyperkube:v1.20.8-rancher1] exists on host [192.168.164.13]"
time="2021-09-24T20:42:18Z" level=info msg="Starting container [kube-scheduler] on host [192.168.164.13], try #1"
time="2021-09-24T20:42:18Z" level=info msg="[healthcheck] service [kube-controller-manager] on host [192.168.164.12] is healthy"
time="2021-09-24T20:42:18Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:42:18Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.12], try #1"
time="2021-09-24T20:42:18Z" level=info msg="[controlplane] Successfully started [kube-scheduler] container on host [192.168.164.13]"
time="2021-09-24T20:42:18Z" level=info msg="[healthcheck] Start Healthcheck on service [kube-scheduler] on host [192.168.164.13]"
time="2021-09-24T20:42:18Z" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [192.168.164.12]"
time="2021-09-24T20:42:18Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.12], try #1"
time="2021-09-24T20:42:18Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.12]"
time="2021-09-24T20:42:18Z" level=info msg="Image [rancher/hyperkube:v1.20.8-rancher1] exists on host [192.168.164.12]"
time="2021-09-24T20:42:19Z" level=info msg="Starting container [kube-scheduler] on host [192.168.164.12], try #1"
time="2021-09-24T20:42:19Z" level=info msg="[controlplane] Successfully started [kube-scheduler] container on host [192.168.164.12]"
time="2021-09-24T20:42:19Z" level=info msg="[healthcheck] Start Healthcheck on service [kube-scheduler] on host [192.168.164.12]"
time="2021-09-24T20:42:20Z" level=info msg="[healthcheck] service [kube-scheduler] on host [192.168.164.12] is healthy"
time="2021-09-24T20:42:20Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:42:20Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.12], try #1"
time="2021-09-24T20:42:20Z" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [192.168.164.12]"
time="2021-09-24T20:42:20Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.12], try #1"
time="2021-09-24T20:42:20Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.12]"
time="2021-09-24T20:42:22Z" level=info msg="[healthcheck] service [kube-controller-manager] on host [192.168.164.14] is healthy"
time="2021-09-24T20:42:22Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:42:22Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.14], try #1"
time="2021-09-24T20:42:23Z" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [192.168.164.14]"
time="2021-09-24T20:42:23Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.14], try #1"
time="2021-09-24T20:42:23Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.14]"
time="2021-09-24T20:42:23Z" level=info msg="Image [rancher/hyperkube:v1.20.8-rancher1] exists on host [192.168.164.14]"
time="2021-09-24T20:42:23Z" level=info msg="Starting container [kube-scheduler] on host [192.168.164.14], try #1"
time="2021-09-24T20:42:23Z" level=info msg="[healthcheck] service [kube-scheduler] on host [192.168.164.13] is healthy"
time="2021-09-24T20:42:23Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:42:24Z" level=info msg="[controlplane] Successfully started [kube-scheduler] container on host [192.168.164.14]"
time="2021-09-24T20:42:24Z" level=info msg="[healthcheck] Start Healthcheck on service [kube-scheduler] on host [192.168.164.14]"
time="2021-09-24T20:42:24Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.13], try #1"
time="2021-09-24T20:42:24Z" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [192.168.164.13]"
time="2021-09-24T20:42:24Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.13], try #1"
time="2021-09-24T20:42:24Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.13]"
time="2021-09-24T20:42:29Z" level=info msg="[healthcheck] service [kube-scheduler] on host [192.168.164.14] is healthy"
time="2021-09-24T20:42:29Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:42:29Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.14], try #1"
time="2021-09-24T20:42:30Z" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [192.168.164.14]"
time="2021-09-24T20:42:30Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.14], try #1"
time="2021-09-24T20:42:30Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.14]"
time="2021-09-24T20:42:30Z" level=info msg="[controlplane] Successfully started Controller Plane.."
time="2021-09-24T20:42:30Z" level=info msg="[authz] Creating rke-job-deployer ServiceAccount"
time="2021-09-24T20:42:30Z" level=info msg="[authz] rke-job-deployer ServiceAccount created successfully"
time="2021-09-24T20:42:30Z" level=info msg="[authz] Creating system:node ClusterRoleBinding"
time="2021-09-24T20:42:30Z" level=info msg="[authz] system:node ClusterRoleBinding created successfully"
time="2021-09-24T20:42:30Z" level=info msg="[authz] Creating kube-apiserver proxy ClusterRole and ClusterRoleBinding"
time="2021-09-24T20:42:30Z" level=info msg="[authz] kube-apiserver proxy ClusterRole and ClusterRoleBinding created successfully"
time="2021-09-24T20:42:30Z" level=info msg="Successfully Deployed state file at [/rancher-new/rancher-master/terraform-provider-rke-tmp-168971691/cluster.rkestate]"
time="2021-09-24T20:42:30Z" level=info msg="[state] Saving full cluster state to Kubernetes"
time="2021-09-24T20:42:31Z" level=info msg="[state] Successfully Saved full cluster state to Kubernetes ConfigMap: full-cluster-state"
time="2021-09-24T20:42:31Z" level=info msg="[worker] Building up Worker Plane.."
time="2021-09-24T20:42:31Z" level=info msg="Checking if container [service-sidekick] is running on host [192.168.164.12], try #1"
time="2021-09-24T20:42:31Z" level=info msg="Checking if container [service-sidekick] is running on host [192.168.164.14], try #1"
time="2021-09-24T20:42:31Z" level=info msg="Checking if container [service-sidekick] is running on host [192.168.164.13], try #1"
time="2021-09-24T20:42:31Z" level=info msg="[sidekick] Sidekick container already created on host [192.168.164.12]"
time="2021-09-24T20:42:31Z" level=info msg="[sidekick] Sidekick container already created on host [192.168.164.14]"
time="2021-09-24T20:42:31Z" level=info msg="[sidekick] Sidekick container already created on host [192.168.164.13]"
time="2021-09-24T20:42:31Z" level=info msg="Image [rancher/hyperkube:v1.20.8-rancher1] exists on host [192.168.164.12]"
time="2021-09-24T20:42:31Z" level=info msg="Image [rancher/hyperkube:v1.20.8-rancher1] exists on host [192.168.164.14]"
time="2021-09-24T20:42:31Z" level=info msg="Image [rancher/hyperkube:v1.20.8-rancher1] exists on host [192.168.164.13]"
time="2021-09-24T20:42:31Z" level=info msg="Starting container [kubelet] on host [192.168.164.12], try #1"
time="2021-09-24T20:42:31Z" level=info msg="Starting container [kubelet] on host [192.168.164.14], try #1"
time="2021-09-24T20:42:31Z" level=info msg="Starting container [kubelet] on host [192.168.164.13], try #1"
time="2021-09-24T20:42:31Z" level=info msg="[worker] Successfully started [kubelet] container on host [192.168.164.12]"
time="2021-09-24T20:42:31Z" level=info msg="[healthcheck] Start Healthcheck on service [kubelet] on host [192.168.164.12]"
time="2021-09-24T20:42:31Z" level=info msg="[worker] Successfully started [kubelet] container on host [192.168.164.13]"
time="2021-09-24T20:42:31Z" level=info msg="[healthcheck] Start Healthcheck on service [kubelet] on host [192.168.164.13]"
time="2021-09-24T20:42:31Z" level=info msg="[worker] Successfully started [kubelet] container on host [192.168.164.14]"
time="2021-09-24T20:42:31Z" level=info msg="[healthcheck] Start Healthcheck on service [kubelet] on host [192.168.164.14]"
time="2021-09-24T20:42:42Z" level=info msg="[healthcheck] service [kubelet] on host [192.168.164.12] is healthy"
time="2021-09-24T20:42:42Z" level=info msg="[healthcheck] service [kubelet] on host [192.168.164.13] is healthy"
time="2021-09-24T20:42:42Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:42:42Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:42:42Z" level=info msg="[healthcheck] service [kubelet] on host [192.168.164.14] is healthy"
time="2021-09-24T20:42:42Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.12], try #1"
time="2021-09-24T20:42:42Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.13], try #1"
time="2021-09-24T20:42:42Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:42:42Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.14], try #1"
time="2021-09-24T20:42:42Z" level=info msg="[worker] Successfully started [rke-log-linker] container on host [192.168.164.12]"
time="2021-09-24T20:42:42Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.12], try #1"
time="2021-09-24T20:42:42Z" level=info msg="[worker] Successfully started [rke-log-linker] container on host [192.168.164.13]"
time="2021-09-24T20:42:42Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.13], try #1"
time="2021-09-24T20:42:43Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.12]"
time="2021-09-24T20:42:43Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.13]"
time="2021-09-24T20:42:43Z" level=info msg="Image [rancher/hyperkube:v1.20.8-rancher1] exists on host [192.168.164.12]"
time="2021-09-24T20:42:43Z" level=info msg="Image [rancher/hyperkube:v1.20.8-rancher1] exists on host [192.168.164.13]"
time="2021-09-24T20:42:43Z" level=info msg="Starting container [kube-proxy] on host [192.168.164.12], try #1"
time="2021-09-24T20:42:43Z" level=info msg="Starting container [kube-proxy] on host [192.168.164.13], try #1"
time="2021-09-24T20:42:43Z" level=info msg="[worker] Successfully started [rke-log-linker] container on host [192.168.164.14]"
time="2021-09-24T20:42:43Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.14], try #1"
time="2021-09-24T20:42:43Z" level=info msg="[worker] Successfully started [kube-proxy] container on host [192.168.164.12]"
time="2021-09-24T20:42:43Z" level=info msg="[healthcheck] Start Healthcheck on service [kube-proxy] on host [192.168.164.12]"
time="2021-09-24T20:42:43Z" level=info msg="[worker] Successfully started [kube-proxy] container on host [192.168.164.13]"
time="2021-09-24T20:42:43Z" level=info msg="[healthcheck] Start Healthcheck on service [kube-proxy] on host [192.168.164.13]"
time="2021-09-24T20:42:43Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.14]"
time="2021-09-24T20:42:43Z" level=info msg="Image [rancher/hyperkube:v1.20.8-rancher1] exists on host [192.168.164.14]"
time="2021-09-24T20:42:43Z" level=info msg="Starting container [kube-proxy] on host [192.168.164.14], try #1"
time="2021-09-24T20:42:43Z" level=info msg="[healthcheck] service [kube-proxy] on host [192.168.164.12] is healthy"
time="2021-09-24T20:42:43Z" level=info msg="[healthcheck] service [kube-proxy] on host [192.168.164.13] is healthy"
time="2021-09-24T20:42:43Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:42:43Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:42:43Z" level=info msg="[worker] Successfully started [kube-proxy] container on host [192.168.164.14]"
time="2021-09-24T20:42:43Z" level=info msg="[healthcheck] Start Healthcheck on service [kube-proxy] on host [192.168.164.14]"
time="2021-09-24T20:42:43Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.13], try #1"
time="2021-09-24T20:42:43Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.12], try #1"
time="2021-09-24T20:42:44Z" level=info msg="[worker] Successfully started [rke-log-linker] container on host [192.168.164.13]"
time="2021-09-24T20:42:44Z" level=info msg="[healthcheck] service [kube-proxy] on host [192.168.164.14] is healthy"
time="2021-09-24T20:42:44Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.13], try #1"
time="2021-09-24T20:42:44Z" level=info msg="[worker] Successfully started [rke-log-linker] container on host [192.168.164.12]"
time="2021-09-24T20:42:44Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:42:44Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.13]"
time="2021-09-24T20:42:44Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.12], try #1"
time="2021-09-24T20:42:44Z" level=info msg="Starting container [rke-log-linker] on host [192.168.164.14], try #1"
time="2021-09-24T20:42:44Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.12]"
time="2021-09-24T20:42:44Z" level=info msg="[worker] Successfully started [rke-log-linker] container on host [192.168.164.14]"
time="2021-09-24T20:42:44Z" level=info msg="Removing container [rke-log-linker] on host [192.168.164.14], try #1"
time="2021-09-24T20:42:45Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.164.14]"
time="2021-09-24T20:42:45Z" level=info msg="[worker] Successfully started Worker Plane.."
time="2021-09-24T20:42:45Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.14]"
time="2021-09-24T20:42:45Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.13]"
time="2021-09-24T20:42:45Z" level=info msg="Image [rancher/rke-tools:v0.1.75] exists on host [192.168.164.12]"
time="2021-09-24T20:42:45Z" level=info msg="Starting container [rke-log-cleaner] on host [192.168.164.13], try #1"
time="2021-09-24T20:42:45Z" level=info msg="Starting container [rke-log-cleaner] on host [192.168.164.14], try #1"
time="2021-09-24T20:42:45Z" level=info msg="Starting container [rke-log-cleaner] on host [192.168.164.12], try #1"
time="2021-09-24T20:42:45Z" level=info msg="[cleanup] Successfully started [rke-log-cleaner] container on host [192.168.164.13]"
time="2021-09-24T20:42:45Z" level=info msg="Removing container [rke-log-cleaner] on host [192.168.164.13], try #1"
time="2021-09-24T20:42:45Z" level=info msg="[cleanup] Successfully started [rke-log-cleaner] container on host [192.168.164.12]"
time="2021-09-24T20:42:45Z" level=info msg="[remove/rke-log-cleaner] Successfully removed container on host [192.168.164.13]"
time="2021-09-24T20:42:45Z" level=info msg="[cleanup] Successfully started [rke-log-cleaner] container on host [192.168.164.14]"
time="2021-09-24T20:42:45Z" level=info msg="Removing container [rke-log-cleaner] on host [192.168.164.14], try #1"
time="2021-09-24T20:42:45Z" level=info msg="Removing container [rke-log-cleaner] on host [192.168.164.12], try #1"
time="2021-09-24T20:42:45Z" level=info msg="[remove/rke-log-cleaner] Successfully removed container on host [192.168.164.12]"
time="2021-09-24T20:42:45Z" level=info msg="[remove/rke-log-cleaner] Successfully removed container on host [192.168.164.14]"
time="2021-09-24T20:42:45Z" level=info msg="[sync] Syncing nodes Labels and Taints"
time="2021-09-24T20:42:46Z" level=info msg="[sync] Successfully synced nodes Labels and Taints"
time="2021-09-24T20:42:46Z" level=info msg="[network] Setting up network plugin: calico"
time="2021-09-24T20:42:46Z" level=info msg="[addons] Saving ConfigMap for addon rke-network-plugin to Kubernetes"
time="2021-09-24T20:42:46Z" level=info msg="[addons] Successfully saved ConfigMap for addon rke-network-plugin to Kubernetes"
time="2021-09-24T20:42:46Z" level=info msg="[addons] Executing deploy job rke-network-plugin"
time="2021-09-24T20:43:01Z" level=info msg="[addons] Setting up coredns"
time="2021-09-24T20:43:01Z" level=info msg="[addons] Saving ConfigMap for addon rke-coredns-addon to Kubernetes"
time="2021-09-24T20:43:01Z" level=info msg="[addons] Successfully saved ConfigMap for addon rke-coredns-addon to Kubernetes"
time="2021-09-24T20:43:01Z" level=info msg="[addons] Executing deploy job rke-coredns-addon"
time="2021-09-24T20:43:06Z" level=info msg="[addons] CoreDNS deployed successfully"
time="2021-09-24T20:43:06Z" level=info msg="[dns] DNS provider coredns deployed successfully"
time="2021-09-24T20:43:06Z" level=info msg="[addons] Setting up Metrics Server"
time="2021-09-24T20:43:06Z" level=info msg="[addons] Saving ConfigMap for addon rke-metrics-addon to Kubernetes"
time="2021-09-24T20:43:06Z" level=info msg="[addons] Successfully saved ConfigMap for addon rke-metrics-addon to Kubernetes"
time="2021-09-24T20:43:06Z" level=info msg="[addons] Executing deploy job rke-metrics-addon"
time="2021-09-24T20:43:11Z" level=info msg="[addons] Metrics Server deployed successfully"
time="2021-09-24T20:43:11Z" level=info msg="[ingress] Setting up nginx ingress controller"
time="2021-09-24T20:43:11Z" level=info msg="[addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes"
time="2021-09-24T20:43:11Z" level=info msg="[addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes"
time="2021-09-24T20:43:11Z" level=info msg="[addons] Executing deploy job rke-ingress-controller"
time="2021-09-24T20:43:17Z" level=info msg="[ingress] ingress controller nginx deployed successfully"
time="2021-09-24T20:43:17Z" level=info msg="[addons] Setting up user addons"
time="2021-09-24T20:43:17Z" level=info msg="[addons] no user addons defined"
time="2021-09-24T20:43:17Z" level=info msg="Finished building Kubernetes cluster successfully"
time="2021-09-24T20:43:17Z" level=info msg="Reading RKE cluster 3b461922-73bd-4faa-a8ac-62481568ddf9 ..."
Thanks in advance