Hallo
I am trying to install a new Rancher on a newly installed Ubuntu 20.04.1 system, using RancherD following these instructions:
When I get to the step where I run systemctl start rancherd-server.service, I get several error messages, logged through systemd.
Like:
...
Jan 22 10:52:12 kub rancherd[2982]: {"level":"warn","ts":"2021-01-22T10:52:12.615+0100","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///https://127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""}
...
Jan 22 10:53:07 kub rancherd[2982]: time="2021-01-22T10:53:07+01:00" level=info msg="Waiting for kubelet to be ready on node kub: Get \"https://127.0.0.1:6443/api/v1/nodes/kub\": dial tcp 127.0.0.1:6443: connect: connection refused"
Jan 22 10:53:12 kub rancherd[2982]: time="2021-01-22T10:53:12+01:00" level=warning msg="Unable to watch for tunnel endpoints: Get \"https://127.0.0.1:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&resourceVersion=0&watch=true\": dial tcp 127.0.0.1:6443: connect: connection refused"
...
Jan 22 10:53:27 kub rancherd[2982]: time="2021-01-22T10:53:27+01:00" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
My “/etc/rancher/rke2/config.yaml”
token: [RANDOM Characters]
tls-san:
- kub.h.mydomain.tld
I also tried with https://kub.h.mydomain.tld, like in the doc, but in the section 2 part of the doc is an example without the https, so I tried both, but the result did not change.
I have also edited “/etc/hosts” to make sure the hostname is resolved to IP addresses.
127.0.0.1 kub.h.mydomain.tld kub
192.168.178.54 kub.h.mydomain.tld kub
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
After running systemctl start rancherd-server.service:
netstat -l -n -v -p -t
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 5281/kube-proxy
tcp 0 0 127.0.0.1:9099 0.0.0.0:* LISTEN 6288/calico-node
tcp 0 0 127.0.0.1:10251 0.0.0.0:* LISTEN 3399/kube-scheduler
tcp 0 0 192.168.178.54:2379 0.0.0.0:* LISTEN 3148/etcd
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 3148/etcd
tcp 0 0 127.0.0.1:10252 0.0.0.0:* LISTEN 3400/kube-controlle
tcp 0 0 192.168.178.54:2380 0.0.0.0:* LISTEN 3148/etcd
tcp 0 0 127.0.0.1:2381 0.0.0.0:* LISTEN 3148/etcd
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 8053/nginx: master
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 8053/nginx: master
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 8053/nginx: master
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 8053/nginx: master
tcp 0 0 0.0.0.0:8181 0.0.0.0:* LISTEN 8053/nginx: master
tcp 0 0 0.0.0.0:8181 0.0.0.0:* LISTEN 8053/nginx: master
tcp 0 0 0.0.0.0:8181 0.0.0.0:* LISTEN 8053/nginx: master
tcp 0 0 0.0.0.0:8181 0.0.0.0:* LISTEN 8053/nginx: master
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 633/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 783/sshd: /usr/sbin
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 667/cupsd
tcp 0 0 127.0.0.1:10010 0.0.0.0:* LISTEN 2997/containerd
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 8053/nginx: master
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 8053/nginx: master
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 8053/nginx: master
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 8053/nginx: master
tcp 0 0 127.0.0.1:10245 0.0.0.0:* LISTEN 8033/nginx-ingress-
tcp 0 0 127.0.0.1:10246 0.0.0.0:* LISTEN 8053/nginx: master
tcp 0 0 127.0.0.1:10247 0.0.0.0:* LISTEN 8053/nginx: master
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 3021/kubelet
tcp6 0 0 :::10250 :::* LISTEN 3021/kubelet
tcp6 0 0 :::6443 :::* LISTEN 3236/kube-apiserver
tcp6 0 0 :::10254 :::* LISTEN 8033/nginx-ingress-
tcp6 0 0 :::10256 :::* LISTEN 5281/kube-proxy
tcp6 0 0 :::80 :::* LISTEN 8053/nginx: master
tcp6 0 0 :::80 :::* LISTEN 8053/nginx: master
tcp6 0 0 :::80 :::* LISTEN 8053/nginx: master
tcp6 0 0 :::80 :::* LISTEN 8053/nginx: master
tcp6 0 0 :::8181 :::* LISTEN 8053/nginx: master
tcp6 0 0 :::8181 :::* LISTEN 8053/nginx: master
tcp6 0 0 :::8181 :::* LISTEN 8053/nginx: master
tcp6 0 0 :::8181 :::* LISTEN 8053/nginx: master
tcp6 0 0 :::22 :::* LISTEN 783/sshd: /usr/sbin
tcp6 0 0 ::1:631 :::* LISTEN 667/cupsd
tcp6 0 0 :::443 :::* LISTEN 8053/nginx: master
tcp6 0 0 :::443 :::* LISTEN 8053/nginx: master
tcp6 0 0 :::443 :::* LISTEN 8053/nginx: master
tcp6 0 0 :::443 :::* LISTEN 8053/nginx: master
tcp6 0 0 :::9345 :::* LISTEN 2982/rancherd serve
tcp6 0 0 :::9091 :::* LISTEN 6288/calico-node
Comparing with the error posted on top of the post:
As you can see, kube-apiserver is listening on :::6443
:: is an IPv6 reference, but I can also connect to it using IPv4 addresses. (telnet 127.0.0.1 6443 gets a connect)
Another problem is that not all pods are started.
The documentation says which pods should be up after the installation:
If you watch the pods, you will see the following pods installed:
helm-operationpods in thecattle-systemnamespace- a
rancherpod andrancher-webhookpod in thecattle-systemnamespace- a
fleet-agent,fleet-controller, andgitjobpod in thefleet-systemnamespace- a
rancher-operatorpod in therancher-operator-systemnamespace
But when I check, my result differ:
kubectl get pod -n cattle-system
NAME READY STATUS RESTARTS AGE
helm-operation-2btvk 0/2 Completed 0 58m
helm-operation-blrb6 0/2 Completed 0 58m
helm-operation-mqfbm 0/2 Completed 0 44m
helm-operation-s6dtr 0/2 Completed 0 58m
helm-operation-tkxb9 0/2 Completed 0 58m
helm-operation-z2rtt 0/2 Completed 0 43m
helm-operation-z652n 0/2 Completed 0 58m
rancher-m6xtq 1/1 Running 0 60m
rancher-webhook-798c5599d9-zpzmz 1/1 Running 0 58m
Here the complete log of the startup for reference.
# journalctl -eu rancherd-server -f
-- Logs begin at Thu 2021-01-21 20:38:16 CET. --
Jan 22 10:51:44 kub systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
Jan 22 10:51:44 kub systemd[1]: Started Rancher Kubernetes Engine v2 (server).
Jan 22 10:51:45 kub rancherd[2982]: time="2021-01-22T10:51:45+01:00" level=warning msg="not running in CIS 1.5 mode"
Jan 22 10:51:45 kub rancherd[2982]: time="2021-01-22T10:51:45+01:00" level=info msg="Pulling runtime image \"index.docker.io/rancher/rancher-runtime:v2.5.5\""
Jan 22 10:51:45 kub rancherd[2982]: Unable to determine config dir: could not determine home directory
Jan 22 10:51:45 kub rancherd[2982]: No matching credentials were found, falling back on anonymous
Jan 22 10:51:46 kub rancherd[2982]: time="2021-01-22T10:51:46+01:00" level=info msg="Extracting \"index.docker.io/rancher/rancher-runtime:v2.5.5\" \"/bin/\" to \"/var/lib/rancher/rke2/data/v2.5.5-7202e0b56d5c/bin\""
Jan 22 10:51:47 kub rancherd[2982]: time="2021-01-22T10:51:47+01:00" level=info msg="Extracting file \"bin/kubectl\""
Jan 22 10:51:48 kub rancherd[2982]: time="2021-01-22T10:51:48+01:00" level=info msg="Extracting file \"bin/kubelet\""
Jan 22 10:51:50 kub rancherd[2982]: time="2021-01-22T10:51:50+01:00" level=info msg="Extracting file \"bin/containerd\""
Jan 22 10:51:51 kub rancherd[2982]: time="2021-01-22T10:51:51+01:00" level=info msg="Extracting file \"bin/containerd-shim\""
Jan 22 10:51:51 kub rancherd[2982]: time="2021-01-22T10:51:51+01:00" level=info msg="Extracting file \"bin/containerd-shim-runc-v1\""
Jan 22 10:51:52 kub rancherd[2982]: time="2021-01-22T10:51:52+01:00" level=info msg="Extracting file \"bin/containerd-shim-runc-v2\""
Jan 22 10:51:52 kub rancherd[2982]: time="2021-01-22T10:51:52+01:00" level=info msg="Extracting file \"bin/ctr\""
Jan 22 10:51:53 kub rancherd[2982]: time="2021-01-22T10:51:53+01:00" level=info msg="Extracting file \"bin/crictl\""
Jan 22 10:51:53 kub rancherd[2982]: time="2021-01-22T10:51:53+01:00" level=info msg="Extracting file \"bin/runc\""
Jan 22 10:51:54 kub rancherd[2982]: time="2021-01-22T10:51:54+01:00" level=info msg="Extracting file \"bin/socat\""
Jan 22 10:51:54 kub rancherd[2982]: time="2021-01-22T10:51:54+01:00" level=info msg="Done extracting \"index.docker.io/rancher/rancher-runtime:v2.5.5\""
Jan 22 10:51:54 kub rancherd[2982]: time="2021-01-22T10:51:54+01:00" level=info msg="Extracting \"index.docker.io/rancher/rancher-runtime:v2.5.5\" \"/charts/\" to \"/var/lib/rancher/rke2/server/manifests\""
Jan 22 10:51:54 kub rancherd[2982]: time="2021-01-22T10:51:54+01:00" level=info msg="Extracting file \"charts/rancher-namespace.yaml\""
Jan 22 10:51:54 kub rancherd[2982]: time="2021-01-22T10:51:54+01:00" level=info msg="Extracting file \"charts/rancher.yaml\""
Jan 22 10:51:54 kub rancherd[2982]: time="2021-01-22T10:51:54+01:00" level=info msg="Extracting file \"charts/rke2-canal.yaml\""
Jan 22 10:51:54 kub rancherd[2982]: time="2021-01-22T10:51:54+01:00" level=info msg="Extracting file \"charts/rke2-coredns.yaml\""
Jan 22 10:51:54 kub rancherd[2982]: time="2021-01-22T10:51:54+01:00" level=info msg="Extracting file \"charts/rke2-ingress-nginx.yaml\""
Jan 22 10:51:54 kub rancherd[2982]: time="2021-01-22T10:51:54+01:00" level=info msg="Extracting file \"charts/rke2-kube-proxy.yaml\""
Jan 22 10:51:54 kub rancherd[2982]: time="2021-01-22T10:51:54+01:00" level=info msg="Extracting file \"charts/rke2-metrics-server.yaml\""
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Done extracting \"index.docker.io/rancher/rancher-runtime:v2.5.5\""
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Updated HelmChart \"/var/lib/rancher/rke2/server/manifests/rancher.yaml\" to apply --system-default-registry modifications"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Updated HelmChart \"/var/lib/rancher/rke2/server/manifests/rke2-canal.yaml\" to apply --system-default-registry modifications"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Updated HelmChart \"/var/lib/rancher/rke2/server/manifests/rke2-coredns.yaml\" to apply --system-default-registry modifications"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Updated HelmChart \"/var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml\" to apply --system-default-registry modifications"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Updated HelmChart \"/var/lib/rancher/rke2/server/manifests/rke2-kube-proxy.yaml\" to apply --system-default-registry modifications"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Updated HelmChart \"/var/lib/rancher/rke2/server/manifests/rke2-metrics-server.yaml\" to apply --system-default-registry modifications"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Starting rke2 v2.5.5 (HEAD)"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Managed etcd cluster initializing"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="certificate CN=system:admin,O=system:masters signed by CN=rke2-client-ca@1611309122: notBefore=2021-01-22 09:52:02 +0000 UTC notAfter=2022-01-22 09:52:02 +0000 UTC"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="certificate CN=system:kube-controller-manager signed by CN=rke2-client-ca@1611309122: notBefore=2021-01-22 09:52:02 +0000 UTC notAfter=2022-01-22 09:52:02 +0000 UTC"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="certificate CN=system:kube-scheduler signed by CN=rke2-client-ca@1611309122: notBefore=2021-01-22 09:52:02 +0000 UTC notAfter=2022-01-22 09:52:02 +0000 UTC"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="certificate CN=kube-apiserver signed by CN=rke2-client-ca@1611309122: notBefore=2021-01-22 09:52:02 +0000 UTC notAfter=2022-01-22 09:52:02 +0000 UTC"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="certificate CN=system:kube-proxy signed by CN=rke2-client-ca@1611309122: notBefore=2021-01-22 09:52:02 +0000 UTC notAfter=2022-01-22 09:52:02 +0000 UTC"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="certificate CN=system:rke2-controller signed by CN=rke2-client-ca@1611309122: notBefore=2021-01-22 09:52:02 +0000 UTC notAfter=2022-01-22 09:52:02 +0000 UTC"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="certificate CN=cloud-controller-manager signed by CN=rke2-client-ca@1611309122: notBefore=2021-01-22 09:52:02 +0000 UTC notAfter=2022-01-22 09:52:02 +0000 UTC"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="certificate CN=kube-apiserver signed by CN=rke2-server-ca@1611309122: notBefore=2021-01-22 09:52:02 +0000 UTC notAfter=2022-01-22 09:52:02 +0000 UTC"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="certificate CN=system:auth-proxy signed by CN=rke2-request-header-ca@1611309122: notBefore=2021-01-22 09:52:02 +0000 UTC notAfter=2022-01-22 09:52:02 +0000 UTC"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="certificate CN=etcd-server signed by CN=etcd-server-ca@1611309122: notBefore=2021-01-22 09:52:02 +0000 UTC notAfter=2022-01-22 09:52:02 +0000 UTC"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="certificate CN=etcd-client signed by CN=etcd-server-ca@1611309122: notBefore=2021-01-22 09:52:02 +0000 UTC notAfter=2022-01-22 09:52:02 +0000 UTC"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="certificate CN=etcd-peer signed by CN=etcd-peer-ca@1611309122: notBefore=2021-01-22 09:52:02 +0000 UTC notAfter=2022-01-22 09:52:02 +0000 UTC"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="certificate CN=rke2,O=rke2 signed by CN=rke2-server-ca@1611309122: notBefore=2021-01-22 09:52:02 +0000 UTC notAfter=2022-01-22 09:52:02 +0000 UTC"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=unknown --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=rke2 --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Active TLS secret (ver=) (count 8): map[listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-192.168.178.54:192.168.178.54 listener.cattle.io/cn-kub.h.jan-gerritsen.de:kub.h.jan-gerritsen.de listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=A07C2161A245F3AC60386A9BEEB993A5A18EA7C7]"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Running kube-scheduler --address=127.0.0.1 --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --port=10251 --profiling=false --secure-port=0"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --port=10252 --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --use-service-account-credentials=true"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Node token is available at /var/lib/rancher/rke2/server/token"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="To join node to cluster: rke2 agent -s https://192.168.178.54:9345 -t ${NODE_TOKEN}"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Wrote kubeconfig /etc/rancher/rke2/rke2.yaml"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Run: rancherd kubectl"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Module overlay was already loaded"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Module br_netfilter was already loaded"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Cluster-Http-Server 2021/01/22 10:52:02 http: TLS handshake error from 127.0.0.1:56978: remote error: tls: bad certificate"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Cluster-Http-Server 2021/01/22 10:52:02 http: TLS handshake error from 127.0.0.1:56984: remote error: tls: bad certificate"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="certificate CN=kub signed by CN=rke2-server-ca@1611309122: notBefore=2021-01-22 09:52:02 +0000 UTC notAfter=2022-01-22 09:52:02 +0000 UTC"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="certificate CN=system:node:kub,O=system:nodes signed by CN=rke2-client-ca@1611309122: notBefore=2021-01-22 09:52:02 +0000 UTC notAfter=2022-01-22 09:52:02 +0000 UTC"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Logging containerd to /var/lib/rancher/rke2/agent/containerd/containerd.log"
Jan 22 10:52:02 kub rancherd[2982]: time="2021-01-22T10:52:02+01:00" level=info msg="Running containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/agent/containerd"
Jan 22 10:52:03 kub rancherd[2982]: time="2021-01-22T10:52:03+01:00" level=info msg="Containerd is now running"
Jan 22 10:52:03 kub rancherd[2982]: time="2021-01-22T10:52:03+01:00" level=info msg="Pulling image docker.io/rancher/hardened-etcd:v3.4.13-k3s1..."
Jan 22 10:52:12 kub rancherd[2982]: {"level":"warn","ts":"2021-01-22T10:52:12.615+0100","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///https://127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""}
Jan 22 10:52:12 kub rancherd[2982]: time="2021-01-22T10:52:12+01:00" level=info msg="Failed to test data store connection: context deadline exceeded"
Jan 22 10:52:13 kub rancherd[2982]: time="2021-01-22T10:52:13+01:00" level=info msg="Pulling image docker.io/rancher/hardened-kubernetes:v1.18.12-rke2r2-amd64..."
Jan 22 10:52:27 kub rancherd[2982]: {"level":"warn","ts":"2021-01-22T10:52:27.610+0100","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///https://127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""}
Jan 22 10:52:27 kub rancherd[2982]: time="2021-01-22T10:52:27+01:00" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Jan 22 10:52:27 kub rancherd[2982]: {"level":"warn","ts":"2021-01-22T10:52:27.615+0100","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///https://127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""}
Jan 22 10:52:27 kub rancherd[2982]: time="2021-01-22T10:52:27+01:00" level=info msg="Failed to test data store connection: context deadline exceeded"
Jan 22 10:52:32 kub rancherd[2982]: time="2021-01-22T10:52:32+01:00" level=info msg="Waiting for etcd server to become available"
Jan 22 10:52:37 kub rancherd[2982]: time="2021-01-22T10:52:37+01:00" level=info msg="Connecting to proxy" url="wss://127.0.0.1:9345/v1-rke2/connect"
Jan 22 10:52:37 kub rancherd[2982]: time="2021-01-22T10:52:37+01:00" level=info msg="Handling backend connection request [kub]"
Jan 22 10:52:37 kub rancherd[2982]: time="2021-01-22T10:52:37+01:00" level=info msg="Running kubelet --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=/run/k3s/containerd/containerd.sock --container-runtime=remote --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=kub --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --kubelet-cgroups=/systemd/system.slice --log-file-max-size=50 --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --logtostderr=false --node-labels= --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --runtime-cgroups=/systemd/system.slice --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key"
Jan 22 10:52:37 kub rancherd[2982]: time="2021-01-22T10:52:37+01:00" level=info msg="Waiting for kubelet to be ready on node kub: Get \"https://127.0.0.1:6443/api/v1/nodes/kub\": dial tcp 127.0.0.1:6443: connect: connection refused"
Jan 22 10:52:37 kub rancherd[3021]: Flag --address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 22 10:52:37 kub rancherd[3021]: Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 22 10:52:37 kub rancherd[3021]: Flag --authentication-token-webhook has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 22 10:52:37 kub rancherd[3021]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 22 10:52:37 kub rancherd[3021]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 22 10:52:37 kub rancherd[3021]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 22 10:52:37 kub rancherd[3021]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 22 10:52:37 kub rancherd[3021]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 22 10:52:37 kub rancherd[3021]: Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.
Jan 22 10:52:37 kub rancherd[3021]: Flag --eviction-hard has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 22 10:52:37 kub rancherd[3021]: Flag --eviction-minimum-reclaim has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 22 10:52:37 kub rancherd[3021]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 22 10:52:37 kub rancherd[3021]: Flag --healthz-bind-address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 22 10:52:37 kub rancherd[3021]: Flag --kubelet-cgroups has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 22 10:52:37 kub rancherd[3021]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 22 10:52:37 kub rancherd[3021]: Flag --read-only-port has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 22 10:52:37 kub rancherd[3021]: Flag --resolv-conf has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 22 10:52:37 kub rancherd[3021]: Flag --serialize-image-pulls has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 22 10:52:37 kub rancherd[3021]: Flag --tls-cert-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 22 10:52:37 kub rancherd[3021]: Flag --tls-private-key-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 22 10:52:42 kub rancherd[2982]: time="2021-01-22T10:52:42+01:00" level=warning msg="Unable to watch for tunnel endpoints: Get \"https://127.0.0.1:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&resourceVersion=0&watch=true\": dial tcp 127.0.0.1:6443: connect: connection refused"
Jan 22 10:52:42 kub rancherd[2982]: {"level":"warn","ts":"2021-01-22T10:52:42.610+0100","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///https://127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""}
Jan 22 10:52:42 kub rancherd[2982]: time="2021-01-22T10:52:42+01:00" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Jan 22 10:52:42 kub rancherd[2982]: {"level":"warn","ts":"2021-01-22T10:52:42.616+0100","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///https://127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""}
Jan 22 10:52:42 kub rancherd[2982]: time="2021-01-22T10:52:42+01:00" level=info msg="Failed to test data store connection: context deadline exceeded"
Jan 22 10:52:47 kub rancherd[2982]: time="2021-01-22T10:52:47+01:00" level=warning msg="Unable to watch for tunnel endpoints: Get \"https://127.0.0.1:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&resourceVersion=0&watch=true\": dial tcp 127.0.0.1:6443: connect: connection refused"
Jan 22 10:52:52 kub rancherd[2982]: time="2021-01-22T10:52:52+01:00" level=warning msg="Unable to watch for tunnel endpoints: Get \"https://127.0.0.1:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&resourceVersion=0&watch=true\": dial tcp 127.0.0.1:6443: connect: connection refused"
Jan 22 10:52:57 kub rancherd[2982]: time="2021-01-22T10:52:57+01:00" level=warning msg="Unable to watch for tunnel endpoints: Get \"https://127.0.0.1:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&resourceVersion=0&watch=true\": dial tcp 127.0.0.1:6443: connect: connection refused"
Jan 22 10:52:57 kub rancherd[2982]: {"level":"warn","ts":"2021-01-22T10:52:57.610+0100","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-81595a1f-c74b-478b-8b5d-d8978af1c85f/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""}
Jan 22 10:52:57 kub rancherd[2982]: time="2021-01-22T10:52:57+01:00" level=error msg="Failed to get recorded learner progress from etcd: context deadline exceeded"
Jan 22 10:52:57 kub rancherd[2982]: {"level":"warn","ts":"2021-01-22T10:52:57.616+0100","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-81595a1f-c74b-478b-8b5d-d8978af1c85f/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""}
Jan 22 10:52:57 kub rancherd[2982]: time="2021-01-22T10:52:57+01:00" level=info msg="Failed to test data store connection: context deadline exceeded"
Jan 22 10:53:02 kub rancherd[2982]: time="2021-01-22T10:53:02+01:00" level=warning msg="Unable to watch for tunnel endpoints: Get \"https://127.0.0.1:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&resourceVersion=0&watch=true\": dial tcp 127.0.0.1:6443: connect: connection refused"
Jan 22 10:53:02 kub rancherd[2982]: time="2021-01-22T10:53:02+01:00" level=info msg="Waiting for etcd server to become available"
Jan 22 10:53:07 kub rancherd[2982]: time="2021-01-22T10:53:07+01:00" level=warning msg="Unable to watch for tunnel endpoints: Get \"https://127.0.0.1:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&resourceVersion=0&watch=true\": dial tcp 127.0.0.1:6443: connect: connection refused"
Jan 22 10:53:07 kub rancherd[2982]: time="2021-01-22T10:53:07+01:00" level=info msg="Waiting for kubelet to be ready on node kub: Get \"https://127.0.0.1:6443/api/v1/nodes/kub\": dial tcp 127.0.0.1:6443: connect: connection refused"
Jan 22 10:53:12 kub rancherd[2982]: time="2021-01-22T10:53:12+01:00" level=warning msg="Unable to watch for tunnel endpoints: Get \"https://127.0.0.1:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&resourceVersion=0&watch=true\": dial tcp 127.0.0.1:6443: connect: connection refused"
Jan 22 10:53:12 kub rancherd[2982]: {"level":"warn","ts":"2021-01-22T10:53:12.610+0100","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-81595a1f-c74b-478b-8b5d-d8978af1c85f/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""}
Jan 22 10:53:12 kub rancherd[2982]: time="2021-01-22T10:53:12+01:00" level=error msg="Failed to get recorded learner progress from etcd: context deadline exceeded"
Jan 22 10:53:12 kub rancherd[2982]: {"level":"warn","ts":"2021-01-22T10:53:12.617+0100","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-81595a1f-c74b-478b-8b5d-d8978af1c85f/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""}
Jan 22 10:53:12 kub rancherd[2982]: time="2021-01-22T10:53:12+01:00" level=info msg="Failed to test data store connection: context deadline exceeded"
Jan 22 10:53:17 kub rancherd[2982]: time="2021-01-22T10:53:17+01:00" level=warning msg="Unable to watch for tunnel endpoints: Get \"https://127.0.0.1:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&resourceVersion=0&watch=true\": dial tcp 127.0.0.1:6443: connect: connection refused"
Jan 22 10:53:17 kub rancherd[2982]: time="2021-01-22T10:53:17+01:00" level=info msg="etcd data store connection OK"
Jan 22 10:53:17 kub rancherd[2982]: time="2021-01-22T10:53:17+01:00" level=info msg="Waiting for API server to become available"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="Applying Cluster Role Bindings"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="Applying Pod Security Policies"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="Kube API server is now running"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="rke2 is up and running"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="Setting Cluster RoleBinding: kube-apiserver-kubelet-admin"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="Setting PSP: global-unrestricted-psp"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="Setting Cluster Role: system:rke2-controller"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="Creating CRD addons.k3s.cattle.io"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="Setting Cluster RoleBinding: system:rke2-controller"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="Creating CRD helmcharts.helm.cattle.io"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="Setting Cluster Role: global-unrestricted-psp-clusterrole"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="Creating CRD helmchartconfigs.helm.cattle.io"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="Setting Cluster Role: cloud-controller-manager"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="Setting Cluster RoleBinding: global-unrestricted-psp-rolebinding"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="Waiting for CRD helmcharts.helm.cattle.io to become available"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="Setting PSP: system-unrestricted-psp"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="Setting Cluster RoleBinding: cloud-controller-manager"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="Cluster Role Bindings applied successfully"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="Setting Cluster Role: system-unrestricted-psp-role"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="Setting Cluster RoleBinding: system-unrestricted-node-psp-rolebinding"
Jan 22 10:53:22 kub rancherd[2982]: time="2021-01-22T10:53:22+01:00" level=info msg="Setting Cluster RoleBinding: system-unrestricted-svc-acct-psp-rolebinding"
Jan 22 10:53:23 kub rancherd[2982]: time="2021-01-22T10:53:23+01:00" level=info msg="Done waiting for CRD helmcharts.helm.cattle.io to become available"
Jan 22 10:53:23 kub rancherd[2982]: time="2021-01-22T10:53:23+01:00" level=info msg="Waiting for CRD helmchartconfigs.helm.cattle.io to become available"
Jan 22 10:53:23 kub rancherd[2982]: time="2021-01-22T10:53:23+01:00" level=info msg="Done waiting for CRD helmchartconfigs.helm.cattle.io to become available"
Jan 22 10:53:23 kub rancherd[2982]: I0122 10:53:23.781925 2982 leaderelection.go:243] attempting to acquire leader lease kube-system/rke2...
Jan 22 10:53:23 kub rancherd[2982]: time="2021-01-22T10:53:23+01:00" level=info msg="Starting k3s.cattle.io/v1, Kind=Addon controller"
Jan 22 10:53:23 kub rancherd[2982]: time="2021-01-22T10:53:23+01:00" level=info msg="Starting /v1, Kind=Node controller"
Jan 22 10:53:23 kub rancherd[2982]: time="2021-01-22T10:53:23+01:00" level=info msg="Waiting for master node kub startup: nodes \"kub\" not found"
Jan 22 10:53:23 kub rancherd[2982]: I0122 10:53:23.789241 2982 leaderelection.go:253] successfully acquired lease kube-system/rke2
Jan 22 10:53:23 kub rancherd[2982]: time="2021-01-22T10:53:23+01:00" level=info msg="Cluster dns configmap has been set successfully"
Jan 22 10:53:23 kub rancherd[2982]: time="2021-01-22T10:53:23+01:00" level=info msg="updating namespace: kube-system"
Jan 22 10:53:23 kub rancherd[2982]: time="2021-01-22T10:53:23+01:00" level=info msg="Pod Security Policies applied successfully"
Jan 22 10:53:23 kub rancherd[2982]: time="2021-01-22T10:53:23+01:00" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChart controller"
Jan 22 10:53:23 kub rancherd[2982]: time="2021-01-22T10:53:23+01:00" level=info msg="Starting /v1, Kind=Pod controller"
Jan 22 10:53:23 kub rancherd[2982]: time="2021-01-22T10:53:23+01:00" level=info msg="Starting /v1, Kind=Endpoints controller"
Jan 22 10:53:23 kub rancherd[2982]: time="2021-01-22T10:53:23+01:00" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChartConfig controller"
Jan 22 10:53:23 kub rancherd[2982]: time="2021-01-22T10:53:23+01:00" level=info msg="Starting /v1, Kind=Service controller"
Jan 22 10:53:23 kub rancherd[2982]: time="2021-01-22T10:53:23+01:00" level=info msg="Starting batch/v1, Kind=Job controller"
Jan 22 10:53:24 kub rancherd[2982]: time="2021-01-22T10:53:24+01:00" level=info msg="Starting /v1, Kind=Secret controller"
Jan 22 10:53:24 kub rancherd[2982]: time="2021-01-22T10:53:24+01:00" level=info msg="labels have been set successfully on node: kub"
Jan 22 10:53:24 kub rancherd[2982]: time="2021-01-22T10:53:24+01:00" level=info msg="Active TLS secret rke2-serving (ver=303) (count 8): map[listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-192.168.178.54:192.168.178.54 listener.cattle.io/cn-kub.h.jan-gerritsen.de:kub.h.jan-gerritsen.de listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=A07C2161A245F3AC60386A9BEEB993A5A18EA7C7]"
Jan 22 10:53:27 kub rancherd[2982]: time="2021-01-22T10:53:27+01:00" level=info msg="Stopped tunnel to 127.0.0.1:9345"
Jan 22 10:53:27 kub rancherd[2982]: time="2021-01-22T10:53:27+01:00" level=info msg="Connecting to proxy" url="wss://192.168.178.54:9345/v1-rke2/connect"
Jan 22 10:53:27 kub rancherd[2982]: time="2021-01-22T10:53:27+01:00" level=info msg="Proxy done" err="context canceled" url="wss://127.0.0.1:9345/v1-rke2/connect"
Jan 22 10:53:27 kub rancherd[2982]: time="2021-01-22T10:53:27+01:00" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
Jan 22 10:53:27 kub rancherd[2982]: time="2021-01-22T10:53:27+01:00" level=info msg="Handling backend connection request [kub]"