New installation of rancher fatal error exit with status 1

Jade_Baxter · July 24, 2021, 5:30am

fresh ubuntu 20.04.2 LTS install

only commands I have run:
sudo apt update
sudo apt upgrade
sudo reboot
sudo curl https://releases.rancher.com/install-docker/20.10.sh | sh
sudo reboot
sudo docker run --privileged -d --restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher

and rancher container just keeps restarting… this is what I am getting:
p are deprecated. Please begin using a config file ASAP.
I0724 05:22:51.189040 34 server.go:407] Version: v1.19.8+k3s1
W0724 05:22:51.190075 34 proxier.go:639] Failed to read file /lib/modules/5.4.0-80-generic/modules.builtin with error open /lib/modules/5.4.0-80-generic/modules.builtin: no such file or directory. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0724 05:22:51.191556 34 proxier.go:649] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0724 05:22:51.192436 34 proxier.go:649] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0724 05:22:51.193392 34 proxier.go:649] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
I0724 05:22:51.193997 34 leaderelection.go:243] attempting to acquire leader lease kube-system/kube-scheduler…
W0724 05:22:51.194475 34 proxier.go:649] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0724 05:22:51.195385 34 proxier.go:649] Failed to load kernel module nf_conntrack with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
time=“2021-07-24T05:22:51.211356409Z” level=info msg=“Waiting for node local-node CIDR not assigned yet”
I0724 05:22:51.214656 34 node.go:136] Successfully retrieved node IP: 172.17.0.2
I0724 05:22:51.214734 34 server_others.go:143] kube-proxy node IP is an IPv4 address (172.17.0.2), assume IPv4 operation
I0724 05:22:51.218038 34 server_others.go:186] Using iptables Proxier.
I0724 05:22:51.218684 34 server.go:650] Version: v1.19.8+k3s1
I0724 05:22:51.219367 34 conntrack.go:103] Set sysctl ‘net/netfilter/nf_conntrack_max’ to 163840
F0724 05:22:51.219420 34 server.go:495] open /proc/sys/net/netfilter/nf_conntrack_max: permission denied
2021/07/24 05:22:51 [FATAL] k3s exited with: exit status 1
2021/07/24 05:22:54 [INFO] Rancher version v2.5.9 (3c5418944) is starting
2021/07/24 05:22:54 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Agent:false Features: ClusterRegistry:}
2021/07/24 05:22:54 [INFO] Listening on /tmp/log.sock
2021/07/24 05:22:54 [INFO] Running etcd --data-dir=management-state/etcd --heartbeat-interval=500 --election-timeout=5000
2021-07-24 05:22:54.300989 W | pkg/flags: unrecognized environment variable ETCD_URL_arm64=https://github.com/etcd-io/etcd/releases/download/v3.4.3/etcd-v3.4.3-linux-arm64.tar.gz
2021-07-24 05:22:54.301349 W | pkg/flags: unrecognized environment variable ETCD_URL_amd64=https://github.com/etcd-io/etcd/releases/download/v3.4.3/etcd-v3.4.3-linux-amd64.tar.gz
2021-07-24 05:22:54.301544 W | pkg/flags: unrecognized environment variable ETCD_UNSUPPORTED_ARCH=amd64
2021-07-24 05:22:54.301737 W | pkg/flags: unrecognized environment variable ETCD_URL=ETCD_URL_amd64
[WARNING] Deprecated ‘–logger=capnslog’ flag is set; use ‘–logger=zap’ flag instead
2021-07-24 05:22:54.302196 I | etcdmain: etcd Version: 3.4.3
2021-07-24 05:22:54.302342 I | etcdmain: Git SHA: 3cf2f69b5
2021-07-24 05:22:54.302489 I | etcdmain: Go Version: go1.12.12
2021-07-24 05:22:54.302661 I | etcdmain: Go OS/Arch: linux/amd64
2021-07-24 05:22:54.302857 I | etcdmain: setting maximum number of CPUs to 5, total number of available CPUs is 5
2021-07-24 05:22:54.303172 N | etcdmain: the server is already initialized as member before, starting as etcd member…
[WARNING] Deprecated ‘–logger=capnslog’ flag is set; use ‘–logger=zap’ flag instead
2021-07-24 05:22:54.304954 I | embed: name = default
2021-07-24 05:22:54.305114 I | embed: data dir = management-state/etcd
2021-07-24 05:22:54.305269 I | embed: member dir = management-state/etcd/member
2021-07-24 05:22:54.305424 I | embed: heartbeat = 500ms
2021-07-24 05:22:54.305572 I | embed: election = 5000ms
2021-07-24 05:22:54.305770 I | embed: snapshot count = 100000
2021-07-24 05:22:54.305936 I | embed: advertise client URLs = http://localhost:2379
2021-07-24 05:22:54.306031 I | embed: initial advertise peer URLs = http://localhost:2380
2021-07-24 05:22:54.306097 I | embed: initial cluster =
2021-07-24 05:22:54.321171 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 887
raft2021/07/24 05:22:54 INFO: 8e9e05c52164694d switched to configuration voters=()
raft2021/07/24 05:22:54 INFO: 8e9e05c52164694d became follower at term 22
raft2021/07/24 05:22:54 INFO: newRaft 8e9e05c52164694d [peers: [], term: 22, commit: 887, applied: 0, lastindex: 887, lastterm: 22]
2021-07-24 05:22:54.358094 W | auth: simple token is not cryptographically signed
2021-07-24 05:22:54.390149 I | etcdserver: starting server… [version: 3.4.3, cluster version: to_be_decided]
raft2021/07/24 05:22:54 INFO: 8e9e05c52164694d switched to configuration voters=(10276657743932975437)
2021-07-24 05:22:54.392276 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
2021-07-24 05:22:54.392558 N | etcdserver/membership: set the initial cluster version to 3.4
2021-07-24 05:22:54.392653 I | etcdserver/api: enabled capabilities for version 3.4
2021-07-24 05:22:54.395380 I | embed: listening for peers on 127.0.0.1:2380
raft2021/07/24 05:23:02 INFO: 8e9e05c52164694d is starting a new election at term 22
raft2021/07/24 05:23:02 INFO: 8e9e05c52164694d became candidate at term 23
raft2021/07/24 05:23:02 INFO: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 23
raft2021/07/24 05:23:02 INFO: 8e9e05c52164694d became leader at term 23
raft2021/07/24 05:23:02 INFO: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 23
2021-07-24 05:23:02.946234 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
2021-07-24 05:23:02.946301 I | embed: ready to serve client requests
2021-07-24 05:23:02.947827 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
2021/07/24 05:23:03 [INFO] Waiting for server to become available: Get “https://127.0.0.1:6443/version?timeout=15m0s”: dial tcp 127.0.0.1:6443: connect: connection refused
time=“2021-07-24T05:23:03.550108651Z” level=info msg=“Starting k3s v1.19.8+k3s1 (95fc76b2)”
time=“2021-07-24T05:23:03.550775625Z” level=info msg=“Cluster bootstrap already complete”
time=“2021-07-24T05:23:03.587600819Z” level=info msg=“Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=unknown --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=http://localhost:2379 --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=k3s --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key”
I0724 05:23:03.590797 31 server.go:652] external host was not specified, using 172.17.0.2
I0724 05:23:03.591539 31 server.go:177] Version: v1.19.8+k3s1
I0724 05:23:03.605870 31 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0724 05:23:03.606374 31 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0724 05:23:03.608543 31 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0724 05:23:03.608733 31 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0724 05:23:03.660016 31 master.go:271] Using reconciler: lease
W0724 05:23:04.353355 31 genericapiserver.go:412] Skipping API batch/v2alpha1 because it has no resources.
W0724 05:23:04.387776 31 genericapiserver.go:412] Skipping API discovery.k8s.io/v1alpha1 because it has no resources.
W0724 05:23:04.458702 31 genericapiserver.go:412] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0724 05:23:04.493240 31 genericapiserver.go:412] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0724 05:23:04.499435 31 genericapiserver.go:412] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0724 05:23:04.525432 31 genericapiserver.go:412] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0724 05:23:04.557049 31 genericapiserver.go:412] Skipping API apps/v1beta2 because it has no resources.
W0724 05:23:04.557113 31 genericapiserver.go:412] Skipping API apps/v1beta1 because it has no resources.
I0724 05:23:04.579308 31 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0724 05:23:04.579348 31 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
time=“2021-07-24T05:23:04.603038727Z” level=info msg=“Running kube-scheduler --address=127.0.0.1 --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --port=10251 --profiling=false --secure-port=0”
time=“2021-07-24T05:23:04.603685041Z” level=info msg=“Waiting for API server to become available”
I0724 05:23:04.603821 31 registry.go:173] Registering SelectorSpread plugin
I0724 05:23:04.603849 31 registry.go:173] Registering SelectorSpread plugin
time=“2021-07-24T05:23:04.604437390Z” level=info msg=“Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --port=10252 --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --use-service-account-credentials=true”
time=“2021-07-24T05:23:04.608187729Z” level=info msg=“Node token is available at /var/lib/rancher/k3s/server/token”
time=“2021-07-24T05:23:04.608248273Z” level=info msg=“To join node to cluster: k3s agent -s https://172.17.0.2:6443 -t ${NODE_TOKEN}”
time=“2021-07-24T05:23:04.610492678Z” level=info msg=“Wrote kubeconfig /etc/rancher/k3s/k3s.yaml”
time=“2021-07-24T05:23:04.610671003Z” level=info msg=“Run: k3s kubectl”
time=“2021-07-24T05:23:04.611197751Z” level=info msg=“Module overlay was already loaded”
time=“2021-07-24T05:23:04.611267703Z” level=info msg=“Module nf_conntrack was already loaded”
time=“2021-07-24T05:23:04.611350506Z” level=info msg=“Module br_netfilter was already loaded”
time=“2021-07-24T05:23:04.611447381Z” level=info msg=“Module iptable_nat was already loaded”
time=“2021-07-24T05:23:04.634127717Z” level=info msg=“Cluster-Http-Server 2021/07/24 05:23:04 http: TLS handshake error from 127.0.0.1:43782: remote error: tls: bad certificate”
time=“2021-07-24T05:23:04.643242697Z” level=info msg=“Cluster-Http-Server 2021/07/24 05:23:04 http: TLS handshake error from 127.0.0.1:43790: remote error: tls: bad certificate”
time=“2021-07-24T05:23:04.663346154Z” level=info msg=“certificate CN=local-node signed by CN=k3s-server-ca@1627103731: notBefore=2021-07-24 05:15:31 +0000 UTC notAfter=2022-07-24 05:23:04 +0000 UTC”
time=“2021-07-24T05:23:04.668207781Z” level=info msg=“certificate CN=system:node:local-node,O=system:nodes signed by CN=k3s-client-ca@1627103731: notBefore=2021-07-24 05:15:31 +0000 UTC notAfter=2022-07-24 05:23:04 +0000 UTC”
time=“2021-07-24T05:23:04.680230028Z” level=info msg=“Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log”
time=“2021-07-24T05:23:04.680458976Z” level=info msg=“Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd”
W0724 05:23:05.085061 7 warnings.go:80] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0724 05:23:05.125407 7 warnings.go:80] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0724 05:23:05.156004 7 warnings.go:80] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time=“2021-07-24T05:23:05.684798635Z” level=info msg=“Containerd is now running”
I0724 05:23:08.665527 31 dynamic_cafile_content.go:167] Starting request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt
I0724 05:23:08.665548 31 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt
I0724 05:23:08.666379 31 dynamic_serving_content.go:130] Starting serving-cert::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key
I0724 05:23:08.667037 31 secure_serving.go:197] Serving securely on 127.0.0.1:6444
I0724 05:23:08.667238 31 tlsconfig.go:240] Starting DynamicServingCertificateController
I0724 05:23:08.667393 31 customresource_discovery_controller.go:209] Starting DiscoveryController
I0724 05:23:08.668087 31 available_controller.go:475] Starting AvailableConditionController
I0724 05:23:08.668120 31 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0724 05:23:08.668632 31 apiservice_controller.go:97] Starting APIServiceRegistrationController
I0724 05:23:08.668667 31 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0724 05:23:08.668755 31 autoregister_controller.go:141] Starting autoregister controller
I0724 05:23:08.668774 31 cache.go:32] Waiting for caches to sync for autoregister controller
I0724 05:23:08.669704 31 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I0724 05:23:08.669746 31 shared_informer.go:240] Waiting for caches to sync for cluster_authentication_trust_controller
I0724 05:23:08.669824 31 dynamic_serving_content.go:130] Starting aggregator-proxy-cert::/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt::/var/lib/rancher/k3s/server/tls/client-auth-proxy.key
I0724 05:23:08.669992 31 controller.go:83] Starting OpenAPI AggregationController
I0724 05:23:08.670155 31 controller.go:86] Starting OpenAPI controller
I0724 05:23:08.670297 31 naming_controller.go:291] Starting NamingConditionController
I0724 05:23:08.670370 31 establishing_controller.go:76] Starting EstablishingController
I0724 05:23:08.670402 31 nonstructuralschema_controller.go:186] Starting NonStructuralSchemaConditionController
I0724 05:23:08.670533 31 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I0724 05:23:08.670597 31 crd_finalizer.go:266] Starting CRDFinalizer
I0724 05:23:08.670652 31 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt
I0724 05:23:08.670753 31 dynamic_cafile_content.go:167] Starting request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt
I0724 05:23:08.677319 31 crdregistration_controller.go:111] Starting crd-autoregister controller
I0724 05:23:08.677389 31 shared_informer.go:240] Waiting for caches to sync for crd-autoregister
time=“2021-07-24T05:23:08.686110191Z” level=info msg=“Running cloud-controller-manager --allocate-node-cidrs=true --allow-untagged-cloud=true --bind-address=127.0.0.1 --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --node-status-update-frequency=1m --profiling=false --secure-port=0”
Flag --allow-untagged-cloud has been deprecated, This flag is deprecated and will be removed in a future release. A cluster-id will be required on cloud instances.
I0724 05:23:08.746589 31 controllermanager.go:127] Version: v1.19.8+k3s1
W0724 05:23:08.746780 31 controllermanager.go:139] detected a cluster without a ClusterID. A ClusterID will be required in the future. Please tag your cluster to avoid any future issues
I0724 05:23:08.746928 31 leaderelection.go:243] attempting to acquire leader lease kube-system/cloud-controller-manager…
E0724 05:23:08.752579 31 leaderelection.go:325] error retrieving resource lock kube-system/cloud-controller-manager: endpoints “cloud-controller-manager” is forbidden: User “cloud-controller-manager” cannot get resource “endpoints” in API group “” in the namespace “kube-system”
I0724 05:23:08.768691 31 cache.go:39] Caches are synced for AvailableConditionController controller
I0724 05:23:08.768742 31 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0724 05:23:08.768824 31 cache.go:39] Caches are synced for autoregister controller
I0724 05:23:08.772181 31 shared_informer.go:247] Caches are synced for cluster_authentication_trust_controller
I0724 05:23:08.778189 31 shared_informer.go:247] Caches are synced for crd-autoregister
E0724 05:23:08.833806 31 controller.go:156] Unable to remove old endpoints from kubernetes service: no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service
time=“2021-07-24T05:23:09.637741361Z” level=info msg=“Connecting to proxy” url=“wss://172.17.0.2:6443/v1-k3s/connect”
time=“2021-07-24T05:23:09.642132772Z” level=info msg=“Handling backend connection request [local-node]”
time=“2021-07-24T05:23:09.644062810Z” level=warning msg=“Disabling CPU quotas due to missing cpu.cfs_period_us”
time=“2021-07-24T05:23:09.644323720Z” level=info msg=“Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/rancher/k3s/data/ab1b521f47b3cbc273e0c667521084cff1ea3540d55b425ac7ed30c27daf92f8/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --container-runtime-endpoint=/run/k3s/containerd/containerd.sock --container-runtime=remote --containerd=/run/k3s/containerd/containerd.sock --cpu-cfs-quota=false --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=local-node --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --kubelet-cgroups=/k3s --node-labels= --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --runtime-cgroups=/k3s --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key”
time=“2021-07-24T05:23:09.646503090Z” level=info msg=“Running kube-proxy --cluster-cidr=10.42.0.0/16 --healthz-bind-address=127.0.0.1 --hostname-override=local-node --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables”
Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.
W0724 05:23:09.647498 31 server.go:226] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
I0724 05:23:09.648317 31 server.go:407] Version: v1.19.8+k3s1
W0724 05:23:09.648647 31 proxier.go:639] Failed to read file /lib/modules/5.4.0-80-generic/modules.builtin with error open /lib/modules/5.4.0-80-generic/modules.builtin: no such file or directory. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0724 05:23:09.655417 31 proxier.go:649] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0724 05:23:09.656237 31 proxier.go:649] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0724 05:23:09.656883 31 proxier.go:649] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0724 05:23:09.657751 31 proxier.go:649] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0724 05:23:09.662170 31 proxier.go:649] Failed to load kernel module nf_conntrack with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
time=“2021-07-24T05:23:09.663784910Z” level=info msg=“Waiting for node local-node CIDR not assigned yet”
I0724 05:23:09.665392 31 controller.go:132] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
I0724 05:23:09.675580 31 storage_scheduling.go:143] all system priority classes are created successfully or already exist.
I0724 05:23:09.678794 31 node.go:136] Successfully retrieved node IP: 172.17.0.2
I0724 05:23:09.678870 31 server_others.go:143] kube-proxy node IP is an IPv4 address (172.17.0.2), assume IPv4 operation
I0724 05:23:09.683177 31 server_others.go:186] Using iptables Proxier.
I0724 05:23:09.683781 31 server.go:650] Version: v1.19.8+k3s1
I0724 05:23:09.684851 31 conntrack.go:103] Set sysctl ‘net/netfilter/nf_conntrack_max’ to 163840
F0724 05:23:09.684904 31 server.go:495] open /proc/sys/net/netfilter/nf_conntrack_max: permission denied
2021/07/24 05:23:09 [FATAL] k3s exited with: exit status 1

superseb · July 24, 2021, 11:05am

Why are there 2 topics for this? Rancher admin ui never starts

Jade_Baxter · July 24, 2021, 11:29am

so first one was orginal cluster (thats been running for a year or longer)… and then I tried to build a completely new one.
And discovered that I couldn’t build a new one… So not sure if they are related or not.

Jade

TiTidom · July 26, 2021, 7:13am

Hi,

I had the same problem yesterday after upgrading to the last kernel “Ubuntu 20.04 LTS : Linux 5.4.0-80-generic”.
After rolling back to the n-1 version of the kernel (5.4.0-77-generic) i was able to restart my rancher (rancher 2.5.9 single node on docker 20.10.7).

If needed :

You can roll back the kernel version easily via the “grub” menu : on boot, choose the grub menu et the menu “Advanced”, and choose “5.4.0-77” kernel version.

After, in linux remove the 5.4.0-80 kernel files via :
sudo apt remove linux-image-5.4.0-80-generic linux-headers-5.4.0-80-generic linux-image-unsigned-5.4.0-80-generic

and finally :
sudo apt autoremove

TiTidom.

Indirectelex · July 26, 2021, 8:43am

Am glad to see I am not the only one with that issue. Like Jade_Baster I’ve bring the log on my post Rancher UI can't be reached - #18 by Indirectelex I made the experiment on Ubuntu server 18.04.5 and 20.04 lts with the same results

Jade_Baxter · July 26, 2021, 8:51am

How do I install older kernel? looks like 5.4.0-80 is the only option from grub menu.
and I did a test remove of kernel on a test vm and ignore the you will break your shiz if you removal kernel… and it broke.

im not expert sorry… much appreciate some help here!

TiTidom · July 26, 2021, 6:15pm

Hi,

You can try this (on an Ubuntu 20.04) :

sudo apt install linux-image-5.4.0-77-generic linux-headers-5.4.0-77-generic

It will install the linux kernel 5.4.0-77 (this is the version i had just before the 5.4.0-80, and the one on which my rancher is running) and it will add it to grub menu, and after that, on reboot you will be able to choose this version 77 from the “Advanced” grub menu.

Jade_Baxter · August 4, 2021, 5:14am

after a lot of testing.

That didnt work.

I have tried different versions of docker, rancher and nothing seems to work!

Here is my last few lines of logs… any ideas?

goroutine 2561 [select]:
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.WaitFor(0xc0061b6200, 0xc00eede300, 0xc00ac81200, 0x0, 0x0)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:539 +0x11d
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.PollUntil(0xdf8475800, 0xc00eede300, 0xc00ffb0a80, 0x0, 0x0)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:492 +0xc5
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0xdf8475800, 0xc00eede300, 0xc00ffb0a80, 0xb, 0xc011212f48)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:511 +0xb3
created by github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).Run
        /go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates/dynamic_cafile_content.go:174 +0x2b3

goroutine 2578 [select, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.contextForChannel.func1(0xc00ffb0a80, 0xc00eede320, 0x4c77280, 0xc00fc5aa40)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:279 +0xbd
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.contextForChannel
        /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:278 +0x8c

goroutine 2579 [select]:
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc00ac812c0, 0xdf8475800, 0x0, 0xc00ac81260)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:588 +0x17b
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:571 +0x8c

goroutine 2585 [chan receive]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:772 +0x5d
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00fada760)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc01120ff60, 0x4bb9920, 0xc009819d40, 0x3e3d801, 0xc00ac814a0)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00fada760, 0x3b9aca00, 0x0, 0x6572206c6c697701, 0xc00ac814a0)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).run(0xc01062a600)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x95
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc006ca33a0, 0xc00eede5f0)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

goroutine 2581 [chan receive, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc006ca3340, 0xc00ac81320)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:628 +0x53
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:56 +0x2e
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00f7430f0, 0xc00eec9020)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

goroutine 2582 [chan receive, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*controller).Run.func1(0xc00ffb0a80, 0xc0055fbe60)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/controller.go:127 +0x34
created by github.com/rancher/k3s/vendor/k8s.E0804 05:12:48.369635      35 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}

vincent · August 4, 2021, 8:43pm

That’s a stack trace, and the beginning of the section that looks like that would be the interesting part; the end is a bunch of goroutines (“threads”) that had nothing to do with what happened but just happen to be running.

Jade_Baxter · August 5, 2021, 12:19am

I just went and got a blob of logs, I suspect these failures and every VM a wall of logs is actually causing my IO issues.
It looks like my SSD is using 1% wear our per week! So it would be great to get to the bottom of this.
Before my 650$ SSD writes itself to death.

I0804 21:37:19.507151      34 trace.go:116] Trace[1650914652]: "List" url:/apis/rbac.authorization.k8s.io/v1/clusterrolebindings,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:14.681908853 +0000 UTC m=+15.497464276) (total time: 4.825210905s):
Trace[1650914652]: [4.823403895s] [4.823394056s] Listing from storage done
2021-08-04 21:37:19.688251 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "error:context deadline exceeded" took too long (2.000020101s) to execute
2021/08/04 21:37:21 [INFO] Waiting for server to become available: the server is currently unable to handle the request
2021-08-04 21:37:21.250682 W | wal: sync duration of 4.712136779s, expected less than 1s
2021-08-04 21:37:21.286047 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (596.989153ms) to execute
2021-08-04 21:37:21.286216 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/default/kubernetes\" " with result "range_response_count:1 size:277" took too long (1.782885688s) to execute
2021-08-04 21:37:21.286254 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:aggregate-to-admin\" " with result "range_response_count:1 size:831" took too long (1.776263069s) to execute
I0804 21:37:21.287224      34 trace.go:116] Trace[2106552881]: "Get" url:/api/v1/namespaces/default/endpoints/kubernetes,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:19.502277857 +0000 UTC m=+20.317833269) (total time: 1.784861954s):
Trace[2106552881]: [1.784732787s] [1.784720903s] About to write a response
I0804 21:37:21.287414      34 trace.go:116] Trace[639215876]: "Get" url:/apis/rbac.authorization.k8s.io/v1/clusterroles/system:aggregate-to-admin,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:19.50943629 +0000 UTC m=+20.324991702) (total time: 1.777909971s):
Trace[639215876]: [1.777780207s] [1.777771877s] About to write a response
E0804 21:37:21.288642      34 controller.go:156] Unable to remove old endpoints from kubernetes service: no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service
2021/08/04 21:37:23 [INFO] Waiting for server to become available: the server is currently unable to handle the request
2021-08-04 21:37:24.036127 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (2.631871003s) to execute
2021-08-04 21:37:24.036288 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (531.605946ms) to execute
2021-08-04 21:37:24.036383 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (2.347622973s) to execute
2021-08-04 21:37:24.036426 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:certificates.k8s.io:certificatesigningrequests:nodeclient\" " with result "range_response_count:1 size:697" took too long (2.684047709s) to execute
I0804 21:37:24.037227      34 trace.go:116] Trace[2089915500]: "Get" url:/apis/rbac.authorization.k8s.io/v1/clusterroles/system:certificates.k8s.io:certificatesigningrequests:nodeclient,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:21.351771887 +0000 UTC m=+22.167327307) (total time: 2.685392047s):
Trace[2089915500]: [2.685283669s] [2.685274549s] About to write a response
2021/08/04 21:37:25 [INFO] Waiting for server to become available: the server is currently unable to handle the request
2021-08-04 21:37:26.173865 W | wal: sync duration of 1.010663311s, expected less than 1s
2021-08-04 21:37:26.577447 W | etcdserver: request "header:<ID:7587856269140755382 > txn:<compare:<target:MOD key:\"/registry/masterleases/172.17.0.2\" mod_revision:0 > success:<request_put:<key:\"/registry/masterleases/172.17.0.2\" value_size:65 lease:7587856269140755380 >> failure:<request_range:<key:\"/registry/masterleases/172.17.0.2\" > >>" with result "size:20" took too long (1.414046943s) to execute
2021-08-04 21:37:26.577802 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (1.889151848s) to execute
I0804 21:37:26.578616      34 trace.go:116] Trace[1515445373]: "GuaranteedUpdate etcd3" type:*v1.Endpoints (started: 2021-08-04 21:37:24.207813959 +0000 UTC m=+25.023369383) (total time: 2.37073092s):
Trace[1515445373]: [2.370674666s] [2.369296182s] Transaction committed
I0804 21:37:26.579767      34 registry.go:150] Registering EvenPodsSpread predicate and priority function
I0804 21:37:26.579828      34 registry.go:150] Registering EvenPodsSpread predicate and priority function
time="2021-08-04T21:37:26.581104914Z" level=info msg="Running kube-scheduler --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --port=10251 --secure-port=0"
time="2021-08-04T21:37:26.582354363Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --port=10252 --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --use-service-account-credentials=true"
I0804 21:37:26.599344      34 controllermanager.go:161] Version: v1.18.8+k3s1
I0804 21:37:26.601459      34 deprecated_insecure_serving.go:53] Serving insecurely on [::]:10252
I0804 21:37:26.601543      34 leaderelection.go:242] attempting to acquire leader lease  kube-system/kube-controller-manager...
time="2021-08-04T21:37:26.602962319Z" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --allow-untagged-cloud=true --bind-address=127.0.0.1 --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --node-status-update-frequency=1m --secure-port=0"
Flag --allow-untagged-cloud has been deprecated, This flag is deprecated and will be removed in a future release. A cluster-id will be required on cloud instances.
I0804 21:37:26.605937      34 registry.go:150] Registering EvenPodsSpread predicate and priority function
I0804 21:37:26.605995      34 registry.go:150] Registering EvenPodsSpread predicate and priority function
W0804 21:37:26.608990      34 authorization.go:47] Authorization is disabled
W0804 21:37:26.609031      34 authentication.go:40] Authentication is disabled
I0804 21:37:26.609049      34 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:10251
I0804 21:37:26.617454      34 controllermanager.go:120] Version: v1.18.8+k3s1
W0804 21:37:26.617797      34 controllermanager.go:132] detected a cluster without a ClusterID.  A ClusterID will be required in the future.  Please tag your cluster to avoid any future issues
I0804 21:37:26.618051      34 leaderelection.go:242] attempting to acquire leader lease  kube-system/cloud-controller-manager...
I0804 21:37:26.710105      34 leaderelection.go:242] attempting to acquire leader lease  kube-system/kube-scheduler...
2021-08-04 21:37:32.309865 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:578" took too long (5.597216281s) to execute
2021-08-04 21:37:32.310103 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:596" took too long (2.515031253s) to execute
2021-08-04 21:37:32.310643 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/cloud-controller-manager\" " with result "range_response_count:1 size:598" took too long (3.339977935s) to execute
2021-08-04 21:37:32.312151 W | etcdserver: read-only range request "key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitions0\" " with result "range_response_count:94 size:371930" took too long (5.085179186s) to execute
2021-08-04 21:37:32.312376 W | etcdserver: read-only range request "key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitions0\" " with result "range_response_count:94 size:371930" took too long (5.613036124s) to execute
I0804 21:37:32.313636      34 trace.go:116] Trace[8002812]: "Get" url:/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:28.970063705 +0000 UTC m=+29.785619126) (total time: 3.343496837s):
Trace[8002812]: [3.343385809s] [3.343369865s] About to write a response
I0804 21:37:32.314111      34 trace.go:116] Trace[888439527]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-scheduler,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:26.712106071 +0000 UTC m=+27.527661494) (total time: 5.60197453s):
Trace[888439527]: [5.601935537s] [5.601906392s] About to write a response
I0804 21:37:32.314563      34 trace.go:116] Trace[797559310]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:29.794445301 +0000 UTC m=+30.610000858) (total time: 2.520082755s):
Trace[797559310]: [2.520043095s] [2.520005273s] About to write a response
I0804 21:37:32.355084      34 trace.go:116] Trace[1054663085]: "List etcd3" key:/apiextensions.k8s.io/customresourcedefinitions,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:26.698650722 +0000 UTC m=+27.514206143) (total time: 5.656379067s):
Trace[1054663085]: [5.656379067s] [5.656379067s] END
I0804 21:37:32.357938      34 trace.go:116] Trace[230938352]: "List etcd3" key:/apiextensions.k8s.io/customresourcedefinitions,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:27.226322528 +0000 UTC m=+28.041877951) (total time: 5.131586633s):
Trace[230938352]: [5.131586633s] [5.131586633s] END
I0804 21:37:32.389680      34 trace.go:116] Trace[218574836]: "List" url:/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:26.698624802 +0000 UTC m=+27.514180223) (total time: 5.690977144s):
Trace[218574836]: [5.65651178s] [5.656497967s] Listing from storage done
I0804 21:37:32.394255      34 trace.go:116] Trace[656729179]: "List" url:/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:37:27.226292467 +0000 UTC m=+28.041848128) (total time: 5.167910097s):
Trace[656729179]: [5.131684375s] [5.131667043s] Listing from storage done
time="2021-08-04T21:37:32.704580271Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-1.81.0.tgz"
time="2021-08-04T21:37:32.705554059Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/rolebindings.yaml"
time="2021-08-04T21:37:32.705841893Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/ccm.yaml"
time="2021-08-04T21:37:32.706252789Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml"
time="2021-08-04T21:37:32.807415573Z" level=info msg="Starting k3s.cattle.io/v1, Kind=Addon controller"
time="2021-08-04T21:37:32.808126502Z" level=info msg="Waiting for master node  startup: resource name may not be empty"
I0804 21:37:32.808298      34 leaderelection.go:242] attempting to acquire leader lease  kube-system/k3s...
time="2021-08-04T21:37:32.808793201Z" level=info msg="Node token is available at /var/lib/rancher/k3s/server/token"
time="2021-08-04T21:37:32.808881530Z" level=info msg="To join node to cluster: k3s agent -s https://172.17.0.2:6443 -t ${NODE_TOKEN}"
I0804 21:37:32.820640      34 leaderelection.go:252] successfully acquired lease kube-system/k3s
2021-08-04 21:37:32.831621 I | http: TLS handshake error from 127.0.0.1:57276: remote error: tls: bad certificate
time="2021-08-04T21:37:32.842580282Z" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
time="2021-08-04T21:37:32.842871546Z" level=info msg="Run: k3s kubectl"
time="2021-08-04T21:37:32.843161127Z" level=info msg="k3s is up and running"
time="2021-08-04T21:37:32.843755579Z" level=info msg="module overlay was already loaded"
time="2021-08-04T21:37:32.843972121Z" level=info msg="module nf_conntrack was already loaded"
time="2021-08-04T21:37:32.844183405Z" level=info msg="module br_netfilter was already loaded"
2021-08-04 21:37:32.848156 I | http: TLS handshake error from 127.0.0.1:57284: remote error: tls: bad certificate
2021-08-04 21:37:32.860744 I | http: TLS handshake error from 127.0.0.1:57292: remote error: tls: bad certificate
time="2021-08-04T21:37:32.908211332Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
time="2021-08-04T21:37:32.908714383Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
time="2021-08-04T21:37:32.908918759Z" level=info msg="Waiting for containerd startup: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: connection refused\""
time="2021-08-04T21:37:33.912961440Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
time="2021-08-04T21:37:34.916906088Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
time="2021-08-04T21:37:35.919852761Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
2021-08-04 21:37:36.636905 W | wal: sync duration of 3.817543913s, expected less than 1s
2021-08-04 21:37:36.637947 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/features/unsupported-storage-drivers\" " with result "range_response_count:1 size:746" took too long (3.787092708s) to execute
2021-08-04 21:37:36.638035 W | etcdserver: read-only range request "key:\"/registry/secrets/kube-system/k3s-serving\" " with result "range_response_count:1 size:2278" took too long (2.972115236s) to execute
2021-08-04 21:37:36.638451 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/k3s\" " with result "range_response_count:1 size:508" took too long (3.815319618s) to execute
2021-08-04 21:37:36.638613 W | etcdserver: read-only range request "key:\"/registry/helm.cattle.io/helmcharts/\" range_end:\"/registry/helm.cattle.io/helmcharts0\" limit:500 " with result "range_response_count:0 size:7" took too long (3.747223605s) to execute
2021-08-04 21:37:36.639001 W | etcdserver: read-only range request "key:\"/registry/helm.cattle.io/helmcharts/\" range_end:\"/registry/helm.cattle.io/helmcharts0\" limit:10000 " with result "range_response_count:0 size:7" took too long (3.748573313s) to execute
2021-08-04 21:37:36.639157 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/cloud-controller-manager\" " with result "range_response_count:1 size:598" took too long (768.580393ms) to execute
2021-08-04 21:37:36.639367 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/features\" range_end:\"/registry/management.cattle.io/featuret\" count_only:true " with result "range_response_count:0 size:9" took too long (3.774884651s) to execute
2021-08-04 21:37:36.639490 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/features/\" range_end:\"/registry/management.cattle.io/features0\" limit:10000 " with result "range_response_count:3 size:2168" took too long (3.775248775s) to execute
2021-08-04 21:37:36.639683 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:596" took too long (1.254538349s) to execute
2021-08-04 21:37:36.640013 W | etcdserver: read-only range request "key:\"/registry/helm.cattle.io/helmcharts\" range_end:\"/registry/helm.cattle.io/helmchartt\" count_only:true " with result "range_response_count:0 size:7" took too long (3.749531567s) to execute
2021-08-04 21:37:36.640157 W | etcdserver: read-only range request "key:\"/registry/namespaces/default\" " with result "range_response_count:1 size:979" took too long (2.433121496s) to execute
2021-08-04 21:37:36.640382 W | etcdserver: read-only range request "key:\"/registry/minions/local-node\" " with result "range_response_count:1 size:5788" took too long (2.828664605s) to execute
2021-08-04 21:37:36.641162 W | etcdserver: read-only range request "key:\"/registry/clusterrolebindings/\" range_end:\"/registry/clusterrolebindings0\" " with result "range_response_count:93 size:76427" took too long (3.789060318s) to execute
I0804 21:37:36.642937      34 trace.go:116] Trace[639411826]: "Get" url:/api/v1/namespaces/kube-system/configmaps/k3s,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:32.822842571 +0000 UTC m=+33.638397984) (total time: 3.819963244s):
Trace[639411826]: [3.819687907s] [3.81967925s] About to write a response
I0804 21:37:36.644576      34 trace.go:116] Trace[998888116]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:35.383941584 +0000 UTC m=+36.199497005) (total time: 1.260458442s):
Trace[998888116]: [1.260238306s] [1.260184764s] About to write a response
I0804 21:37:36.645925      34 trace.go:116] Trace[1092890937]: "Get" url:/api/v1/namespaces/kube-system/secrets/k3s-serving,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:33.664906593 +0000 UTC m=+34.480462016) (total time: 2.980965321s):
Trace[1092890937]: [2.980494564s] [2.980476599s] About to write a response
I0804 21:37:36.646771      34 trace.go:116] Trace[524276838]: "Get" url:/apis/management.cattle.io/v3/features/unsupported-storage-drivers,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:37:32.849778256 +0000 UTC m=+33.665333677) (total time: 3.79692944s):
Trace[524276838]: [3.796244694s] [3.796208567s] About to write a response
I0804 21:37:36.647469      34 trace.go:116] Trace[287172946]: "List etcd3" key:/helm.cattle.io/helmcharts,resourceVersion:0,limit:500,continue: (started: 2021-08-04 21:37:32.891084571 +0000 UTC m=+33.706639995) (total time: 3.756295281s):
Trace[287172946]: [3.756295281s] [3.756295281s] END
I0804 21:37:36.660746      34 trace.go:116] Trace[2145214703]: "List" url:/apis/helm.cattle.io/v1/helmcharts,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:32.891016693 +0000 UTC m=+33.706572114) (total time: 3.769693647s):
Trace[2145214703]: [3.769442889s] [3.769386376s] Listing from storage done
I0804 21:37:36.650985      34 trace.go:116] Trace[540723056]: "List etcd3" key:/management.cattle.io/features,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:32.863845789 +0000 UTC m=+33.679401214) (total time: 3.787034937s):
Trace[540723056]: [3.787034937s] [3.787034937s] END
I0804 21:37:36.651176      34 trace.go:116] Trace[497601668]: "Get" url:/api/v1/nodes/local-node,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:33.810957711 +0000 UTC m=+34.626513132) (total time: 2.840121026s):
Trace[497601668]: [2.831916973s] [2.831870589s] About to write a response
I0804 21:37:36.646829      34 trace.go:116] Trace[971510249]: "Get" url:/api/v1/namespaces/default,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:34.206293866 +0000 UTC m=+35.021849287) (total time: 2.439715117s):
Trace[971510249]: [2.439424838s] [2.439406079s] About to write a response
I0804 21:37:36.652621      34 trace.go:116] Trace[1018010203]: "List etcd3" key:/clusterrolebindings,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:32.851756489 +0000 UTC m=+33.667311912) (total time: 3.8008273s):
Trace[1018010203]: [3.8008273s] [3.8008273s] END
I0804 21:37:36.662409      34 trace.go:116] Trace[144746405]: "List" url:/apis/rbac.authorization.k8s.io/v1/clusterrolebindings,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:32.85170656 +0000 UTC m=+33.667261995) (total time: 3.810670603s):
Trace[144746405]: [3.810453438s] [3.810411542s] Listing from storage done
I0804 21:37:36.658235      34 trace.go:116] Trace[633053414]: "List etcd3" key:/helm.cattle.io/helmcharts,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:32.889684779 +0000 UTC m=+33.705240200) (total time: 3.768480761s):
Trace[633053414]: [3.768480761s] [3.768480761s] END
I0804 21:37:36.660373      34 trace.go:116] Trace[1171465763]: "Get" url:/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:35.869836494 +0000 UTC m=+36.685391916) (total time: 790.483409ms):
Trace[1171465763]: [790.436109ms] [790.419577ms] About to write a response
time="2021-08-04T21:37:36.726032130Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChart controller"
time="2021-08-04T21:37:36.726053828Z" level=info msg="Starting batch/v1, Kind=Job controller"
time="2021-08-04T21:37:36.826372914Z" level=info msg="Starting /v1, Kind=Node controller"
time="2021-08-04T21:37:36.826481743Z" level=info msg="Starting /v1, Kind=Endpoints controller"
time="2021-08-04T21:37:36.826676764Z" level=info msg="Starting /v1, Kind=Secret controller"
time="2021-08-04T21:37:36.826402935Z" level=info msg="Starting /v1, Kind=Pod controller"
time="2021-08-04T21:37:36.827585211Z" level=info msg="Starting /v1, Kind=Service controller"
time="2021-08-04T21:37:36.922511022Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
time="2021-08-04T21:37:37.926925825Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
time="2021-08-04T21:37:38.929364949Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
2021-08-04 21:37:38.989786 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/default/kubernetes\" " with result "range_response_count:1 size:277" took too long (2.272139827s) to execute
2021-08-04 21:37:38.989949 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/features/istio-virtual-service-ui\" " with result "range_response_count:1 size:798" took too long (2.268829771s) to execute
2021-08-04 21:37:38.990310 W | etcdserver: read-only range request "key:\"/registry/clusterrolebindings/cloud-controller-manager\" " with result "range_response_count:1 size:1741" took too long (2.301310945s) to execute
2021-08-04 21:37:38.990349 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/k3s\" " with result "range_response_count:1 size:508" took too long (326.464561ms) to execute
2021-08-04 21:37:38.990799 W | etcdserver: read-only range request "key:\"/registry/secrets/kube-system/k3s-serving\" " with result "range_response_count:1 size:2278" took too long (2.160348777s) to execute
I0804 21:37:38.991918      34 trace.go:116] Trace[2100397584]: "GuaranteedUpdate etcd3" type:*rbac.ClusterRoleBinding (started: 2021-08-04 21:37:36.683774389 +0000 UTC m=+37.499329814) (total time: 2.307993884s):
Trace[2100397584]: [2.307993884s] [2.307776771s] END
I0804 21:37:38.992340      34 trace.go:116] Trace[1752332777]: "Get" url:/api/v1/namespaces/kube-system/secrets/k3s-serving,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:36.829759538 +0000 UTC m=+37.645314950) (total time: 2.162490056s):
Trace[1752332777]: [2.161956812s] [2.161940351s] About to write a response
I0804 21:37:38.992758      34 trace.go:116] Trace[1282650318]: "Patch" url:/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:36.677196971 +0000 UTC m=+37.492752394) (total time: 2.315510914s):
Trace[1282650318]: [2.31514241s] [2.304268246s] Object stored in database
I0804 21:37:38.992980      34 trace.go:116] Trace[52405334]: "GuaranteedUpdate etcd3" type:*unstructured.Unstructured (started: 2021-08-04 21:37:36.719511387 +0000 UTC m=+37.535066811) (total time: 2.273426946s):
Trace[52405334]: [2.273426946s] [2.273315458s] END
I0804 21:37:38.993663      34 trace.go:116] Trace[55329941]: "Get" url:/api/v1/namespaces/default/endpoints/kubernetes,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:36.716868855 +0000 UTC m=+37.532424276) (total time: 2.276747806s):
Trace[55329941]: [2.276583296s] [2.276568454s] About to write a response
I0804 21:37:38.994273      34 trace.go:116] Trace[200967240]: "Update" url:/apis/management.cattle.io/v3/features/istio-virtual-service-ui,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:37:36.718916971 +0000 UTC m=+37.534472392) (total time: 2.27532111s):
Trace[200967240]: [2.27495189s] [2.274436657s] Object stored in database
I0804 21:37:39.001302      34 controller.go:606] quota admission added evaluator for: addons.k3s.cattle.io
time="2021-08-04T21:37:39.530524546Z" level=info msg="Active TLS secret k3s-serving (ver=2198) (count 8): map[field.cattle.io/projectId:local:p-g9nfj listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-172.17.0.2:172.17.0.2 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/hash:1042dae54ecd440e586d608fc064d597df082ce324a7538cbdf3d291a8a6e360]"
time="2021-08-04T21:37:39.932698360Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
2021-08-04 21:37:40.283479 W | wal: sync duration of 1.282800091s, expected less than 1s
time="2021-08-04T21:37:40.935302541Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
time="2021-08-04T21:37:41.937513795Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
2021-08-04 21:37:42.394045 W | wal: sync duration of 2.110258443s, expected less than 1s
2021-08-04 21:37:42.809048 W | etcdserver: read-only range request "key:\"/registry/k3s.cattle.io/addons/kube-system/ccm\" " with result "range_response_count:1 size:650" took too long (3.807087376s) to execute
2021-08-04 21:37:42.809556 W | etcdserver: request "header:<ID:7587856269140755430 > lease_revoke:<id:694d7b1318db03b4>" with result "size:32" took too long (2.525584174s) to execute
2021-08-04 21:37:42.810132 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:578" took too long (3.140276148s) to execute
2021-08-04 21:37:42.810552 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/features/multi-cluster-management\" " with result "range_response_count:0 size:7" took too long (3.278506434s) to execute
2021-08-04 21:37:42.810855 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:596" took too long (1.987817402s) to execute
2021-08-04 21:37:42.811309 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/cloud-controller-manager\" " with result "range_response_count:1 size:598" took too long (738.384626ms) to execute
2021-08-04 21:37:42.811709 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/k3s\" " with result "range_response_count:1 size:508" took too long (1.806378647s) to execute
I0804 21:37:42.812982      34 trace.go:116] Trace[1977221883]: "Get" url:/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:42.07224807 +0000 UTC m=+42.887803491) (total time: 740.595703ms):
Trace[1977221883]: [740.44856ms] [740.430433ms] About to write a response
I0804 21:37:42.813571      34 trace.go:116] Trace[585636766]: "GuaranteedUpdate etcd3" type:*unstructured.Unstructured (started: 2021-08-04 21:37:38.999793191 +0000 UTC m=+39.815348656) (total time: 3.81374111s):
Trace[585636766]: [3.81374111s] [3.813513632s] END
I0804 21:37:42.813755      34 trace.go:116] Trace[1836211634]: "Get" url:/apis/management.cattle.io/v3/features/multi-cluster-management,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:37:39.531246135 +0000 UTC m=+40.346801570) (total time: 3.282466322s):
Trace[1836211634]: [3.282466322s] [3.28237029s] END
I0804 21:37:42.813771      34 trace.go:116] Trace[569524558]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-scheduler,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:39.66928526 +0000 UTC m=+40.484840672) (total time: 3.144443148s):
Trace[569524558]: [3.1443828s] [3.144303707s] About to write a response
I0804 21:37:42.814263      34 trace.go:116] Trace[953909209]: "Update" url:/apis/k3s.cattle.io/v1/namespaces/kube-system/addons/ccm,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:38.998847587 +0000 UTC m=+39.814403001) (total time: 3.815353195s):
Trace[953909209]: [3.814871499s] [3.814068006s] Object stored in database
I0804 21:37:42.815158      34 trace.go:116] Trace[30939452]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:40.82235717 +0000 UTC m=+41.637912582) (total time: 1.992731885s):
Trace[30939452]: [1.992670395s] [1.992611884s] About to write a response
I0804 21:37:42.817907      34 trace.go:116] Trace[2107692972]: "Get" url:/api/v1/namespaces/kube-system/configmaps/k3s,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:41.004752954 +0000 UTC m=+41.820308366) (total time: 1.813062479s):
Trace[2107692972]: [1.812970694s] [1.812954413s] About to write a response
I0804 21:37:42.830733      34 controller.go:606] quota admission added evaluator for: endpoints
time="2021-08-04T21:37:42.939771545Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
2021-08-04 21:37:43.821109 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/features/fleet\" " with result "range_response_count:1 size:642" took too long (992.902412ms) to execute
I0804 21:37:43.823179      34 trace.go:116] Trace[1635767301]: "GuaranteedUpdate etcd3" type:*core.Endpoints (started: 2021-08-04 21:37:42.82918093 +0000 UTC m=+43.644736388) (total time: 993.921556ms):
Trace[1635767301]: [993.742176ms] [992.104698ms] Transaction committed
I0804 21:37:43.823472      34 trace.go:116] Trace[1652850135]: "Update" url:/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:42.825541657 +0000 UTC m=+43.641097088) (total time: 997.891268ms):
Trace[1652850135]: [997.752741ms] [994.381631ms] Object stored in database
I0804 21:37:43.823705      34 trace.go:116] Trace[1381373005]: "GuaranteedUpdate etcd3" type:*unstructured.Unstructured (started: 2021-08-04 21:37:42.823063924 +0000 UTC m=+43.638619361) (total time: 1.000577374s):
Trace[1381373005]: [1.000577374s] [999.057409ms] END
I0804 21:37:43.824191      34 trace.go:116] Trace[681760686]: "Update" url:/apis/management.cattle.io/v3/features/fleet,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:37:42.820926501 +0000 UTC m=+43.636481913) (total time: 1.003215832s):
Trace[681760686]: [1.002875757s] [1.002304788s] Object stored in database
time="2021-08-04T21:37:43.942376446Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
time="2021-08-04T21:37:44.952239750Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
time="2021-08-04T21:37:45.955077486Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
2021-08-04 21:37:46.562497 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:596" took too long (1.371474497s) to execute
2021-08-04 21:37:46.562869 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/cloud-controller-manager\" " with result "range_response_count:1 size:505" took too long (2.728039538s) to execute
2021-08-04 21:37:46.564115 W | etcdserver: read-only range request "key:\"/registry/serviceaccounts/\" range_end:\"/registry/serviceaccounts0\" " with result "range_response_count:56 size:22828" took too long (3.729322347s) to execute
2021-08-04 21:37:46.567035 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:578" took too long (1.669619884s) to execute
2021-08-04 21:37:46.567844 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/k3s\" " with result "range_response_count:1 size:508" took too long (1.737553857s) to execute
2021-08-04 21:37:46.570167 W | etcdserver: read-only range request "key:\"/registry/namespaces/default\" " with result "range_response_count:1 size:979" took too long (2.362019217s) to execute
2021-08-04 21:37:46.572912 W | etcdserver: read-only range request "key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitions0\" " with result "range_response_count:94 size:371930" took too long (2.69941809s) to execute
I0804 21:37:46.575033      34 trace.go:116] Trace[856340912]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-scheduler,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:44.896682775 +0000 UTC m=+45.712238197) (total time: 1.678274285s):
Trace[856340912]: [1.678171763s] [1.678131485s] About to write a response
I0804 21:37:46.575634      34 trace.go:116] Trace[371082943]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:45.189358214 +0000 UTC m=+46.004913638) (total time: 1.386246084s):
Trace[371082943]: [1.386147897s] [1.386109761s] About to write a response
I0804 21:37:46.575100      34 trace.go:116] Trace[860374852]: "Get" url:/api/v1/namespaces/kube-system/configmaps/k3s,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:44.82952788 +0000 UTC m=+45.645083292) (total time: 1.745510625s):
Trace[860374852]: [1.745327012s] [1.745310695s] About to write a response
I0804 21:37:46.576384      34 trace.go:116] Trace[1247262942]: "Get" url:/api/v1/namespaces/default,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:44.206659122 +0000 UTC m=+45.022214534) (total time: 2.369684125s):
Trace[1247262942]: [2.369600232s] [2.369566106s] About to write a response
I0804 21:37:46.575113      34 trace.go:116] Trace[859539008]: "Get" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:43.827362595 +0000 UTC m=+44.642918044) (total time: 2.747595524s):
Trace[859539008]: [2.747515479s] [2.747500909s] About to write a response
I0804 21:37:46.577637      34 trace.go:116] Trace[2106966267]: "List etcd3" key:/serviceaccounts,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:42.833655318 +0000 UTC m=+43.649210734) (total time: 3.743868453s):
Trace[2106966267]: [3.743868453s] [3.743868453s] END
I0804 21:37:46.578125      34 trace.go:116] Trace[1183369946]: "List" url:/api/v1/serviceaccounts,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:42.833571719 +0000 UTC m=+43.649127135) (total time: 3.74447372s):
Trace[1183369946]: [3.744100001s] [3.744040361s] Listing from storage done
I0804 21:37:46.584223      34 controller.go:606] quota admission added evaluator for: leases.coordination.k8s.io
I0804 21:37:46.635981      34 trace.go:116] Trace[671553149]: "List etcd3" key:/apiextensions.k8s.io/customresourcedefinitions,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:43.872513881 +0000 UTC m=+44.688069301) (total time: 2.763417332s):
Trace[671553149]: [2.763417332s] [2.763417332s] END
I0804 21:37:46.673109      34 trace.go:116] Trace[987805534]: "List" url:/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:37:43.87245374 +0000 UTC m=+44.688009152) (total time: 2.800589674s):
Trace[987805534]: [2.763708001s] [2.763666529s] Listing from storage done
time="2021-08-04T21:37:46.957363947Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
2021-08-04 21:37:47.651956 W | wal: sync duration of 1.069484576s, expected less than 1s
time="2021-08-04T21:37:47.959560858Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
time="2021-08-04T21:37:48.961917924Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
2021-08-04 21:37:48.973215 W | etcdserver: read-only range request "key:\"/registry/services/specs/default/kubernetes\" " with result "range_response_count:1 size:601" took too long (2.388276377s) to execute
2021-08-04 21:37:48.974528 W | etcdserver: request "header:<ID:7587856269140755448 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/cloud-controller-manager\" mod_revision:66064418 > success:<request_put:<key:\"/registry/leases/kube-system/cloud-controller-manager\" value_size:420 >> failure:<request_range:<key:\"/registry/leases/kube-system/cloud-controller-manager\" > >>" with result "size:20" took too long (1.321380863s) to execute
I0804 21:37:48.975476      34 trace.go:116] Trace[1362383775]: "GuaranteedUpdate etcd3" type:*coordination.Lease (started: 2021-08-04 21:37:46.582827829 +0000 UTC m=+47.398383275) (total time: 2.392580707s):
Trace[1362383775]: [2.392549091s] [2.390875921s] Transaction committed
I0804 21:37:48.975612      34 trace.go:116] Trace[1085845936]: "GuaranteedUpdate etcd3" type:*core.Endpoints (started: 2021-08-04 21:37:46.589131574 +0000 UTC m=+47.404687052) (total time: 2.386414612s):
Trace[1085845936]: [2.386347579s] [2.384931016s] Transaction committed
I0804 21:37:48.975723      34 trace.go:116] Trace[453555065]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:46.582655481 +0000 UTC m=+47.398210916) (total time: 2.393028416s):
Trace[453555065]: [2.392911068s] [2.392817087s] Object stored in database
I0804 21:37:48.975811      34 trace.go:116] Trace[908395425]: "Update" url:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:46.588625581 +0000 UTC m=+47.404181049) (total time: 2.387142655s):
Trace[908395425]: [2.387050844s] [2.386648244s] Object stored in database
I0804 21:37:48.976625      34 leaderelection.go:252] successfully acquired lease kube-system/cloud-controller-manager
I0804 21:37:48.976872      34 trace.go:116] Trace[14994418]: "Get" url:/api/v1/namespaces/default/services/kubernetes,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:46.583478725 +0000 UTC m=+47.399034153) (total time: 2.393348346s):
Trace[14994418]: [2.393237414s] [2.393226856s] About to write a response
I0804 21:37:48.977478      34 event.go:278] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"cloud-controller-manager", UID:"fd91f4b0-8dd9-4044-a8ea-168aa027562f", APIVersion:"v1", ResourceVersion:"66064445", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' 9919a18ed6be_a4b48ee7-1cce-4c2d-a1b1-9c53af697dac became leader
I0804 21:37:48.979873      34 event.go:278] Event(v1.ObjectReference{Kind:"Lease", Namespace:"kube-system", Name:"cloud-controller-manager", UID:"d5f8f35d-69f3-4b5c-87dc-d2939592bf33", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"66064447", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' 9919a18ed6be_a4b48ee7-1cce-4c2d-a1b1-9c53af697dac became leader
I0804 21:37:48.990624      34 node_controller.go:110] Sending events to api server.
I0804 21:37:48.990917      34 controllermanager.go:247] Started "cloud-node"
I0804 21:37:48.996380      34 node_lifecycle_controller.go:78] Sending events to api server
I0804 21:37:48.996524      34 controllermanager.go:247] Started "cloud-node-lifecycle"
E0804 21:37:49.001842      34 core.go:90] Failed to start service controller: the cloud provider does not support external load balancers
W0804 21:37:49.001877      34 controllermanager.go:244] Skipping "service"
W0804 21:37:49.001902      34 core.go:108] configure-cloud-routes is set, but cloud provider does not support routes. Will not configure cloud provider routes.
W0804 21:37:49.001925      34 controllermanager.go:244] Skipping "route"
2021-08-04 21:37:49.509751 W | wal: sync duration of 1.857382904s, expected less than 1s
2021-08-04 21:37:49.513368 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:485" took too long (2.928286634s) to execute
2021-08-04 21:37:49.513758 W | etcdserver: read-only range request "key:\"/registry/clusterroles/\" range_end:\"/registry/clusterroles0\" " with result "range_response_count:110 size:98467" took too long (2.915075205s) to execute
I0804 21:37:49.514071      34 trace.go:116] Trace[1204267153]: "Get" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:46.581802212 +0000 UTC m=+47.397357624) (total time: 2.932181503s):
Trace[1204267153]: [2.932043593s] [2.932025566s] About to write a response
2021-08-04 21:37:49.515075 W | etcdserver: read-only range request "key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitions0\" " with result "range_response_count:94 size:371930" took too long (2.807497488s) to execute
I0804 21:37:49.522327      34 trace.go:116] Trace[1259579285]: "List etcd3" key:/clusterroles,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:46.59785323 +0000 UTC m=+47.413408673) (total time: 2.924416153s):
Trace[1259579285]: [2.924416153s] [2.924416153s] END
I0804 21:37:49.525361      34 trace.go:116] Trace[1954029444]: "List" url:/apis/rbac.authorization.k8s.io/v1beta1/clusterroles,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:46.597684822 +0000 UTC m=+47.413240254) (total time: 2.927638069s):
Trace[1954029444]: [2.925592217s] [2.925455893s] Listing from storage done
I0804 21:37:49.572752      34 trace.go:116] Trace[648852747]: "List etcd3" key:/apiextensions.k8s.io/customresourcedefinitions,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:46.706841246 +0000 UTC m=+47.522396681) (total time: 2.865862694s):
Trace[648852747]: [2.865862694s] [2.865862694s] END
I0804 21:37:49.608585      34 trace.go:116] Trace[1153063155]: "List" url:/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:37:46.706815563 +0000 UTC m=+47.522370984) (total time: 2.901688262s):
Trace[1153063155]: [2.865986145s] [2.86597282s] Listing from storage done
time="2021-08-04T21:37:49.964769464Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
2021-08-04 21:37:50.588658 W | wal: sync duration of 1.078519604s, expected less than 1s
time="2021-08-04T21:37:50.968077240Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
time="2021-08-04T21:37:51.970042317Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
2021-08-04 21:37:52.605060 W | wal: sync duration of 2.016087054s, expected less than 1s
2021-08-04 21:37:52.606107 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (3.601541418s) to execute
2021-08-04 21:37:52.606795 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:499" took too long (3.62683514s) to execute
2021-08-04 21:37:52.607095 W | etcdserver: read-only range request "key:\"/registry/masterleases/172.17.0.2\" " with result "range_response_count:1 size:135" took too long (3.626888643s) to execute
2021-08-04 21:37:52.607273 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/cloud-controller-manager\" " with result "range_response_count:1 size:598" took too long (3.626901915s) to execute
2021-08-04 21:37:52.607529 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/k3s\" " with result "range_response_count:1 size:508" took too long (4.016678738s) to execute
2021-08-04 21:37:52.607756 W | etcdserver: request "header:<ID:7587856269140755454 > txn:<compare:<target:MOD key:\"/registry/events/kube-system/cloud-controller-manager.16983835ca845a8a\" mod_revision:0 > success:<request_put:<key:\"/registry/events/kube-system/cloud-controller-manager.16983835ca845a8a\" value_size:685 lease:7587856269140755451 >> failure:<>>" with result "size:20" took too long (2.017459936s) to execute
I0804 21:37:52.608420      34 trace.go:116] Trace[524787921]: "Get" url:/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:48.979201597 +0000 UTC m=+49.794757018) (total time: 3.629098278s):
Trace[524787921]: [3.628975516s] [3.628967725s] About to write a response
2021-08-04 21:37:52.608692 W | etcdserver: read-only range request "key:\"/registry/namespaces/cattle-system\" " with result "range_response_count:1 size:947" took too long (2.718625195s) to execute
2021-08-04 21:37:52.610563 W | etcdserver: read-only range request "key:\"/registry/clusterrolebindings/\" range_end:\"/registry/clusterrolebindings0\" " with result "range_response_count:93 size:76427" took too long (3.067115223s) to execute
I0804 21:37:52.608872      34 trace.go:116] Trace[1263633179]: "Get" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:48.978869877 +0000 UTC m=+49.794425298) (total time: 3.629904178s):
Trace[1263633179]: [3.629729534s] [3.629700684s] About to write a response
I0804 21:37:52.608871      34 trace.go:116] Trace[1382440039]: "Get" url:/api/v1/namespaces/kube-system/configmaps/k3s,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:48.589288244 +0000 UTC m=+49.404843656) (total time: 4.019538158s):
Trace[1382440039]: [4.01939998s] [4.019383354s] About to write a response
I0804 21:37:52.611115      34 trace.go:116] Trace[2092361357]: "Get" url:/api/v1/namespaces/cattle-system,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:37:49.889358664 +0000 UTC m=+50.704914086) (total time: 2.72172475s):
Trace[2092361357]: [2.721221729s] [2.721176912s] About to write a response
I0804 21:37:52.617419      34 trace.go:116] Trace[695588863]: "Create" url:/api/v1/namespaces/kube-system/events,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/cloud-controller-manager,client:127.0.0.1 (started: 2021-08-04 21:37:48.980822323 +0000 UTC m=+49.796377771) (total time: 3.636563457s):
Trace[695588863]: [3.636451223s] [3.636184104s] Object stored in database
I0804 21:37:52.617560      34 trace.go:116] Trace[2015473193]: "List etcd3" key:/clusterrolebindings,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:49.542805089 +0000 UTC m=+50.358360504) (total time: 3.074713246s):
Trace[2015473193]: [3.074713246s] [3.074713246s] END
I0804 21:37:52.618435      34 trace.go:116] Trace[1681050472]: "List" url:/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:49.542713411 +0000 UTC m=+50.358268832) (total time: 3.075688003s):
Trace[1681050472]: [3.074904157s] [3.07483841s] Listing from storage done
2021-08-04 21:37:55.033885 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (3.024543154s) to execute
2021-08-04 21:37:55.034043 W | etcdserver: request "header:<ID:7587856269140755459 > lease_grant:<ttl:15-second id:694d7b1318db0402>" with result "size:44" took too long (2.044208443s) to execute
2021-08-04 21:37:55.034175 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:578" took too long (3.095815112s) to execute
I0804 21:37:55.035187      34 trace.go:116] Trace[790643612]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-scheduler,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:51.937327392 +0000 UTC m=+52.752882815) (total time: 3.097777845s):
Trace[790643612]: [3.097620136s] [3.097569326s] About to write a response
I0804 21:37:55.035902      34 trace.go:116] Trace[2139749666]: "GuaranteedUpdate etcd3" type:*coordination.Lease (started: 2021-08-04 21:37:52.614200261 +0000 UTC m=+53.429755698) (total time: 2.421653508s):
Trace[2139749666]: [2.421588605s] [2.420805636s] Transaction committed
I0804 21:37:55.036144      34 trace.go:116] Trace[1275953149]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:52.614008588 +0000 UTC m=+53.429564000) (total time: 2.422085247s):
Trace[1275953149]: [2.421978007s] [2.421890594s] Object stored in database
I0804 21:37:55.036184      34 trace.go:116] Trace[58036118]: "Create" url:/api/v1/namespaces/kube-system/events,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/cloud-controller-manager,client:127.0.0.1 (started: 2021-08-04 21:37:52.619908529 +0000 UTC m=+53.435463941) (total time: 2.416240241s):
Trace[58036118]: [2.416119132s] [2.416032949s] Object stored in database
I0804 21:37:55.035902      34 trace.go:116] Trace[118603121]: "GuaranteedUpdate etcd3" type:*core.ConfigMap (started: 2021-08-04 21:37:52.614335651 +0000 UTC m=+53.429891075) (total time: 2.421517272s):
Trace[118603121]: [2.421453211s] [2.420368723s] Transaction committed
I0804 21:37:55.037027      34 trace.go:116] Trace[826815172]: "Update" url:/api/v1/namespaces/kube-system/configmaps/k3s,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:52.614076233 +0000 UTC m=+53.429631668) (total time: 2.422785863s):
Trace[826815172]: [2.422656394s] [2.422455953s] Object stored in database
I0804 21:37:55.037597      34 leaderelection.go:252] successfully acquired lease kube-system/kube-controller-manager
I0804 21:37:55.038205      34 event.go:278] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-controller-manager", UID:"34608d2c-8b9a-43eb-add2-fff8918dde72", APIVersion:"v1", ResourceVersion:"66064448", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' 9919a18ed6be_027e99c7-3ef1-4b3e-8542-9fa710ff284b became leader
I0804 21:37:55.038282      34 event.go:278] Event(v1.ObjectReference{Kind:"Lease", Namespace:"kube-system", Name:"kube-controller-manager", UID:"3dccda3d-2870-45f9-8a09-4cde7ba5860f", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"66064451", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' 9919a18ed6be_027e99c7-3ef1-4b3e-8542-9fa710ff284b became leader
2021-08-04 21:37:55.070007 W | wal: sync duration of 2.080558102s, expected less than 1s
2021-08-04 21:37:55.071262 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/settings/first-login\" " with result "range_response_count:1 size:563" took too long (2.406623176s) to execute
2021-08-04 21:37:55.071898 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/settings\" range_end:\"/registry/management.cattle.io/settingt\" count_only:true " with result "range_response_count:0 size:9" took too long (2.408721931s) to execute
2021-08-04 21:37:55.072100 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/cloud-controller-manager\" " with result "range_response_count:1 size:503" took too long (2.454310559s) to execute
2021-08-04 21:37:55.072788 W | etcdserver: read-only range request "key:\"/registry/configmaps/\" range_end:\"/registry/configmaps0\" " with result "range_response_count:13 size:31906" took too long (2.446116513s) to execute
2021-08-04 21:37:55.073886 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/settings/\" range_end:\"/registry/management.cattle.io/settings0\" limit:10000 " with result "range_response_count:81 size:61785" took too long (2.410736437s) to execute
I0804 21:37:55.075152      34 trace.go:116] Trace[2133912374]: "Get" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:52.616895003 +0000 UTC m=+53.432450413) (total time: 2.458170838s):
Trace[2133912374]: [2.458001221s] [2.457991937s] About to write a response
I0804 21:37:55.076722      34 trace.go:116] Trace[599529242]: "List etcd3" key:/configmaps,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:52.626067858 +0000 UTC m=+53.441623279) (total time: 2.450562363s):
Trace[599529242]: [2.450562363s] [2.450562363s] END
I0804 21:37:55.077231      34 trace.go:116] Trace[1268639727]: "List" url:/api/v1/configmaps,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:52.62600942 +0000 UTC m=+53.441564843) (total time: 2.451168308s):
Trace[1268639727]: [2.45084102s] [2.45078953s] Listing from storage done
I0804 21:37:55.078358      34 trace.go:116] Trace[501745253]: "Get" url:/apis/management.cattle.io/v3/settings/first-login,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:37:52.664181541 +0000 UTC m=+53.479736976) (total time: 2.414127308s):
Trace[501745253]: [2.413791396s] [2.413736791s] About to write a response
I0804 21:37:55.099638      34 trace.go:116] Trace[1973102802]: "List etcd3" key:/management.cattle.io/settings,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:52.662441915 +0000 UTC m=+53.477997417) (total time: 2.437157971s):
Trace[1973102802]: [2.437157971s] [2.437157971s] END
2021-08-04 21:37:57.733665 W | wal: sync duration of 2.484694298s, expected less than 1s
2021-08-04 21:37:58.139329 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:485" took too long (3.100140445s) to execute
2021-08-04 21:37:58.139526 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:596" took too long (3.099902094s) to execute
2021-08-04 21:37:58.139686 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (3.130943605s) to execute
2021-08-04 21:37:58.139974 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (3.09185573s) to execute
2021-08-04 21:37:58.140263 W | etcdserver: request "header:<ID:7587856269140755468 > lease_grant:<ttl:15-second id:694d7b1318db040b>" with result "size:44" took too long (2.891141235s) to execute
I0804 21:37:58.140841      34 trace.go:116] Trace[678366839]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:55.039271 +0000 UTC m=+55.854826415) (total time: 3.101479744s):
Trace[678366839]: [3.101367064s] [3.101348701s] About to write a response
I0804 21:37:58.140849      34 trace.go:116] Trace[1617861981]: "Get" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:55.038685035 +0000 UTC m=+55.854240456) (total time: 3.102079734s):
Trace[1617861981]: [3.101948176s] [3.101912946s] About to write a response
I0804 21:37:58.142520      34 trace.go:116] Trace[1569664990]: "GuaranteedUpdate etcd3" type:*core.Endpoints (started: 2021-08-04 21:37:55.078480408 +0000 UTC m=+55.894035848) (total time: 3.063974924s):
Trace[1569664990]: [3.06392986s] [3.063155993s] Transaction committed
2021-08-04 21:37:58.142937 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/k3s\" " with result "range_response_count:1 size:508" took too long (1.100610035s) to execute
I0804 21:37:58.143335      34 trace.go:116] Trace[1152121597]: "Update" url:/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:55.078227074 +0000 UTC m=+55.893782511) (total time: 3.065077734s):
Trace[1152121597]: [3.064971769s] [3.064776062s] Object stored in database
2021-08-04 21:37:58.143340 W | etcdserver: read-only range request "key:\"/registry/deployments/\" range_end:\"/registry/deployments0\" " with result "range_response_count:6 size:20789" took too long (3.056785017s) to execute
2021-08-04 21:37:58.143554 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (134.638856ms) to execute
2021-08-04 21:37:58.144096 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/settings/ingress-ip-domain\" " with result "range_response_count:1 size:571" took too long (3.060277762s) to execute
I0804 21:37:58.146047      34 trace.go:116] Trace[1143224240]: "Create" url:/api/v1/namespaces/kube-system/events,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/kube-controller-manager,client:127.0.0.1 (started: 2021-08-04 21:37:55.078760864 +0000 UTC m=+55.894316290) (total time: 3.067243352s):
Trace[1143224240]: [3.067152858s] [3.067073174s] Object stored in database
I0804 21:37:58.146363      34 trace.go:116] Trace[2036443019]: "List etcd3" key:/deployments,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:55.086026815 +0000 UTC m=+55.901582235) (total time: 3.060299124s):
Trace[2036443019]: [3.060299124s] [3.060299124s] END
I0804 21:37:58.146700      34 trace.go:116] Trace[654677306]: "Get" url:/api/v1/namespaces/kube-system/configmaps/k3s,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:57.041518991 +0000 UTC m=+57.857074414) (total time: 1.10514262s):
Trace[654677306]: [1.105072972s] [1.105052329s] About to write a response
I0804 21:37:58.147089      34 trace.go:116] Trace[1677292233]: "GuaranteedUpdate etcd3" type:*v1.Endpoints (started: 2021-08-04 21:37:48.978073768 +0000 UTC m=+49.793629240) (total time: 9.168985657s):
Trace[1677292233]: [3.630117482s] [3.630117482s] initial value restored
Trace[1677292233]: [6.056748875s] [2.426631393s] Transaction prepared
Trace[1677292233]: [9.16403596s] [3.067305911s] Transaction prepared
I0804 21:37:58.147865      34 trace.go:116] Trace[1119830032]: "Get" url:/apis/management.cattle.io/v3/settings/ingress-ip-domain,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:37:55.083463505 +0000 UTC m=+55.899018940) (total time: 3.064365801s):
Trace[1119830032]: [3.064088485s] [3.064049397s] About to write a response
I0804 21:37:58.148678      34 trace.go:116] Trace[1685972099]: "List" url:/apis/apps/v1/deployments,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:55.08587952 +0000 UTC m=+55.901434960) (total time: 3.062754218s):
Trace[1685972099]: [3.060578298s] [3.060525856s] Listing from storage done
jade@rm:~$ docker logs --tail 1000 intelligent_euclid
may have crashed in this repository earlier:
remove the file manually to continue.
, requeuing
2021-08-04 21:31:42.182466 W | wal: sync duration of 4.53214467s, expected less than 1s
2021-08-04 21:31:42.183649 W | etcdserver: read-only range request "key:\"/registry/project.cattle.io/pipelinesettings\" range_end:\"/registry/project.cattle.io/pipelinesettingt\" count_only:true " with result "range_response_count:0 size:9" took too long (5.280390003s) to execute
2021-08-04 21:31:42.183745 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/clusteralertrules/\" range_end:\"/registry/management.cattle.io/clusteralertrules0\" limit:1 " with result "range_response_count:1 size:1165" took too long (5.447657389s) to execute
2021-08-04 21:31:42.183977 W | etcdserver: read-only range request "key:\"/registry/monitoring.coreos.com/prometheusrules/\" range_end:\"/registry/monitoring.coreos.com/prometheusrules0\" limit:10000 " with result "range_response_count:0 size:7" took too long (5.23234094s) to execute
2021-08-04 21:31:42.184151 W | etcdserver: read-only range request "key:\"/registry/monitoring.coreos.com/prometheusrules\" range_end:\"/registry/monitoring.coreos.com/prometheusrulet\" count_only:true " with result "range_response_count:0 size:7" took too long (5.232627424s) to execute
2021-08-04 21:31:42.184383 W | etcdserver: read-only range request "key:\"/registry/monitoring.coreos.com/prometheusrules/\" range_end:\"/registry/monitoring.coreos.com/prometheusrules0\" limit:1 " with result "range_response_count:0 size:7" took too long (5.231388812s) to execute
2021-08-04 21:31:42.184899 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/catalogtemplateversions\" range_end:\"/registry/management.cattle.io/catalogtemplateversiont\" count_only:true " with result "range_response_count:0 size:10" took too long (5.185325912s) to execute
2021-08-04 21:31:42.185206 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/clusteralertrules\" range_end:\"/registry/management.cattle.io/clusteralertrulet\" count_only:true " with result "range_response_count:0 size:9" took too long (5.449437245s) to execute
2021-08-04 21:31:42.185437 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/catalogtemplateversions/\" range_end:\"/registry/management.cattle.io/catalogtemplateversions0\" limit:1 " with result "range_response_count:1 size:1439" took too long (5.184565972s) to execute
2021-08-04 21:31:42.186390 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:503" took too long (5.612172643s) to execute
2021-08-04 21:31:42.186628 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/cloud-controller-manager\" " with result "range_response_count:1 size:505" took too long (5.612481538s) to execute
2021-08-04 21:31:42.187042 W | etcdserver: read-only range request "key:\"/registry/pods/\" range_end:\"/registry/pods0\" limit:1 " with result "range_response_count:1 size:3169" took too long (5.617159401s) to execute
2021-08-04 21:31:42.187356 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/clustertemplates/\" range_end:\"/registry/management.cattle.io/clustertemplates0\" limit:1 " with result "range_response_count:0 size:7" took too long (5.503262683s) to execute
I0804 21:31:42.187897      32 trace.go:116] Trace[1926102680]: "List etcd3" key:/monitoring.coreos.com/prometheusrules,resourceVersion:,limit:1,continue: (started: 2021-08-04 21:31:36.952364713 +0000 UTC m=+53.354978698) (total time: 5.235219444s):
Trace[1926102680]: [5.235219444s] [5.235219444s] END
2021-08-04 21:31:42.188550 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/clustertemplates\" range_end:\"/registry/management.cattle.io/clustertemplatet\" count_only:true " with result "range_response_count:0 size:7" took too long (5.506627037s) to execute
2021-08-04 21:31:42.188750 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/clustertemplates/\" range_end:\"/registry/management.cattle.io/clustertemplates0\" limit:10000 " with result "range_response_count:0 size:7" took too long (5.506942031s) to execute
I0804 21:31:42.189036      32 trace.go:116] Trace[1794800049]: "List" url:/apis/monitoring.coreos.com/v1/prometheusrules,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:31:36.952260451 +0000 UTC m=+53.354874435) (total time: 5.236598459s):
Trace[1794800049]: [5.235911262s] [5.235818983s] Listing from storage done
2021-08-04 21:31:42.189279 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/catalogs/\" range_end:\"/registry/management.cattle.io/catalogs0\" limit:1 " with result "range_response_count:1 size:3483" took too long (5.566155541s) to execute
2021-08-04 21:31:42.189672 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/catalogs\" range_end:\"/registry/management.cattle.io/catalogt\" count_only:true " with result "range_response_count:0 size:9" took too long (5.568873243s) to execute
2021-08-04 21:31:42.189835 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/clusteralertrules/\" range_end:\"/registry/management.cattle.io/clusteralertrules0\" limit:10000 " with result "range_response_count:28 size:35573" took too long (5.451461818s) to execute
2021-08-04 21:31:42.190257 W | etcdserver: read-only range request "key:\"/registry/endpointslices/default/kubernetes\" " with result "range_response_count:1 size:473" took too long (5.617802647s) to execute
I0804 21:31:42.190321      32 trace.go:116] Trace[1860819510]: "List etcd3" key:/management.cattle.io/catalogtemplateversions,resourceVersion:,limit:1,continue: (started: 2021-08-04 21:31:37.00039313 +0000 UTC m=+53.403007124) (total time: 5.189880148s):
Trace[1860819510]: [5.189880148s] [5.189880148s] END
I0804 21:31:42.190296      32 trace.go:116] Trace[1572631335]: "Get" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:31:36.572320423 +0000 UTC m=+52.974934416) (total time: 5.617773233s):
Trace[1572631335]: [5.617391335s] [5.617367865s] About to write a response
I0804 21:31:42.190457      32 trace.go:116] Trace[1581496527]: "List etcd3" key:/monitoring.coreos.com/prometheusrules,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:31:36.950743697 +0000 UTC m=+53.353357692) (total time: 5.239480343s):
Trace[1581496527]: [5.239480343s] [5.239480343s] END
2021-08-04 21:31:42.190578 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/catalogs/\" range_end:\"/registry/management.cattle.io/catalogs0\" limit:10000 " with result "range_response_count:3 size:27859" took too long (5.562870855s) to execute
I0804 21:31:42.190697      32 trace.go:116] Trace[1000164281]: "List etcd3" key:/pods,resourceVersion:,limit:1,continue: (started: 2021-08-04 21:31:36.565461814 +0000 UTC m=+52.968075808) (total time: 5.624832231s):
Trace[1000164281]: [5.624832231s] [5.624832231s] END
2021-08-04 21:31:42.190888 W | etcdserver: read-only range request "key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitions0\" " with result "range_response_count:94 size:371930" took too long (5.601155645s) to execute
I0804 21:31:42.190927      32 trace.go:116] Trace[1156036491]: "List" url:/apis/management.cattle.io/v3/catalogtemplateversions,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:31:37.00028902 +0000 UTC m=+53.402903019) (total time: 5.190590072s):
Trace[1156036491]: [5.190064358s] [5.189973635s] Listing from storage done
I0804 21:31:42.189190      32 trace.go:116] Trace[1257334480]: "List etcd3" key:/management.cattle.io/clusteralertrules,resourceVersion:,limit:1,continue: (started: 2021-08-04 21:31:36.735618792 +0000 UTC m=+53.138232786) (total time: 5.453512524s):
Trace[1257334480]: [5.453512524s] [5.453512524s] END
I0804 21:31:42.191730      32 trace.go:116] Trace[1293761390]: "List" url:/api/v1/pods,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:31:36.565404711 +0000 UTC m=+52.968018707) (total time: 5.626113916s):
Trace[1293761390]: [5.625322139s] [5.625299291s] Listing from storage done
I0804 21:31:42.192845      32 trace.go:116] Trace[346727393]: "List" url:/apis/management.cattle.io/v3/clusteralertrules,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:31:36.73549929 +0000 UTC m=+53.138113275) (total time: 5.457313846s):
Trace[346727393]: [5.456021051s] [5.45591274s] Listing from storage done
I0804 21:31:42.193751      32 trace.go:116] Trace[687364175]: "List etcd3" key:/management.cattle.io/catalogs,resourceVersion:,limit:1,continue: (started: 2021-08-04 21:31:36.622674551 +0000 UTC m=+53.025288545) (total time: 5.571050008s):
Trace[687364175]: [5.571050008s] [5.571050008s] END
I0804 21:31:42.197519      32 trace.go:116] Trace[628760028]: "List" url:/apis/management.cattle.io/v3/catalogs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:31:36.622567743 +0000 UTC m=+53.025181737) (total time: 5.574901879s):
Trace[628760028]: [5.57121988s] [5.57112353s] Listing from storage done
2021-08-04 21:31:42.198699 W | etcdserver: read-only range request "key:\"/registry/project.cattle.io/pipelinesettings/\" range_end:\"/registry/project.cattle.io/pipelinesettings0\" limit:1 " with result "range_response_count:1 size:730" took too long (5.279631924s) to execute
2021-08-04 21:31:42.198971 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/catalogtemplateversions/\" range_end:\"/registry/management.cattle.io/catalogtemplateversions0\" limit:10000 " with result "range_response_count:313 size:421515" took too long (5.19539868s) to execute
2021-08-04 21:31:42.200002 W | etcdserver: read-only range request "key:\"/registry/serviceaccounts/kube-system/certificate-controller\" " with result "range_response_count:1 size:265" took too long (5.611041474s) to execute
2021-08-04 21:31:42.200202 W | etcdserver: read-only range request "key:\"/registry/monitoring.coreos.com/servicemonitors/\" range_end:\"/registry/monitoring.coreos.com/servicemonitors0\" limit:10000 " with result "range_response_count:0 size:7" took too long (5.349253258s) to execute
I0804 21:31:42.200243      32 trace.go:116] Trace[1582082661]: "Get" url:/apis/discovery.k8s.io/v1beta1/namespaces/default/endpointslices/kubernetes,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:31:36.56538729 +0000 UTC m=+52.968001303) (total time: 5.634634748s):
Trace[1582082661]: [5.634505445s] [5.63449296s] About to write a response
2021-08-04 21:31:42.200506 W | etcdserver: read-only range request "key:\"/registry/project.cattle.io/pipelinesettings/\" range_end:\"/registry/project.cattle.io/pipelinesettings0\" limit:10000 " with result "range_response_count:28 size:20285" took too long (5.29349217s) to execute
2021-08-04 21:31:42.201003 W | etcdserver: read-only range request "key:\"/registry/monitoring.coreos.com/servicemonitors/\" range_end:\"/registry/monitoring.coreos.com/servicemonitors0\" limit:1 " with result "range_response_count:0 size:7" took too long (5.347891064s) to execute
2021-08-04 21:31:42.201222 W | etcdserver: read-only range request "key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitions0\" " with result "range_response_count:94 size:371930" took too long (5.611709117s) to execute
I0804 21:31:42.201748      32 trace.go:116] Trace[441484854]: "List etcd3" key:/management.cattle.io/clustertemplates,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:31:36.681103594 +0000 UTC m=+53.083717583) (total time: 5.520587207s):
Trace[441484854]: [5.520587207s] [5.520587207s] END
2021-08-04 21:31:42.201914 W | etcdserver: read-only range request "key:\"/registry/monitoring.coreos.com/servicemonitors\" range_end:\"/registry/monitoring.coreos.com/servicemonitort\" count_only:true " with result "range_response_count:0 size:7" took too long (5.349734674s) to execute
2021-08-04 21:31:42.202116 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/multiclusterapps/\" range_end:\"/registry/management.cattle.io/multiclusterapps0\" limit:1 " with result "range_response_count:0 size:7" took too long (5.397690744s) to execute
I0804 21:31:42.202126      32 trace.go:116] Trace[21560798]: "List etcd3" key:/management.cattle.io/clustertemplates,resourceVersion:,limit:1,continue: (started: 2021-08-04 21:31:36.683420686 +0000 UTC m=+53.086034679) (total time: 5.51868031s):
Trace[21560798]: [5.51868031s] [5.51868031s] END
2021-08-04 21:31:42.202344 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/cattle-controllers\" " with result "range_response_count:1 size:542" took too long (5.397851209s) to execute
I0804 21:31:42.202379      32 trace.go:116] Trace[1224480056]: "List" url:/apis/management.cattle.io/v3/clustertemplates,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:31:36.683295804 +0000 UTC m=+53.085909789) (total time: 5.519050914s):
Trace[1224480056]: [5.518881269s] [5.518767713s] Listing from storage done
2021-08-04 21:31:42.202485 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/multiclusterapps\" range_end:\"/registry/management.cattle.io/multiclusterappt\" count_only:true " with result "range_response_count:0 size:7" took too long (5.398705169s) to execute
2021-08-04 21:31:42.203170 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/multiclusterapps/\" range_end:\"/registry/management.cattle.io/multiclusterapps0\" limit:10000 " with result "range_response_count:0 size:7" took too long (5.405687912s) to execute
I0804 21:31:42.203764      32 trace.go:116] Trace[362571306]: "Get" url:/api/v1/namespaces/kube-system/configmaps/cattle-controllers,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:31:36.798586295 +0000 UTC m=+53.201200285) (total time: 5.405132459s):
Trace[362571306]: [5.405058903s] [5.405001903s] About to write a response
I0804 21:31:42.204193      32 trace.go:116] Trace[760536119]: "List etcd3" key:/management.cattle.io/multiclusterapps,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:31:36.796745788 +0000 UTC m=+53.199359819) (total time: 5.407416522s):
Trace[760536119]: [5.407416522s] [5.407416522s] END
I0804 21:31:42.205932      32 trace.go:116] Trace[860133740]: "List etcd3" key:/management.cattle.io/multiclusterapps,resourceVersion:,limit:1,continue: (started: 2021-08-04 21:31:36.798951829 +0000 UTC m=+53.201565823) (total time: 5.406952866s):
Trace[860133740]: [5.406952866s] [5.406952866s] END
I0804 21:31:42.206305      32 trace.go:116] Trace[2021432974]: "List" url:/apis/management.cattle.io/v3/multiclusterapps,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:31:36.798863995 +0000 UTC m=+53.201477988) (total time: 5.407366367s):
Trace[2021432974]: [5.40712785s] [5.407050654s] Listing from storage done
I0804 21:31:42.207162      32 trace.go:116] Trace[1548731900]: "List etcd3" key:/monitoring.coreos.com/servicemonitors,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:31:36.846379845 +0000 UTC m=+53.248993865) (total time: 5.36072089s):
Trace[1548731900]: [5.36072089s] [5.36072089s] END
I0804 21:31:42.207623      32 trace.go:116] Trace[1042507316]: "List etcd3" key:/monitoring.coreos.com/servicemonitors,resourceVersion:,limit:1,continue: (started: 2021-08-04 21:31:36.848552287 +0000 UTC m=+53.251166279) (total time: 5.359001909s):
Trace[1042507316]: [5.359001909s] [5.359001909s] END
I0804 21:31:42.207966      32 trace.go:116] Trace[1643783821]: "List" url:/apis/monitoring.coreos.com/v1/servicemonitors,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:31:36.848459387 +0000 UTC m=+53.251073381) (total time: 5.359426865s):
Trace[1643783821]: [5.359269036s] [5.359187151s] Listing from storage done
I0804 21:31:42.209432      32 trace.go:116] Trace[586494907]: "List etcd3" key:/project.cattle.io/pipelinesettings,resourceVersion:,limit:1,continue: (started: 2021-08-04 21:31:36.90444031 +0000 UTC m=+53.307054314) (total time: 5.30492569s):
Trace[586494907]: [5.30492569s] [5.30492569s] END
I0804 21:31:42.209927      32 trace.go:116] Trace[523837048]: "List" url:/apis/project.cattle.io/v3/pipelinesettings,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:31:36.904322723 +0000 UTC m=+53.306936761) (total time: 5.305564235s):
Trace[523837048]: [5.305154391s] [5.305054968s] Listing from storage done
I0804 21:31:42.215885      32 trace.go:116] Trace[1041070638]: "List etcd3" key:/management.cattle.io/clusteralertrules,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:31:36.734069411 +0000 UTC m=+53.136683447) (total time: 5.481774927s):
Trace[1041070638]: [5.481774927s] [5.481774927s] END
I0804 21:31:42.217106      32 trace.go:116] Trace[1847708527]: "List etcd3" key:/management.cattle.io/catalogs,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:31:36.620039094 +0000 UTC m=+53.022653091) (total time: 5.597037267s):
Trace[1847708527]: [5.597037267s] [5.597037267s] END
I0804 21:31:42.218413      32 trace.go:116] Trace[1411831583]: "Get" url:/api/v1/namespaces/kube-system/serviceaccounts/certificate-controller,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/kube-controller-manager,client:127.0.0.1 (started: 2021-08-04 21:31:36.584152484 +0000 UTC m=+52.986766480) (total time: 5.634221392s):
Trace[1411831583]: [5.634118239s] [5.634109862s] About to write a response
I0804 21:31:42.219001      32 trace.go:116] Trace[470588203]: "Get" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:31:36.572187899 +0000 UTC m=+52.974801902) (total time: 5.646759538s):
Trace[470588203]: [5.646703674s] [5.646650732s] About to write a response
I0804 21:31:42.224283      32 controllermanager.go:533] Started "csrsigning"
I0804 21:31:42.224667      32 certificate_controller.go:119] Starting certificate controller "csrsigning"
I0804 21:31:42.224759      32 shared_informer.go:223] Waiting for caches to sync for certificate-csrsigning
I0804 21:31:42.225093      32 dynamic_serving_content.go:130] Starting csr-controller::/var/lib/rancher/k3s/server/tls/server-ca.crt::/var/lib/rancher/k3s/server/tls/server-ca.key
I0804 21:31:42.230566      32 trace.go:116] Trace[908655050]: "List etcd3" key:/project.cattle.io/pipelinesettings,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:31:36.902405123 +0000 UTC m=+53.305019145) (total time: 5.328123268s):
Trace[908655050]: [5.328123268s] [5.328123268s] END
I0804 21:31:42.284748      32 trace.go:116] Trace[820824016]: "List etcd3" key:/apiextensions.k8s.io/customresourcedefinitions,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:31:36.583064124 +0000 UTC m=+52.985678120) (total time: 5.701636428s):
Trace[820824016]: [5.701636428s] [5.701636428s] END
I0804 21:31:42.290230      32 trace.go:116] Trace[1636169343]: "List etcd3" key:/apiextensions.k8s.io/customresourcedefinitions,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:31:36.583941321 +0000 UTC m=+52.986555390) (total time: 5.706252611s):
Trace[1636169343]: [5.706252611s] [5.706252611s] END
I0804 21:31:42.328617      32 trace.go:116] Trace[1127913225]: "List" url:/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:31:36.583027254 +0000 UTC m=+52.985641241) (total time: 5.745512779s):
Trace[1127913225]: [5.70189191s] [5.701870968s] Listing from storage done
I0804 21:31:42.331516      32 trace.go:116] Trace[977360776]: "List" url:/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:31:36.583925112 +0000 UTC m=+52.986539118) (total time: 5.747542334s):
Trace[977360776]: [5.706466305s] [5.706456042s] Listing from storage done
I0804 21:31:42.388859      32 trace.go:116] Trace[1687398372]: "List etcd3" key:/management.cattle.io/catalogtemplateversions,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:31:36.999067372 +0000 UTC m=+53.401681401) (total time: 5.389746476s):
Trace[1687398372]: [5.389746476s] [5.389746476s] END
2021-08-04 21:31:42.672028 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/groupmembers/\" range_end:\"/registry/management.cattle.io/groupmembers0\" limit:1 " with result "range_response_count:0 size:7" took too long (3.282042918s) to execute
2021-08-04 21:31:42.673125 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/k3s\" " with result "range_response_count:1 size:508" took too long (1.278185086s) to execute
2021-08-04 21:31:42.673303 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:485" took too long (3.279096505s) to execute
2021-08-04 21:31:42.673419 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/authconfigs/azuread\" " with result "range_response_count:1 size:571" took too long (3.281709188s) to execute
2021-08-04 21:31:42.673518 W | etcdserver: request "header:<ID:7587856269035255061 > txn:<compare:<target:MOD key:\"/registry/services/endpoints/kube-system/cloud-controller-manager\" mod_revision:66064412 > success:<request_put:<key:\"/registry/services/endpoints/kube-system/cloud-controller-manager\" value_size:503 >> failure:<request_range:<key:\"/registry/services/endpoints/kube-system/cloud-controller-manager\" > >>" with result "size:20" took too long (143.505554ms) to execute
I0804 21:31:42.674409      32 trace.go:116] Trace[1818477160]: "Get" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:31:39.392689606 +0000 UTC m=+55.795303645) (total time: 3.28165004s):
Trace[1818477160]: [3.281570721s] [3.281548249s] About to write a response
I0804 21:31:42.676476      32 trace.go:116] Trace[1536534930]: "List etcd3" key:/management.cattle.io/groupmembers,resourceVersion:,limit:1,continue: (started: 2021-08-04 21:31:39.388395704 +0000 UTC m=+55.791009743) (total time: 3.28803892s):
Trace[1536534930]: [3.28803892s] [3.28803892s] END
I0804 21:31:42.675181      32 trace.go:116] Trace[1731687510]: "Get" url:/api/v1/namespaces/kube-system/configmaps/k3s,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:31:41.392907488 +0000 UTC m=+57.795521557) (total time: 1.282241919s):
Trace[1731687510]: [1.282178289s] [1.282161322s] About to write a response
I0804 21:31:42.676008      32 trace.go:116] Trace[1003251188]: "Create" url:/apis/management.cattle.io/v3/authconfigs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:31:36.566017176 +0000 UTC m=+52.968631262) (total time: 6.109935034s):
Trace[1003251188]: [6.109935034s] [6.109440532s] END
I0804 21:31:42.676766      32 trace.go:116] Trace[843142156]: "List" url:/apis/management.cattle.io/v3/groupmembers,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:31:39.388242967 +0000 UTC m=+55.790857120) (total time: 3.288492743s):
Trace[843142156]: [3.288328132s] [3.28821071s] Listing from storage done
2021/08/04 21:31:42 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 reset --hard FETCH_HEAD error: exit status 128, detail: fatal: Unable to create '/var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
, requeuing
E0804 21:31:44.817468      32 leaderelection.go:356] Failed to update lock: resource name may not be empty
I0804 21:31:44.817809      32 leaderelection.go:277] failed to renew lease kube-system/cloud-controller-manager: timed out waiting for the condition
2021-08-04 21:31:44.818035 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/cloud-controller-manager\" " with result "error:context canceled" took too long (2.13764141s) to execute
F0804 21:31:44.818507      32 controllermanager.go:213] leaderelection lost
E0804 21:31:44.817660      32 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}
I0804 21:31:44.818246      32 event.go:278] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"cloud-controller-manager", UID:"fd91f4b0-8dd9-4044-a8ea-168aa027562f", APIVersion:"v1", ResourceVersion:"66064425", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' 9919a18ed6be_fe02e4b6-2270-4233-8442-ce45e48ce3c5 stopped leading
2021-08-04 21:31:45.227174 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/clustertemplaterevisions/\" range_end:\"/registry/management.cattle.io/clustertemplaterevisions0\" limit:1 " with result "error:context canceled" took too long (2.485599029s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.227693 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/clustertemplaterevisions\" range_end:\"/registry/management.cattle.io/clustertemplaterevisiont\" count_only:true " with result "error:context canceled" took too long (2.487125205s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.228068 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/clustertemplaterevisions/\" range_end:\"/registry/management.cattle.io/clustertemplaterevisions0\" limit:10000 " with result "error:context canceled" took too long (2.487270596s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.228618 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/globaldnsproviders\" range_end:\"/registry/management.cattle.io/globaldnsprovidert\" count_only:true " with result "error:context canceled" took too long (2.895730639s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.228945 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/projectcatalogs/\" range_end:\"/registry/management.cattle.io/projectcatalogs0\" limit:1 " with result "error:context canceled" took too long (2.762139207s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.229241 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/projectcatalogs/\" range_end:\"/registry/management.cattle.io/projectcatalogs0\" limit:10000 " with result "error:context canceled" took too long (2.764327589s) to execute
2021-08-04 21:31:45.229268 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/templates/\" range_end:\"/registry/management.cattle.io/templates0\" limit:10000 " with result "error:context canceled" took too long (2.709715228s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.229373 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/templates/\" range_end:\"/registry/management.cattle.io/templates0\" limit:1 " with result "error:context canceled" took too long (2.708681568s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.229432 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/projectcatalogs\" range_end:\"/registry/management.cattle.io/projectcatalogt\" count_only:true " with result "error:context canceled" took too long (2.763792041s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.229539 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/templates\" range_end:\"/registry/management.cattle.io/templatet\" count_only:true " with result "error:context canceled" took too long (2.710034042s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.229866 W | etcdserver: read-only range request "key:\"/registry/fleet.cattle.io/bundlenamespacemappings/\" range_end:\"/registry/fleet.cattle.io/bundlenamespacemappings0\" limit:1 " with result "error:context canceled" took too long (2.8476099s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.230357 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/globaldnsproviders/\" range_end:\"/registry/management.cattle.io/globaldnsproviders0\" limit:1 " with result "error:context canceled" took too long (2.897641117s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.231416 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/clustercatalogs/\" range_end:\"/registry/management.cattle.io/clustercatalogs0\" limit:1 " with result "error:context canceled" took too long (2.949920704s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.231796 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/globaldnsproviders/\" range_end:\"/registry/management.cattle.io/globaldnsproviders0\" limit:10000 " with result "error:context canceled" took too long (2.8976324s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.232219 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/clustercatalogs/\" range_end:\"/registry/management.cattle.io/clustercatalogs0\" limit:10000 " with result "error:context canceled" took too long (2.951871342s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.235370 W | etcdserver: read-only range request "key:\"/registry/fleet.cattle.io/bundlenamespacemappings\" range_end:\"/registry/fleet.cattle.io/bundlenamespacemappingt\" count_only:true " with result "error:context canceled" took too long (2.850632814s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.235425 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/clustercatalogs\" range_end:\"/registry/management.cattle.io/clustercatalogt\" count_only:true " with result "error:context canceled" took too long (2.951532427s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.235855 W | etcdserver: read-only range request "key:\"/registry/fleet.cattle.io/bundlenamespacemappings/\" range_end:\"/registry/fleet.cattle.io/bundlenamespacemappings0\" limit:10000 " with result "error:context canceled" took too long (2.850723148s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.236231 W | etcdserver: read-only range request "key:\"/registry/project.cattle.io/apprevisions\" range_end:\"/registry/project.cattle.io/apprevisiont\" count_only:true " with result "error:context canceled" took too long (2.810146413s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.236390 W | etcdserver: read-only range request "key:\"/registry/project.cattle.io/apprevisions/\" range_end:\"/registry/project.cattle.io/apprevisions0\" limit:10000 " with result "error:context canceled" took too long (2.810222418s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.236583 W | etcdserver: read-only range request "key:\"/registry/events/\" range_end:\"/registry/events0\" limit:1 " with result "error:context canceled" took too long (3.015992541s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.236745 W | etcdserver: read-only range request "key:\"/registry/csinodes/\" range_end:\"/registry/csinodes0\" limit:1 " with result "error:context canceled" took too long (3.029625523s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.236872 W | etcdserver: read-only range request "key:\"/registry/controllerrevisions/\" range_end:\"/registry/controllerrevisions0\" limit:1 " with result "error:context canceled" took too long (3.018514419s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.237045 W | etcdserver: read-only range request "key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitions0\" " with result "error:context canceled" took too long (2.873407912s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.237228 W | etcdserver: read-only range request "key:\"/registry/serviceaccounts/kube-system/ttl-controller\" " with result "error:context canceled" took too long (2.987718697s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.237371 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/cattle-controllers\" " with result "error:context canceled" took too long (553.461009ms) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.237509 W | etcdserver: read-only range request "key:\"/registry/namespaces/default\" " with result "error:context canceled" took too long (3.019132005s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.237659 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "error:context canceled" took too long (553.056944ms) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:31:45.237855 W | etcdserver: read-only range request "key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitions0\" " with result "error:context canceled" took too long (2.868321846s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021/08/04 21:31:45 [FATAL] k3s exited with: exit status 255
2021-08-04 21:31:45.238725 W | etcdserver: read-only range request "key:\"/registry/project.cattle.io/apprevisions/\" range_end:\"/registry/project.cattle.io/apprevisions0\" limit:1 " with result "error:context canceled" took too long (2.812533178s) to execute
WARNING: 2021/08/04 21:31:45 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021/08/04 21:36:39 [INFO] Rancher version v2.5.7 (c824d91cd) is starting
2021/08/04 21:36:39 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Agent:false Features:}
2021/08/04 21:36:39 [INFO] Listening on /tmp/log.sock
2021/08/04 21:36:39 [INFO] Running etcd --data-dir=management-state/etcd --heartbeat-interval=500 --election-timeout=5000
2021-08-04 21:36:39.596044 W | pkg/flags: unrecognized environment variable ETCD_URL_arm64=https://github.com/etcd-io/etcd/releases/download/v3.4.3/etcd-v3.4.3-linux-arm64.tar.gz
2021-08-04 21:36:39.596187 W | pkg/flags: unrecognized environment variable ETCD_URL_amd64=https://github.com/etcd-io/etcd/releases/download/v3.4.3/etcd-v3.4.3-linux-amd64.tar.gz
2021-08-04 21:36:39.596200 W | pkg/flags: unrecognized environment variable ETCD_UNSUPPORTED_ARCH=amd64
2021-08-04 21:36:39.596210 W | pkg/flags: unrecognized environment variable ETCD_URL=ETCD_URL_amd64
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-08-04 21:36:39.596300 I | etcdmain: etcd Version: 3.4.3
2021-08-04 21:36:39.596313 I | etcdmain: Git SHA: 3cf2f69b5
2021-08-04 21:36:39.596320 I | etcdmain: Go Version: go1.12.12
2021-08-04 21:36:39.596327 I | etcdmain: Go OS/Arch: linux/amd64
2021-08-04 21:36:39.596347 I | etcdmain: setting maximum number of CPUs to 5, total number of available CPUs is 5
2021-08-04 21:36:39.596521 N | etcdmain: the server is already initialized as member before, starting as etcd member...
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-08-04 21:36:39.597900 I | embed: name = default
2021-08-04 21:36:39.597932 I | embed: data dir = management-state/etcd
2021-08-04 21:36:39.597944 I | embed: member dir = management-state/etcd/member
2021-08-04 21:36:39.597952 I | embed: heartbeat = 500ms
2021-08-04 21:36:39.597960 I | embed: election = 5000ms
2021-08-04 21:36:39.597980 I | embed: snapshot count = 100000
2021-08-04 21:36:39.598008 I | embed: advertise client URLs = http://localhost:2379
2021-08-04 21:36:39.598022 I | embed: initial advertise peer URLs = http://localhost:2380
2021-08-04 21:36:39.598034 I | embed: initial cluster =
2021-08-04 21:36:39.714962 I | etcdserver: recovered store from snapshot at index 69300705
2021-08-04 21:36:40.464859 I | mvcc: restore compact to 66056439
2021-08-04 21:36:41.202482 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 69330384
raft2021/08/04 21:36:41 INFO: 8e9e05c52164694d switched to configuration voters=(10276657743932975437)
raft2021/08/04 21:36:41 INFO: 8e9e05c52164694d became follower at term 8221
raft2021/08/04 21:36:41 INFO: newRaft 8e9e05c52164694d [peers: [8e9e05c52164694d], term: 8221, commit: 69330384, applied: 69300705, lastindex: 69330384, lastterm: 8221]
2021-08-04 21:36:41.219400 I | etcdserver/api: enabled capabilities for version 3.4
2021-08-04 21:36:41.219450 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32 from store
2021-08-04 21:36:41.219465 I | etcdserver/membership: set the cluster version to 3.4 from store
2021-08-04 21:36:42.320778 I | mvcc: restore compact to 66056439
2021-08-04 21:36:42.543931 W | auth: simple token is not cryptographically signed
2021-08-04 21:36:43.295632 I | etcdserver: starting server... [version: 3.4.3, cluster version: 3.4]
2021-08-04 21:36:43.296561 I | etcdserver: 8e9e05c52164694d as single-node; fast-forwarding 9 ticks (election ticks 10)
2021-08-04 21:36:43.300922 I | embed: listening for peers on 127.0.0.1:2380
raft2021/08/04 21:36:43 INFO: 8e9e05c52164694d is starting a new election at term 8221
raft2021/08/04 21:36:43 INFO: 8e9e05c52164694d became candidate at term 8222
raft2021/08/04 21:36:43 INFO: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 8222
raft2021/08/04 21:36:43 INFO: 8e9e05c52164694d became leader at term 8222
raft2021/08/04 21:36:43 INFO: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 8222
2021-08-04 21:36:45.047986 W | etcdserver: request "ID:7587856250798669313 Method:\"PUT\" Path:\"/0/members/8e9e05c52164694d/attributes\" Val:\"{\\\"name\\\":\\\"default\\\",\\\"clientURLs\\\":[\\\"http://localhost:2379\\\"]}\" " with result "" took too long (1.651023342s) to execute
2021-08-04 21:36:45.657316 W | wal: sync duration of 1.936257209s, expected less than 1s
2021-08-04 21:36:45.753418 W | etcdserver: request "ID:7587856250975549186 Method:\"PUT\" Path:\"/0/members/8e9e05c52164694d/attributes\" Val:\"{\\\"name\\\":\\\"default\\\",\\\"clientURLs\\\":[\\\"http://localhost:2379\\\"]}\" " with result "" took too long (594.613517ms) to execute
2021-08-04 21:36:47.058332 W | etcdserver: request "ID:7587856251154179329 Method:\"PUT\" Path:\"/0/members/8e9e05c52164694d/attributes\" Val:\"{\\\"name\\\":\\\"default\\\",\\\"clientURLs\\\":[\\\"http://localhost:2379\\\"]}\" " with result "" took too long (1.202223899s) to execute
2021-08-04 21:36:48.340501 W | etcdserver: request "ID:7587856251363600897 Method:\"PUT\" Path:\"/0/members/8e9e05c52164694d/attributes\" Val:\"{\\\"name\\\":\\\"default\\\",\\\"clientURLs\\\":[\\\"http://localhost:2379\\\"]}\" " with result "" took too long (1.179680258s) to execute
2021-08-04 21:36:49.643509 W | etcdserver: request "ID:7587856251530666497 Method:\"PUT\" Path:\"/0/members/8e9e05c52164694d/attributes\" Val:\"{\\\"name\\\":\\\"default\\\",\\\"clientURLs\\\":[\\\"http://localhost:2379\\\"]}\" " with result "" took too long (1.200863959s) to execute
2021-08-04 21:36:50.758275 W | etcdserver: request "ID:7587856251678570753 Method:\"PUT\" Path:\"/0/members/8e9e05c52164694d/attributes\" Val:\"{\\\"name\\\":\\\"default\\\",\\\"clientURLs\\\":[\\\"http://localhost:2379\\\"]}\" " with result "" took too long (1.013019645s) to execute
2021-08-04 21:36:54.576964 W | etcdserver: request "ID:7587856251832536065 Method:\"PUT\" Path:\"/0/members/8e9e05c52164694d/attributes\" Val:\"{\\\"name\\\":\\\"default\\\",\\\"clientURLs\\\":[\\\"http://localhost:2379\\\"]}\" " with result "" took too long (3.716152923s) to execute
2021-08-04 21:36:56.681238 W | etcdserver: request "ID:7587856252021512705 Method:\"PUT\" Path:\"/0/members/8e9e05c52164694d/attributes\" Val:\"{\\\"name\\\":\\\"default\\\",\\\"clientURLs\\\":[\\\"http://localhost:2379\\\"]}\" " with result "" took too long (1.99076217s) to execute
2021-08-04 21:36:57.888752 W | etcdserver: request "ID:7587856252338528001 Method:\"PUT\" Path:\"/0/members/8e9e05c52164694d/attributes\" Val:\"{\\\"name\\\":\\\"default\\\",\\\"clientURLs\\\":[\\\"http://localhost:2379\\\"]}\" " with result "" took too long (1.096364154s) to execute
2021-08-04 21:36:58.298311 E | etcdserver: publish error: etcdserver: request timed out, possibly due to previous leader failure
2021-08-04 21:36:58.728741 W | etcdserver: request "ID:7587856252651927810 Method:\"PUT\" Path:\"/0/members/8e9e05c52164694d/attributes\" Val:\"{\\\"name\\\":\\\"default\\\",\\\"clientURLs\\\":[\\\"http://localhost:2379\\\"]}\" " with result "" took too long (729.369046ms) to execute
2021-08-04 21:36:58.813718 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
2021-08-04 21:36:58.814163 I | embed: ready to serve client requests
2021-08-04 21:36:58.816715 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
2021/08/04 21:36:59 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6443/version?timeout=15m0s": dial tcp 127.0.0.1:6443: connect: connection refused
time="2021-08-04T21:36:59.616101748Z" level=info msg="Starting k3s v1.18.8+k3s1 (6b595318)"
time="2021-08-04T21:36:59.616625766Z" level=info msg="Cluster bootstrap already complete"
time="2021-08-04T21:36:59.649816025Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=unknown --authorization-mode=Node,RBAC --basic-auth-file=/var/lib/rancher/k3s/server/cred/passwd --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=http://localhost:2379 --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=k3s --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
Flag --basic-auth-file has been deprecated, Basic authentication mode is deprecated and will be removed in a future release. It is not recommended for production environments.
I0804 21:36:59.651220      34 server.go:645] external host was not specified, using 172.17.0.2
I0804 21:36:59.651813      34 server.go:162] Version: v1.18.8+k3s1
I0804 21:36:59.664999      34 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0804 21:36:59.665039      34 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0804 21:36:59.667219      34 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0804 21:36:59.667255      34 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0804 21:36:59.713189      34 master.go:270] Using reconciler: lease
I0804 21:36:59.756845      34 rest.go:113] the default service ipfamily for this cluster is: IPv4
2021-08-04 21:37:00.265918 W | wal: sync duration of 1.109835094s, expected less than 1s
W0804 21:37:00.346705      34 genericapiserver.go:409] Skipping API batch/v2alpha1 because it has no resources.
W0804 21:37:00.375674      34 genericapiserver.go:409] Skipping API discovery.k8s.io/v1alpha1 because it has no resources.
W0804 21:37:00.409383      34 genericapiserver.go:409] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0804 21:37:00.464441      34 genericapiserver.go:409] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0804 21:37:00.475760      34 genericapiserver.go:409] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0804 21:37:00.520304      34 genericapiserver.go:409] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0804 21:37:00.583428      34 genericapiserver.go:409] Skipping API apps/v1beta2 because it has no resources.
W0804 21:37:00.583516      34 genericapiserver.go:409] Skipping API apps/v1beta1 because it has no resources.
I0804 21:37:00.617239      34 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0804 21:37:00.617307      34 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
2021/08/04 21:37:01 [INFO] Waiting for server to become available: the server is currently unable to handle the request
2021-08-04 21:37:01.500625 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices\" range_end:\"/registry/apiregistration.k8s.io/apiservicet\" count_only:true " with result "range_response_count:0 size:9" took too long (869.373823ms) to execute
2021-08-04 21:37:01.501177 W | etcdserver: read-only range request "key:\"/registry/mutatingwebhookconfigurations\" range_end:\"/registry/mutatingwebhookconfigurationt\" count_only:true " with result "range_response_count:0 size:9" took too long (1.380802453s) to execute
2021-08-04 21:37:01.501599 W | etcdserver: read-only range request "key:\"/registry/mutatingwebhookconfigurations/\" range_end:\"/registry/mutatingwebhookconfigurations0\" limit:10000 " with result "range_response_count:1 size:3815" took too long (1.381684295s) to execute
2021-08-04 21:37:01.501729 W | etcdserver: read-only range request "key:\"/registry/apiextensions.k8s.io/customresourcedefinitions\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitiont\" count_only:true " with result "range_response_count:0 size:9" took too long (1.820642563s) to execute
2021-08-04 21:37:01.501885 W | etcdserver: read-only range request "key:\"/registry/validatingwebhookconfigurations\" range_end:\"/registry/validatingwebhookconfigurationt\" count_only:true " with result "range_response_count:0 size:9" took too long (1.383619588s) to execute
2021-08-04 21:37:01.502303 W | etcdserver: read-only range request "key:\"/registry/ingress\" range_end:\"/registry/ingrest\" count_only:true " with result "range_response_count:0 size:7" took too long (1.498679352s) to execute
2021-08-04 21:37:01.503454 W | etcdserver: read-only range request "key:\"/registry/serviceaccounts/\" range_end:\"/registry/serviceaccounts0\" limit:10000 " with result "range_response_count:56 size:22828" took too long (1.75274863s) to execute
I0804 21:37:01.503643      34 trace.go:116] Trace[1508667968]: "List etcd3" key:/mutatingwebhookconfigurations,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.119421521 +0000 UTC m=+0.934976943) (total time: 1.384056543s):
Trace[1508667968]: [1.384056543s] [1.384056543s] END
2021-08-04 21:37:01.504054 W | etcdserver: read-only range request "key:\"/registry/ingress/\" range_end:\"/registry/ingress0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.500506574s) to execute
2021-08-04 21:37:01.504119 W | etcdserver: read-only range request "key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitions0\" limit:10000 " with result "range_response_count:94 size:371930" took too long (1.823294982s) to execute
2021-08-04 21:37:01.504329 W | etcdserver: read-only range request "key:\"/registry/networkpolicies/\" range_end:\"/registry/networkpolicies0\" limit:10000 " with result "range_response_count:1 size:1443" took too long (1.507732478s) to execute
2021-08-04 21:37:01.504771 W | etcdserver: read-only range request "key:\"/registry/networkpolicies\" range_end:\"/registry/networkpoliciet\" count_only:true " with result "range_response_count:0 size:9" took too long (1.508281794s) to execute
2021-08-04 21:37:01.505111 W | etcdserver: read-only range request "key:\"/registry/ingress/\" range_end:\"/registry/ingress0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.511002267s) to execute
2021-08-04 21:37:01.505372 W | etcdserver: read-only range request "key:\"/registry/ingress\" range_end:\"/registry/ingrest\" count_only:true " with result "range_response_count:0 size:7" took too long (1.511297748s) to execute
2021-08-04 21:37:01.505667 W | etcdserver: read-only range request "key:\"/registry/endpointslices/\" range_end:\"/registry/endpointslices0\" limit:10000 " with result "range_response_count:4 size:3447" took too long (1.513050339s) to execute
I0804 21:37:01.505740      34 trace.go:116] Trace[37364700]: "List etcd3" key:/networkpolicies,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.995218336 +0000 UTC m=+0.810773774) (total time: 1.510488153s):
Trace[37364700]: [1.510488153s] [1.510488153s] END
2021-08-04 21:37:01.505867 W | etcdserver: read-only range request "key:\"/registry/volumeattachments\" range_end:\"/registry/volumeattachmentt\" count_only:true " with result "range_response_count:0 size:7" took too long (1.451010036s) to execute
I0804 21:37:01.506554      34 trace.go:116] Trace[2075078884]: "List etcd3" key:/endpointslices,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.99176149 +0000 UTC m=+0.807316911) (total time: 1.51476776s):
Trace[2075078884]: [1.51476776s] [1.51476776s] END
I0804 21:37:01.507153      34 trace.go:116] Trace[790135877]: "List etcd3" key:/ingress,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.993403742 +0000 UTC m=+0.808959162) (total time: 1.513712826s):
Trace[790135877]: [1.513712826s] [1.513712826s] END
I0804 21:37:01.507731      34 trace.go:116] Trace[1360991085]: "List etcd3" key:/serviceaccounts,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.750209403 +0000 UTC m=+0.565764826) (total time: 1.757477135s):
Trace[1360991085]: [1.757477135s] [1.757477135s] END
2021-08-04 21:37:01.518751 W | etcdserver: read-only range request "key:\"/registry/endpointslices\" range_end:\"/registry/endpointslicet\" count_only:true " with result "range_response_count:0 size:9" took too long (1.513474944s) to execute
2021-08-04 21:37:01.521661 W | etcdserver: read-only range request "key:\"/registry/leases/\" range_end:\"/registry/leases0\" limit:10000 " with result "range_response_count:4 size:2072" took too long (1.51597154s) to execute
2021-08-04 21:37:01.521838 W | etcdserver: read-only range request "key:\"/registry/leases\" range_end:\"/registry/leaset\" count_only:true " with result "range_response_count:0 size:9" took too long (1.516109383s) to execute
2021-08-04 21:37:01.522009 W | etcdserver: read-only range request "key:\"/registry/leases/\" range_end:\"/registry/leases0\" limit:10000 " with result "range_response_count:4 size:2072" took too long (1.516520707s) to execute
2021-08-04 21:37:01.522222 W | etcdserver: read-only range request "key:\"/registry/leases\" range_end:\"/registry/leaset\" count_only:true " with result "range_response_count:0 size:9" took too long (1.516669758s) to execute
2021-08-04 21:37:01.522306 W | etcdserver: read-only range request "key:\"/registry/certificatesigningrequests/\" range_end:\"/registry/certificatesigningrequests0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.52136577s) to execute
2021-08-04 21:37:01.522431 W | etcdserver: read-only range request "key:\"/registry/certificatesigningrequests\" range_end:\"/registry/certificatesigningrequestt\" count_only:true " with result "range_response_count:0 size:7" took too long (1.521614843s) to execute
2021-08-04 21:37:01.522498 W | etcdserver: read-only range request "key:\"/registry/validatingwebhookconfigurations/\" range_end:\"/registry/validatingwebhookconfigurations0\" limit:10000 " with result "range_response_count:1 size:5738" took too long (1.389010379s) to execute
2021-08-04 21:37:01.522645 W | etcdserver: read-only range request "key:\"/registry/cronjobs/\" range_end:\"/registry/cronjobs0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.523855427s) to execute
2021-08-04 21:37:01.522745 W | etcdserver: read-only range request "key:\"/registry/mutatingwebhookconfigurations\" range_end:\"/registry/mutatingwebhookconfigurationt\" count_only:true " with result "range_response_count:0 size:9" took too long (1.39046537s) to execute
2021-08-04 21:37:01.522873 W | etcdserver: read-only range request "key:\"/registry/cronjobs\" range_end:\"/registry/cronjobt\" count_only:true " with result "range_response_count:0 size:7" took too long (1.524071981s) to execute
2021-08-04 21:37:01.522960 W | etcdserver: read-only range request "key:\"/registry/jobs/\" range_end:\"/registry/jobs0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.52689076s) to execute
2021-08-04 21:37:01.523070 W | etcdserver: read-only range request "key:\"/registry/mutatingwebhookconfigurations/\" range_end:\"/registry/mutatingwebhookconfigurations0\" limit:10000 " with result "range_response_count:1 size:3815" took too long (1.390900339s) to execute
2021-08-04 21:37:01.523146 W | etcdserver: read-only range request "key:\"/registry/jobs\" range_end:\"/registry/jobt\" count_only:true " with result "range_response_count:0 size:7" took too long (1.527150496s) to execute
2021-08-04 21:37:01.523191 W | etcdserver: read-only range request "key:\"/registry/horizontalpodautoscalers\" range_end:\"/registry/horizontalpodautoscalert\" count_only:true " with result "range_response_count:0 size:7" took too long (1.531545665s) to execute
2021-08-04 21:37:01.523298 W | etcdserver: read-only range request "key:\"/registry/validatingwebhookconfigurations/\" range_end:\"/registry/validatingwebhookconfigurations0\" limit:10000 " with result "range_response_count:1 size:5738" took too long (1.392095956s) to execute
2021-08-04 21:37:01.523488 W | etcdserver: read-only range request "key:\"/registry/horizontalpodautoscalers/\" range_end:\"/registry/horizontalpodautoscalers0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.532197228s) to execute
2021-08-04 21:37:01.523556 W | etcdserver: read-only range request "key:\"/registry/validatingwebhookconfigurations\" range_end:\"/registry/validatingwebhookconfigurationt\" count_only:true " with result "range_response_count:0 size:9" took too long (1.392350988s) to execute
2021-08-04 21:37:01.523622 W | etcdserver: read-only range request "key:\"/registry/horizontalpodautoscalers\" range_end:\"/registry/horizontalpodautoscalert\" count_only:true " with result "range_response_count:0 size:7" took too long (1.534304575s) to execute
2021-08-04 21:37:01.523722 W | etcdserver: read-only range request "key:\"/registry/controllerrevisions\" range_end:\"/registry/controllerrevisiont\" count_only:true " with result "range_response_count:0 size:7" took too long (1.394632815s) to execute
2021-08-04 21:37:01.523833 W | etcdserver: read-only range request "key:\"/registry/horizontalpodautoscalers/\" range_end:\"/registry/horizontalpodautoscalers0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.535126017s) to execute
2021-08-04 21:37:01.523898 W | etcdserver: read-only range request "key:\"/registry/horizontalpodautoscalers/\" range_end:\"/registry/horizontalpodautoscalers0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.536585618s) to execute
2021-08-04 21:37:01.523989 W | etcdserver: read-only range request "key:\"/registry/controllerrevisions/\" range_end:\"/registry/controllerrevisions0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.394973105s) to execute
2021-08-04 21:37:01.524068 W | etcdserver: read-only range request "key:\"/registry/horizontalpodautoscalers\" range_end:\"/registry/horizontalpodautoscalert\" count_only:true " with result "range_response_count:0 size:7" took too long (1.536786543s) to execute
2021-08-04 21:37:01.524136 W | etcdserver: read-only range request "key:\"/registry/controllers\" range_end:\"/registry/controllert\" count_only:true " with result "range_response_count:0 size:7" took too long (1.750382922s) to execute
2021-08-04 21:37:01.524225 W | etcdserver: read-only range request "key:\"/registry/replicasets/\" range_end:\"/registry/replicasets0\" limit:10000 " with result "range_response_count:7 size:22331" took too long (1.397401743s) to execute
2021-08-04 21:37:01.524473 W | etcdserver: read-only range request "key:\"/registry/controllers/\" range_end:\"/registry/controllers0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.750537251s) to execute
2021-08-04 21:37:01.524543 W | etcdserver: read-only range request "key:\"/registry/replicasets\" range_end:\"/registry/replicasett\" count_only:true " with result "range_response_count:0 size:9" took too long (1.397541238s) to execute
2021-08-04 21:37:01.524601 W | etcdserver: read-only range request "key:\"/registry/daemonsets/\" range_end:\"/registry/daemonsets0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.399774944s) to execute
I0804 21:37:01.525515      34 trace.go:116] Trace[244023432]: "List etcd3" key:/controllers,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.756858494 +0000 UTC m=+0.572413915) (total time: 1.768569763s):
Trace[244023432]: [1.768569763s] [1.768569763s] END
I0804 21:37:01.528314      34 trace.go:116] Trace[1596459209]: "List etcd3" key:/leases,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.989277118 +0000 UTC m=+0.804832544) (total time: 1.539007086s):
Trace[1596459209]: [1.539007086s] [1.539007086s] END
I0804 21:37:01.528848      34 trace.go:116] Trace[38745427]: "List etcd3" key:/leases,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.988276405 +0000 UTC m=+0.803831828) (total time: 1.540536701s):
Trace[38745427]: [1.540536701s] [1.540536701s] END
I0804 21:37:01.526090      34 trace.go:116] Trace[383051757]: "List etcd3" key:/ingress,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.002977521 +0000 UTC m=+0.818532947) (total time: 1.523067236s):
Trace[383051757]: [1.523067236s] [1.523067236s] END
I0804 21:37:01.526588      34 trace.go:116] Trace[1068245273]: "List etcd3" key:/controllerrevisions,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.11208143 +0000 UTC m=+0.927636852) (total time: 1.414480163s):
Trace[1068245273]: [1.414480163s] [1.414480163s] END
I0804 21:37:01.526752      34 trace.go:116] Trace[137224012]: "List etcd3" key:/horizontalpodautoscalers,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.972264788 +0000 UTC m=+0.787820208) (total time: 1.55446229s):
Trace[137224012]: [1.55446229s] [1.55446229s] END
I0804 21:37:01.526908      34 trace.go:116] Trace[1992280103]: "List etcd3" key:/horizontalpodautoscalers,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.974904756 +0000 UTC m=+0.790460179) (total time: 1.55196991s):
Trace[1992280103]: [1.55196991s] [1.55196991s] END
I0804 21:37:01.526897      34 trace.go:116] Trace[1971342135]: "List etcd3" key:/mutatingwebhookconfigurations,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.11602086 +0000 UTC m=+0.931576284) (total time: 1.41079468s):
Trace[1971342135]: [1.41079468s] [1.41079468s] END
I0804 21:37:01.526977      34 trace.go:116] Trace[868672631]: "List etcd3" key:/horizontalpodautoscalers,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.970473978 +0000 UTC m=+0.786029402) (total time: 1.556474239s):
Trace[868672631]: [1.556474239s] [1.556474239s] END
I0804 21:37:01.527247      34 trace.go:116] Trace[366913737]: "List etcd3" key:/jobs,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.979819043 +0000 UTC m=+0.795374477) (total time: 1.547401974s):
Trace[366913737]: [1.547401974s] [1.547401974s] END
I0804 21:37:01.527439      34 trace.go:116] Trace[1803681787]: "List etcd3" key:/cronjobs,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.979786568 +0000 UTC m=+0.795342003) (total time: 1.547627291s):
Trace[1803681787]: [1.547627291s] [1.547627291s] END
I0804 21:37:01.527804      34 trace.go:116] Trace[1978676687]: "List etcd3" key:/certificatesigningrequests,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.984104443 +0000 UTC m=+0.799659871) (total time: 1.543674629s):
Trace[1978676687]: [1.543674629s] [1.543674629s] END
I0804 21:37:01.533159      34 trace.go:116] Trace[976582495]: "List etcd3" key:/replicasets,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.11007297 +0000 UTC m=+0.925628395) (total time: 1.423040553s):
Trace[976582495]: [1.423040553s] [1.423040553s] END
2021-08-04 21:37:01.534077 W | etcdserver: read-only range request "key:\"/registry/services/specs/\" range_end:\"/registry/services/specs0\" limit:10000 " with result "range_response_count:4 size:4725" took too long (1.755686727s) to execute
2021-08-04 21:37:01.534218 W | etcdserver: read-only range request "key:\"/registry/daemonsets\" range_end:\"/registry/daemonsett\" count_only:true " with result "range_response_count:0 size:7" took too long (1.399919325s) to execute
2021-08-04 21:37:01.534324 W | etcdserver: read-only range request "key:\"/registry/statefulsets/\" range_end:\"/registry/statefulsets0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.421722315s) to execute
2021-08-04 21:37:01.534479 W | etcdserver: read-only range request "key:\"/registry/services/specs\" range_end:\"/registry/services/spect\" count_only:true " with result "range_response_count:0 size:9" took too long (1.755968437s) to execute
2021-08-04 21:37:01.534561 W | etcdserver: read-only range request "key:\"/registry/statefulsets\" range_end:\"/registry/statefulsett\" count_only:true " with result "range_response_count:0 size:7" took too long (1.42184349s) to execute
2021-08-04 21:37:01.534625 W | etcdserver: read-only range request "key:\"/registry/serviceaccounts\" range_end:\"/registry/serviceaccountt\" count_only:true " with result "range_response_count:0 size:9" took too long (1.757644953s) to execute
2021-08-04 21:37:01.534710 W | etcdserver: read-only range request "key:\"/registry/deployments/\" range_end:\"/registry/deployments0\" limit:10000 " with result "range_response_count:6 size:20789" took too long (1.423772976s) to execute
2021-08-04 21:37:01.535070 W | etcdserver: read-only range request "key:\"/registry/deployments\" range_end:\"/registry/deploymentt\" count_only:true " with result "range_response_count:0 size:9" took too long (1.423945579s) to execute
2021-08-04 21:37:01.535152 W | etcdserver: read-only range request "key:\"/registry/csidrivers/\" range_end:\"/registry/csidrivers0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.425648981s) to execute
2021-08-04 21:37:01.535232 W | etcdserver: read-only range request "key:\"/registry/csidrivers\" range_end:\"/registry/csidrivert\" count_only:true " with result "range_response_count:0 size:7" took too long (1.425734116s) to execute
2021-08-04 21:37:01.535291 W | etcdserver: read-only range request "key:\"/registry/csinodes/\" range_end:\"/registry/csinodes0\" limit:10000 " with result "range_response_count:1 size:458" took too long (1.427324434s) to execute
2021-08-04 21:37:01.535385 W | etcdserver: read-only range request "key:\"/registry/csinodes\" range_end:\"/registry/csinodet\" count_only:true " with result "range_response_count:0 size:9" took too long (1.427456118s) to execute
2021-08-04 21:37:01.535456 W | etcdserver: read-only range request "key:\"/registry/volumeattachments/\" range_end:\"/registry/volumeattachments0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.429678687s) to execute
2021-08-04 21:37:01.535557 W | etcdserver: read-only range request "key:\"/registry/volumeattachments\" range_end:\"/registry/volumeattachmentt\" count_only:true " with result "range_response_count:0 size:7" took too long (1.429781222s) to execute
2021-08-04 21:37:01.535622 W | etcdserver: read-only range request "key:\"/registry/storageclasses/\" range_end:\"/registry/storageclasses0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.448897469s) to execute
2021-08-04 21:37:01.535714 W | etcdserver: read-only range request "key:\"/registry/storageclasses\" range_end:\"/registry/storageclasset\" count_only:true " with result "range_response_count:0 size:7" took too long (1.449260867s) to execute
2021-08-04 21:37:01.535785 W | etcdserver: read-only range request "key:\"/registry/csidrivers/\" range_end:\"/registry/csidrivers0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.451563709s) to execute
2021-08-04 21:37:01.535860 W | etcdserver: read-only range request "key:\"/registry/csidrivers\" range_end:\"/registry/csidrivert\" count_only:true " with result "range_response_count:0 size:7" took too long (1.451674406s) to execute
2021-08-04 21:37:01.535924 W | etcdserver: read-only range request "key:\"/registry/csinodes\" range_end:\"/registry/csinodet\" count_only:true " with result "range_response_count:0 size:9" took too long (1.452733823s) to execute
2021-08-04 21:37:01.535998 W | etcdserver: read-only range request "key:\"/registry/csinodes/\" range_end:\"/registry/csinodes0\" limit:10000 " with result "range_response_count:1 size:458" took too long (1.453535952s) to execute
2021-08-04 21:37:01.536109 W | etcdserver: read-only range request "key:\"/registry/volumeattachments/\" range_end:\"/registry/volumeattachments0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.45497702s) to execute
2021-08-04 21:37:01.536195 W | etcdserver: read-only range request "key:\"/registry/clusterrolebindings\" range_end:\"/registry/clusterrolebindingt\" count_only:true " with result "range_response_count:0 size:9" took too long (1.477454436s) to execute
2021-08-04 21:37:01.536303 W | etcdserver: read-only range request "key:\"/registry/storageclasses\" range_end:\"/registry/storageclasset\" count_only:true " with result "range_response_count:0 size:7" took too long (1.457024969s) to execute
2021-08-04 21:37:01.536462 W | etcdserver: read-only range request "key:\"/registry/pods/\" range_end:\"/registry/pods0\" limit:10000 " with result "range_response_count:6 size:20082" took too long (1.76189127s) to execute
2021-08-04 21:37:01.536724 W | etcdserver: read-only range request "key:\"/registry/pods\" range_end:\"/registry/podt\" count_only:true " with result "range_response_count:0 size:9" took too long (1.76203194s) to execute
2021-08-04 21:37:01.536871 W | etcdserver: read-only range request "key:\"/registry/ingressclasses\" range_end:\"/registry/ingressclasset\" count_only:true " with result "range_response_count:0 size:7" took too long (1.505813941s) to execute
2021-08-04 21:37:01.537137 W | etcdserver: read-only range request "key:\"/registry/minions\" range_end:\"/registry/miniont\" count_only:true " with result "range_response_count:0 size:9" took too long (1.765011156s) to execute
I0804 21:37:01.537125      34 trace.go:116] Trace[336674949]: "List etcd3" key:/statefulsets,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.085941566 +0000 UTC m=+0.901496987) (total time: 1.451158809s):
Trace[336674949]: [1.451158809s] [1.451158809s] END
I0804 21:37:01.533661      34 trace.go:116] Trace[252585745]: "List etcd3" key:/validatingwebhookconfigurations,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.114278879 +0000 UTC m=+0.929834317) (total time: 1.419345107s):
Trace[252585745]: [1.419345107s] [1.419345107s] END
2021-08-04 21:37:01.537742 W | etcdserver: read-only range request "key:\"/registry/minions/\" range_end:\"/registry/minions0\" limit:10000 " with result "range_response_count:1 size:5788" took too long (1.76519865s) to execute
2021-08-04 21:37:01.537819 W | etcdserver: read-only range request "key:\"/registry/services/endpoints\" range_end:\"/registry/services/endpointt\" count_only:true " with result "range_response_count:0 size:9" took too long (1.767049286s) to execute
I0804 21:37:01.538045      34 trace.go:116] Trace[1109229594]: "List etcd3" key:/csidrivers,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.057578051 +0000 UTC m=+0.873133475) (total time: 1.480418409s):
Trace[1109229594]: [1.480418409s] [1.480418409s] END
2021-08-04 21:37:01.538132 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/\" range_end:\"/registry/services/endpoints0\" limit:10000 " with result "range_response_count:7 size:3719" took too long (1.76732248s) to execute
2021-08-04 21:37:01.538297 W | etcdserver: read-only range request "key:\"/registry/events\" range_end:\"/registry/eventt\" count_only:true " with result "range_response_count:0 size:11" took too long (1.815322205s) to execute
2021-08-04 21:37:01.538546 W | etcdserver: read-only range request "key:\"/registry/namespaces\" range_end:\"/registry/namespacet\" count_only:true " with result "range_response_count:0 size:9" took too long (1.775357199s) to execute
2021-08-04 21:37:01.538735 W | etcdserver: read-only range request "key:\"/registry/namespaces/\" range_end:\"/registry/namespaces0\" limit:10000 " with result "range_response_count:21 size:22755" took too long (1.775839386s) to execute
2021-08-04 21:37:01.538870 W | etcdserver: read-only range request "key:\"/registry/configmaps/\" range_end:\"/registry/configmaps0\" limit:10000 " with result "range_response_count:13 size:31906" took too long (1.77935302s) to execute
2021-08-04 21:37:01.538986 W | etcdserver: read-only range request "key:\"/registry/configmaps\" range_end:\"/registry/configmapt\" count_only:true " with result "range_response_count:0 size:9" took too long (1.779522537s) to execute
2021-08-04 21:37:01.539308 W | etcdserver: read-only range request "key:\"/registry/persistentvolumeclaims/\" range_end:\"/registry/persistentvolumeclaims0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.781367602s) to execute
2021-08-04 21:37:01.539440 W | etcdserver: read-only range request "key:\"/registry/persistentvolumeclaims\" range_end:\"/registry/persistentvolumeclaimt\" count_only:true " with result "range_response_count:0 size:7" took too long (1.78144481s) to execute
2021-08-04 21:37:01.539506 W | etcdserver: read-only range request "key:\"/registry/persistentvolumes/\" range_end:\"/registry/persistentvolumes0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.783313183s) to execute
I0804 21:37:01.533791      34 trace.go:116] Trace[1379360331]: "List etcd3" key:/validatingwebhookconfigurations,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.117437516 +0000 UTC m=+0.932992965) (total time: 1.416330726s):
Trace[1379360331]: [1.416330726s] [1.416330726s] END
2021-08-04 21:37:01.539622 W | etcdserver: read-only range request "key:\"/registry/persistentvolumes\" range_end:\"/registry/persistentvolumet\" count_only:true " with result "range_response_count:0 size:7" took too long (1.783461005s) to execute
I0804 21:37:01.535128      34 trace.go:116] Trace[143615917]: "List etcd3" key:/services/specs,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.752115307 +0000 UTC m=+0.567670730) (total time: 1.782975081s):
Trace[143615917]: [1.782975081s] [1.782975081s] END
2021-08-04 21:37:01.539785 W | etcdserver: read-only range request "key:\"/registry/secrets\" range_end:\"/registry/secrett\" count_only:true " with result "range_response_count:0 size:9" took too long (1.78510394s) to execute
2021-08-04 21:37:01.540092 W | etcdserver: read-only range request "key:\"/registry/secrets/\" range_end:\"/registry/secrets0\" limit:10000 " with result "range_response_count:73 size:452420" took too long (1.787361243s) to execute
I0804 21:37:01.535418      34 trace.go:116] Trace[1078591278]: "List etcd3" key:/daemonsets,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.107775105 +0000 UTC m=+0.923330543) (total time: 1.427599845s):
Trace[1078591278]: [1.427599845s] [1.427599845s] END
I0804 21:37:01.536862      34 trace.go:116] Trace[401062017]: "List etcd3" key:/deployments,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.084305598 +0000 UTC m=+0.899861019) (total time: 1.452528569s):
Trace[401062017]: [1.452528569s] [1.452528569s] END
I0804 21:37:01.539460      34 trace.go:116] Trace[1586426426]: "List etcd3" key:/storageclasses,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.059796104 +0000 UTC m=+0.875351527) (total time: 1.479372231s):
Trace[1586426426]: [1.479372231s] [1.479372231s] END
2021-08-04 21:37:01.541427 W | etcdserver: read-only range request "key:\"/registry/clusterrolebindings\" range_end:\"/registry/clusterrolebindingt\" count_only:true " with result "range_response_count:0 size:9" took too long (1.468995564s) to execute
2021-08-04 21:37:01.541547 W | etcdserver: read-only range request "key:\"/registry/clusterrolebindings/\" range_end:\"/registry/clusterrolebindings0\" limit:10000 " with result "range_response_count:93 size:76427" took too long (1.482302398s) to execute
I0804 21:37:01.539837      34 trace.go:116] Trace[1938092159]: "List etcd3" key:/volumeattachments,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.078917366 +0000 UTC m=+0.894472788) (total time: 1.460732624s):
Trace[1938092159]: [1.460732624s] [1.460732624s] END
I0804 21:37:01.540902      34 trace.go:116] Trace[1280072177]: "List etcd3" key:/csidrivers,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.082673353 +0000 UTC m=+0.898228773) (total time: 1.458201163s):
Trace[1280072177]: [1.458201163s] [1.458201163s] END
I0804 21:37:01.541043      34 trace.go:116] Trace[623207152]: "List etcd3" key:/csinodes,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.081056807 +0000 UTC m=+0.896612279) (total time: 1.459962191s):
Trace[623207152]: [1.459962191s] [1.459962191s] END
I0804 21:37:01.542596      34 trace.go:116] Trace[289326713]: "List etcd3" key:/namespaces,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.735159541 +0000 UTC m=+0.550714961) (total time: 1.807399264s):
Trace[289326713]: [1.807399264s] [1.807399264s] END
I0804 21:37:01.543922      34 trace.go:116] Trace[717976800]: "List etcd3" key:/configmaps,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.732561437 +0000 UTC m=+0.548117009) (total time: 1.811328918s):
Trace[717976800]: [1.811328918s] [1.811328918s] END
2021-08-04 21:37:01.543954 W | etcdserver: read-only range request "key:\"/registry/storageclasses/\" range_end:\"/registry/storageclasses0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.4620789s) to execute
2021-08-04 21:37:01.544034 W | etcdserver: read-only range request "key:\"/registry/priorityclasses/\" range_end:\"/registry/priorityclasses0\" limit:10000 " with result "range_response_count:2 size:798" took too long (1.463684631s) to execute
I0804 21:37:01.543040      34 trace.go:116] Trace[776176670]: "List etcd3" key:/persistentvolumes,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.728856108 +0000 UTC m=+0.544411557) (total time: 1.814154525s):
Trace[776176670]: [1.814154525s] [1.814154525s] END
2021-08-04 21:37:01.544226 W | etcdserver: read-only range request "key:\"/registry/priorityclasses\" range_end:\"/registry/priorityclasset\" count_only:true " with result "range_response_count:0 size:9" took too long (1.463838675s) to execute
2021-08-04 21:37:01.544282 W | etcdserver: read-only range request "key:\"/registry/priorityclasses\" range_end:\"/registry/priorityclasset\" count_only:true " with result "range_response_count:0 size:9" took too long (1.465264581s) to execute
2021-08-04 21:37:01.544387 W | etcdserver: read-only range request "key:\"/registry/priorityclasses/\" range_end:\"/registry/priorityclasses0\" limit:10000 " with result "range_response_count:2 size:798" took too long (1.465479883s) to execute
2021-08-04 21:37:01.544489 W | etcdserver: read-only range request "key:\"/registry/clusterroles/\" range_end:\"/registry/clusterroles0\" limit:10000 " with result "range_response_count:110 size:98467" took too long (1.484291552s) to execute
2021-08-04 21:37:01.544810 W | etcdserver: read-only range request "key:\"/registry/clusterroles\" range_end:\"/registry/clusterrolet\" count_only:true " with result "range_response_count:0 size:9" took too long (1.48456117s) to execute
2021-08-04 21:37:01.545069 W | etcdserver: read-only range request "key:\"/registry/clusterrolebindings/\" range_end:\"/registry/clusterrolebindings0\" limit:10000 " with result "range_response_count:93 size:76427" took too long (1.470377705s) to execute
I0804 21:37:01.545345      34 trace.go:116] Trace[1256485917]: "List etcd3" key:/storageclasses,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.052775324 +0000 UTC m=+0.868330748) (total time: 1.492543297s):
Trace[1256485917]: [1.492543297s] [1.492543297s] END
I0804 21:37:01.545438      34 trace.go:116] Trace[742223568]: "List etcd3" key:/priorityclasses,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.051061683 +0000 UTC m=+0.866617104) (total time: 1.494350812s):
Trace[742223568]: [1.494350812s] [1.494350812s] END
2021-08-04 21:37:01.546082 W | etcdserver: read-only range request "key:\"/registry/resourcequotas\" range_end:\"/registry/resourcequotat\" count_only:true " with result "range_response_count:0 size:7" took too long (1.790391258s) to execute
I0804 21:37:01.546114      34 trace.go:116] Trace[1981356464]: "List etcd3" key:/priorityclasses,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.049596272 +0000 UTC m=+0.865151695) (total time: 1.49649254s):
Trace[1981356464]: [1.49649254s] [1.49649254s] END
2021-08-04 21:37:01.546357 W | etcdserver: read-only range request "key:\"/registry/rolebindings/\" range_end:\"/registry/rolebindings0\" limit:10000 " with result "range_response_count:97 size:78255" took too long (1.486494127s) to execute
2021-08-04 21:37:01.546956 W | etcdserver: read-only range request "key:\"/registry/resourcequotas/\" range_end:\"/registry/resourcequotas0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.790842262s) to execute
2021-08-04 21:37:01.547157 W | etcdserver: read-only range request "key:\"/registry/limitranges\" range_end:\"/registry/limitranget\" count_only:true " with result "range_response_count:0 size:7" took too long (1.792001051s) to execute
2021-08-04 21:37:01.547370 W | etcdserver: read-only range request "key:\"/registry/rolebindings\" range_end:\"/registry/rolebindingt\" count_only:true " with result "range_response_count:0 size:9" took too long (1.486736364s) to execute
2021-08-04 21:37:01.547503 W | etcdserver: read-only range request "key:\"/registry/limitranges/\" range_end:\"/registry/limitranges0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.792457558s) to execute
2021-08-04 21:37:01.547723 W | etcdserver: read-only range request "key:\"/registry/roles\" range_end:\"/registry/rolet\" count_only:true " with result "range_response_count:0 size:9" took too long (1.488124122s) to execute
2021-08-04 21:37:01.547934 W | etcdserver: read-only range request "key:\"/registry/roles/\" range_end:\"/registry/roles0\" limit:10000 " with result "range_response_count:46 size:44611" took too long (1.489235928s) to execute
2021-08-04 21:37:01.548197 W | etcdserver: read-only range request "key:\"/registry/podsecuritypolicy/\" range_end:\"/registry/podsecuritypolicy0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.500335206s) to execute
2021-08-04 21:37:01.548341 W | etcdserver: read-only range request "key:\"/registry/podsecuritypolicy\" range_end:\"/registry/podsecuritypolicz\" count_only:true " with result "range_response_count:0 size:7" took too long (1.501055209s) to execute
I0804 21:37:01.548366      34 trace.go:116] Trace[1514001911]: "List etcd3" key:/minions,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.745569714 +0000 UTC m=+0.561125137) (total time: 1.802763314s):
Trace[1514001911]: [1.802763314s] [1.802763314s] END
2021-08-04 21:37:01.548426 W | etcdserver: read-only range request "key:\"/registry/poddisruptionbudgets\" range_end:\"/registry/poddisruptionbudgett\" count_only:true " with result "range_response_count:0 size:7" took too long (1.504360157s) to execute
2021-08-04 21:37:01.548489 W | etcdserver: read-only range request "key:\"/registry/poddisruptionbudgets/\" range_end:\"/registry/poddisruptionbudgets0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.504455554s) to execute
2021-08-04 21:37:01.548588 W | etcdserver: read-only range request "key:\"/registry/runtimeclasses\" range_end:\"/registry/runtimeclasset\" count_only:true " with result "range_response_count:0 size:7" took too long (1.510736004s) to execute
2021-08-04 21:37:01.548695 W | etcdserver: read-only range request "key:\"/registry/runtimeclasses/\" range_end:\"/registry/runtimeclasses0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.511137374s) to execute
2021-08-04 21:37:01.548901 W | etcdserver: read-only range request "key:\"/registry/clusterroles/\" range_end:\"/registry/clusterroles0\" limit:10000 " with result "range_response_count:110 size:98467" took too long (1.475020735s) to execute
I0804 21:37:01.549072      34 trace.go:116] Trace[190552363]: "List etcd3" key:/poddisruptionbudgets,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.012937475 +0000 UTC m=+0.828492908) (total time: 1.536108917s):
Trace[190552363]: [1.536108917s] [1.536108917s] END
2021-08-04 21:37:01.549635 W | etcdserver: read-only range request "key:\"/registry/clusterroles\" range_end:\"/registry/clusterrolet\" count_only:true " with result "range_response_count:0 size:9" took too long (1.475268707s) to execute
I0804 21:37:01.543241      34 trace.go:116] Trace[2103710551]: "List etcd3" key:/persistentvolumeclaims,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.730672458 +0000 UTC m=+0.546227879) (total time: 1.812516407s):
Trace[2103710551]: [1.812516407s] [1.812516407s] END
2021-08-04 21:37:01.549871 W | etcdserver: read-only range request "key:\"/registry/rolebindings\" range_end:\"/registry/rolebindingt\" count_only:true " with result "range_response_count:0 size:9" took too long (1.476558544s) to execute
2021-08-04 21:37:01.550157 W | etcdserver: read-only range request "key:\"/registry/rolebindings/\" range_end:\"/registry/rolebindings0\" limit:10000 " with result "range_response_count:97 size:78255" took too long (1.477265355s) to execute
I0804 21:37:01.543315      34 trace.go:116] Trace[94155535]: "List etcd3" key:/services/endpoints,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.743722712 +0000 UTC m=+0.559278137) (total time: 1.799567558s):
Trace[94155535]: [1.799567558s] [1.799567558s] END
2021-08-04 21:37:01.550433 W | etcdserver: read-only range request "key:\"/registry/roles/\" range_end:\"/registry/roles0\" limit:10000 " with result "range_response_count:46 size:44611" took too long (1.478984302s) to execute
I0804 21:37:01.543496      34 trace.go:116] Trace[1166809543]: "List etcd3" key:/pods,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.748399094 +0000 UTC m=+0.563954525) (total time: 1.795070131s):
Trace[1166809543]: [1.795070131s] [1.795070131s] END
I0804 21:37:01.543545      34 trace.go:116] Trace[628039705]: "List etcd3" key:/csinodes,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.055745666 +0000 UTC m=+0.871301147) (total time: 1.487764103s):
Trace[628039705]: [1.487764103s] [1.487764103s] END
2021-08-04 21:37:01.551027 W | etcdserver: read-only range request "key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitions0\" limit:10000 " with result "range_response_count:94 size:371930" took too long (1.839797146s) to execute
I0804 21:37:01.551080      34 trace.go:116] Trace[1887982411]: "List etcd3" key:/roles,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.027693998 +0000 UTC m=+0.843249479) (total time: 1.523360228s):
Trace[1887982411]: [1.523360228s] [1.523360228s] END
I0804 21:37:01.551544      34 trace.go:116] Trace[1035136480]: "List etcd3" key:/podsecuritypolicy,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.015477859 +0000 UTC m=+0.831033287) (total time: 1.536042316s):
Trace[1035136480]: [1.536042316s] [1.536042316s] END
I0804 21:37:01.552052      34 trace.go:116] Trace[868611774]: "List etcd3" key:/clusterrolebindings,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.045202896 +0000 UTC m=+0.860758311) (total time: 1.50681284s):
Trace[868611774]: [1.50681284s] [1.50681284s] END
2021-08-04 21:37:01.552211 W | etcdserver: read-only range request "key:\"/registry/roles\" range_end:\"/registry/rolet\" count_only:true " with result "range_response_count:0 size:9" took too long (1.479320673s) to execute
2021-08-04 21:37:01.552356 W | etcdserver: read-only range request "key:\"/registry/podtemplates/\" range_end:\"/registry/podtemplates0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.799725614s) to execute
2021-08-04 21:37:01.552495 W | etcdserver: read-only range request "key:\"/registry/podtemplates\" range_end:\"/registry/podtemplatet\" count_only:true " with result "range_response_count:0 size:7" took too long (1.799855286s) to execute
2021-08-04 21:37:01.552656 W | etcdserver: read-only range request "key:\"/registry/apiextensions.k8s.io/customresourcedefinitions\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitiont\" count_only:true " with result "range_response_count:0 size:9" took too long (1.839964396s) to execute
I0804 21:37:01.552906      34 trace.go:116] Trace[1631490575]: "List etcd3" key:/resourcequotas,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.725668646 +0000 UTC m=+0.541224068) (total time: 1.827214694s):
Trace[1631490575]: [1.827214694s] [1.827214694s] END
2021-08-04 21:37:01.552925 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/\" range_end:\"/registry/apiregistration.k8s.io/apiservices0\" limit:10000 " with result "range_response_count:40 size:24960" took too long (897.911755ms) to execute
2021-08-04 21:37:01.553069 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/\" range_end:\"/registry/apiregistration.k8s.io/apiservices0\" limit:10000 " with result "range_response_count:40 size:24960" took too long (890.610629ms) to execute
2021-08-04 21:37:01.553201 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices\" range_end:\"/registry/apiregistration.k8s.io/apiservicet\" count_only:true " with result "range_response_count:0 size:9" took too long (898.393798ms) to execute
2021-08-04 21:37:01.553393 W | etcdserver: read-only range request "key:\"/registry/ingressclasses/\" range_end:\"/registry/ingressclasses0\" limit:10000 " with result "range_response_count:0 size:7" took too long (1.519478839s) to execute
2021-08-04 21:37:01.553515 W | etcdserver: read-only range request "key:\"/registry/events\" range_end:\"/registry/eventt\" count_only:true " with result "range_response_count:0 size:11" took too long (1.412586905s) to execute
I0804 21:37:01.552148      34 trace.go:116] Trace[856911571]: "List etcd3" key:/clusterrolebindings,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.032133822 +0000 UTC m=+0.847689242) (total time: 1.519988484s):
Trace[856911571]: [1.519988484s] [1.519988484s] END
I0804 21:37:01.552556      34 trace.go:116] Trace[1111226085]: "List etcd3" key:/limitranges,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.724187179 +0000 UTC m=+0.539742601) (total time: 1.828338413s):
Trace[1111226085]: [1.828338413s] [1.828338413s] END
I0804 21:37:01.556246      34 trace.go:116] Trace[215206426]: "List etcd3" key:/clusterroles,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.031317848 +0000 UTC m=+0.846873323) (total time: 1.524895166s):
Trace[215206426]: [1.524895166s] [1.524895166s] END
I0804 21:37:01.557263      34 trace.go:116] Trace[1146103131]: "List etcd3" key:/podtemplates,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.720681573 +0000 UTC m=+0.536237004) (total time: 1.836555657s):
Trace[1146103131]: [1.836555657s] [1.836555657s] END
I0804 21:37:01.556542      34 trace.go:116] Trace[2035456595]: "List etcd3" key:/runtimeclasses,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.006509811 +0000 UTC m=+0.822065240) (total time: 1.550006964s):
Trace[2035456595]: [1.550006964s] [1.550006964s] END
I0804 21:37:01.559447      34 trace.go:116] Trace[1619025759]: "List etcd3" key:/apiregistration.k8s.io/apiservices,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.622705016 +0000 UTC m=+1.438260438) (total time: 936.716549ms):
Trace[1619025759]: [936.716549ms] [936.716549ms] END
I0804 21:37:01.559938      34 trace.go:116] Trace[620953830]: "List etcd3" key:/ingressclasses,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.00463104 +0000 UTC m=+0.820186461) (total time: 1.555266654s):
Trace[620953830]: [1.555266654s] [1.555266654s] END
I0804 21:37:01.561180      34 trace.go:116] Trace[199952425]: "List etcd3" key:/apiregistration.k8s.io/apiservices,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.630462353 +0000 UTC m=+1.446017790) (total time: 930.692254ms):
Trace[199952425]: [930.692254ms] [930.692254ms] END
I0804 21:37:01.561575      34 trace.go:116] Trace[1712014354]: "List etcd3" key:/rolebindings,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.029726219 +0000 UTC m=+0.845281643) (total time: 1.531805087s):
Trace[1712014354]: [1.531805087s] [1.531805087s] END
I0804 21:37:01.563116      34 trace.go:116] Trace[64389309]: "List etcd3" key:/roles,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.04126115 +0000 UTC m=+0.856816588) (total time: 1.521826678s):
Trace[64389309]: [1.521826678s] [1.521826678s] END
I0804 21:37:01.564701      34 trace.go:116] Trace[1874505309]: "List etcd3" key:/rolebindings,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.042636618 +0000 UTC m=+0.858192038) (total time: 1.522037858s):
Trace[1874505309]: [1.522037858s] [1.522037858s] END
I0804 21:37:01.564959      34 trace.go:116] Trace[490321594]: "List etcd3" key:/clusterroles,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.043900102 +0000 UTC m=+0.859455523) (total time: 1.521030697s):
Trace[490321594]: [1.521030697s] [1.521030697s] END
I0804 21:37:01.566570      34 trace.go:116] Trace[527721883]: "List etcd3" key:/volumeattachments,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:00.054371945 +0000 UTC m=+0.869927370) (total time: 1.512173089s):
Trace[527721883]: [1.512173089s] [1.512173089s] END
I0804 21:37:01.567772      34 trace.go:116] Trace[966016577]: "List etcd3" key:/secrets,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.727202852 +0000 UTC m=+0.542758290) (total time: 1.84054181s):
Trace[966016577]: [1.84054181s] [1.84054181s] END
I0804 21:37:01.599463      34 trace.go:116] Trace[311312494]: "List etcd3" key:/apiextensions.k8s.io/customresourcedefinitions,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.678664466 +0000 UTC m=+0.494219915) (total time: 1.92073994s):
Trace[311312494]: [1.92073994s] [1.92073994s] END
I0804 21:37:01.638460      34 trace.go:116] Trace[525958317]: "List etcd3" key:/apiextensions.k8s.io/customresourcedefinitions,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:36:59.680612959 +0000 UTC m=+0.496168387) (total time: 1.957784578s):
Trace[525958317]: [1.957784578s] [1.957784578s] END
2021/08/04 21:37:03 [INFO] Waiting for server to become available: the server is currently unable to handle the request
I0804 21:37:04.654918      34 dynamic_cafile_content.go:167] Starting request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt
I0804 21:37:04.655183      34 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt
I0804 21:37:04.655649      34 dynamic_serving_content.go:130] Starting serving-cert::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key
I0804 21:37:04.656488      34 secure_serving.go:178] Serving securely on 127.0.0.1:6444
I0804 21:37:04.656567      34 tlsconfig.go:240] Starting DynamicServingCertificateController
I0804 21:37:04.656693      34 apiservice_controller.go:94] Starting APIServiceRegistrationController
I0804 21:37:04.656751      34 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0804 21:37:04.656808      34 autoregister_controller.go:141] Starting autoregister controller
I0804 21:37:04.656865      34 cache.go:32] Waiting for caches to sync for autoregister controller
I0804 21:37:04.658819      34 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I0804 21:37:04.658872      34 shared_informer.go:223] Waiting for caches to sync for cluster_authentication_trust_controller
I0804 21:37:04.658933      34 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt
I0804 21:37:04.659043      34 dynamic_cafile_content.go:167] Starting request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt
I0804 21:37:04.659275      34 crd_finalizer.go:266] Starting CRDFinalizer
I0804 21:37:04.660031      34 crdregistration_controller.go:111] Starting crd-autoregister controller
I0804 21:37:04.660069      34 shared_informer.go:223] Waiting for caches to sync for crd-autoregister
I0804 21:37:04.656823      34 controller.go:81] Starting OpenAPI AggregationController
I0804 21:37:04.660208      34 controller.go:86] Starting OpenAPI controller
I0804 21:37:04.660318      34 available_controller.go:387] Starting AvailableConditionController
I0804 21:37:04.660342      34 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0804 21:37:04.660359      34 naming_controller.go:291] Starting NamingConditionController
I0804 21:37:04.660421      34 establishing_controller.go:76] Starting EstablishingController
I0804 21:37:04.660502      34 nonstructuralschema_controller.go:186] Starting NonStructuralSchemaConditionController
I0804 21:37:04.660601      34 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I0804 21:37:04.660324      34 customresource_discovery_controller.go:209] Starting DiscoveryController
I0804 21:37:04.757224      34 cache.go:39] Caches are synced for autoregister controller
I0804 21:37:04.757224      34 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0804 21:37:04.760074      34 shared_informer.go:230] Caches are synced for cluster_authentication_trust_controller
I0804 21:37:04.761604      34 shared_informer.go:230] Caches are synced for crd-autoregister
I0804 21:37:04.762538      34 cache.go:39] Caches are synced for AvailableConditionController controller
2021/08/04 21:37:05 [INFO] Waiting for server to become available: the server is currently unable to handle the request
I0804 21:37:05.654723      34 controller.go:130] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
I0804 21:37:05.655192      34 controller.go:130] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
2021/08/04 21:37:07 [INFO] Waiting for server to become available: the server is currently unable to handle the request
2021-08-04 21:37:07.360154 W | wal: sync duration of 2.62586461s, expected less than 1s
2021-08-04 21:37:07.688769 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "error:context deadline exceeded" took too long (2.000140039s) to execute
2021-08-04 21:37:09.182803 W | etcdserver: request "header:<ID:7587856269140755218 > txn:<compare:<target:MOD key:\"/registry/masterleases/172.17.0.2\" mod_revision:66064413 > success:<request_put:<key:\"/registry/masterleases/172.17.0.2\" value_size:65 lease:7587856269140755216 >> failure:<request_range:<key:\"/registry/masterleases/172.17.0.2\" > >>" with result "size:20" took too long (1.820943321s) to execute
I0804 21:37:09.184358      34 trace.go:116] Trace[876476100]: "GuaranteedUpdate etcd3" type:*v1.Endpoints (started: 2021-08-04 21:37:04.732450836 +0000 UTC m=+5.548006267) (total time: 4.451819879s):
Trace[876476100]: [4.451769656s] [4.449031895s] Transaction committed
2021-08-04 21:37:09.184537 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1.k3s.cattle.io\" " with result "range_response_count:1 size:620" took too long (4.423758888s) to execute
2021/08/04 21:37:09 [INFO] Waiting for server to become available: the server is currently unable to handle the request
2021-08-04 21:37:10.688838 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "error:context deadline exceeded" took too long (1.999846876s) to execute
2021/08/04 21:37:11 [INFO] Waiting for server to become available: the server is currently unable to handle the request
2021/08/04 21:37:13 [INFO] Waiting for server to become available: the server is currently unable to handle the request
2021-08-04 21:37:13.467840 W | wal: sync duration of 6.106915846s, expected less than 1s
2021-08-04 21:37:14.668888 W | etcdserver: read-only range request "key:\"/registry/priorityclasses/system-node-critical\" " with result "range_response_count:1 size:386" took too long (9.006065663s) to execute
2021-08-04 21:37:14.669138 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v3.project.cattle.io\" " with result "range_response_count:1 size:632" took too long (9.908336889s) to execute
2021-08-04 21:37:14.669349 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/default/kubernetes\" " with result "range_response_count:1 size:277" took too long (5.483400128s) to execute
2021-08-04 21:37:14.669615 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1.gitjob.cattle.io\" " with result "range_response_count:1 size:629" took too long (9.907593346s) to execute
2021-08-04 21:37:14.669701 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1.k3s.cattle.io\" " with result "range_response_count:1 size:620" took too long (5.478271982s) to execute
2021-08-04 21:37:14.669767 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (2.981464521s) to execute
2021-08-04 21:37:14.669848 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1.helm.cattle.io\" " with result "range_response_count:1 size:623" took too long (9.907844774s) to execute
2021-08-04 21:37:14.669976 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/v1.rancher.cattle.io\" " with result "range_response_count:1 size:632" took too long (9.907939173s) to execute
I0804 21:37:14.670435      34 trace.go:116] Trace[550826673]: "Get" url:/apis/scheduling.k8s.io/v1/priorityclasses/system-node-critical,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:05.662420508 +0000 UTC m=+6.477975928) (total time: 9.007890379s):
Trace[550826673]: [9.007646378s] [9.007636698s] About to write a response
I0804 21:37:14.670551      34 trace.go:116] Trace[1556014158]: "Get" url:/api/v1/namespaces/default/endpoints/kubernetes,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:09.185508312 +0000 UTC m=+10.001063724) (total time: 5.485006105s):
Trace[1556014158]: [5.484940151s] [5.484922255s] About to write a response
2021-08-04 21:37:14.671321 W | etcdserver: read-only range request "key:\"/registry/clusterroles/\" range_end:\"/registry/clusterroles0\" " with result "range_response_count:110 size:98467" took too long (9.009841129s) to execute
I0804 21:37:14.673269      34 storage_scheduling.go:143] all system priority classes are created successfully or already exist.
I0804 21:37:14.675823      34 trace.go:116] Trace[876028663]: "List etcd3" key:/clusterroles,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:05.660938739 +0000 UTC m=+6.476494158) (total time: 9.014852897s):
Trace[876028663]: [9.014852897s] [9.014852897s] END
I0804 21:37:14.676543      34 trace.go:116] Trace[814269609]: "Delete" url:/apis/apiregistration.k8s.io/v1/apiservices/v1.rancher.cattle.io,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:04.760727337 +0000 UTC m=+5.576282777) (total time: 9.915781834s):
Trace[814269609]: [9.915718786s] [9.915257363s] Object deleted from database
I0804 21:37:14.678003      34 trace.go:116] Trace[2046700275]: "List" url:/apis/rbac.authorization.k8s.io/v1/clusterroles,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:05.660909554 +0000 UTC m=+6.476464989) (total time: 9.017061687s):
Trace[2046700275]: [9.014954961s] [9.014938695s] Listing from storage done
2021/08/04 21:37:15 [INFO] Waiting for server to become available: the server is currently unable to handle the request
2021-08-04 21:37:16.157804 W | etcdserver: request "header:<ID:7587856269140755228 > txn:<compare:<target:MOD key:\"/registry/apiregistration.k8s.io/apiservices/v3.project.cattle.io\" mod_revision:66064395 > success:<request_delete_range:<key:\"/registry/apiregistration.k8s.io/apiservices/v3.project.cattle.io\" > > failure:<request_range:<key:\"/registry/apiregistration.k8s.io/apiservices/v3.project.cattle.io\" > >>" with result "size:22" took too long (1.2998419s) to execute
2021-08-04 21:37:16.159092 W | etcdserver: read-only range request "key:\"/registry/masterleases/172.17.0.2\" " with result "range_response_count:1 size:135" took too long (1.482071608s) to execute
I0804 21:37:16.159906      34 trace.go:116] Trace[1807121803]: "Create" url:/apis/apiregistration.k8s.io/v1/apiservices,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:14.680601157 +0000 UTC m=+15.496156585) (total time: 1.479244717s):
Trace[1807121803]: [1.479144482s] [1.479010004s] Object stored in database
I0804 21:37:16.160046      34 trace.go:116] Trace[1071216537]: "Delete" url:/apis/apiregistration.k8s.io/v1/apiservices/v1.k3s.cattle.io,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:04.759856893 +0000 UTC m=+5.575412334) (total time: 11.400140277s):
Trace[1071216537]: [11.400091782s] [11.399850236s] Object deleted from database
I0804 21:37:16.160087      34 trace.go:116] Trace[1387436049]: "Delete" url:/apis/apiregistration.k8s.io/v1/apiservices/v3.project.cattle.io,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:04.759284727 +0000 UTC m=+5.574840247) (total time: 11.400766234s):
Trace[1387436049]: [11.400689562s] [11.400426679s] Object deleted from database
I0804 21:37:16.160276      34 trace.go:116] Trace[756049688]: "Delete" url:/apis/apiregistration.k8s.io/v1/apiservices/v1.helm.cattle.io,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:04.760269124 +0000 UTC m=+5.575824545) (total time: 11.399977601s):
Trace[756049688]: [11.399946556s] [11.39872043s] Object deleted from database
I0804 21:37:16.160385      34 trace.go:116] Trace[983621691]: "Delete" url:/apis/apiregistration.k8s.io/v1/apiservices/v1.gitjob.cattle.io,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:04.759162251 +0000 UTC m=+5.574717663) (total time: 11.401192821s):
Trace[983621691]: [11.401149978s] [11.399470703s] Object deleted from database
2021-08-04 21:37:16.537530 W | wal: sync duration of 1.679626408s, expected less than 1s
2021/08/04 21:37:17 [INFO] Waiting for server to become available: the server is currently unable to handle the request
2021/08/04 21:37:19 [INFO] Waiting for server to become available: the server is currently unable to handle the request
2021-08-04 21:37:19.500200 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (4.812174892s) to execute
2021-08-04 21:37:19.500284 W | etcdserver: request "header:<ID:7587856269140755234 > txn:<compare:<target:MOD key:\"/registry/masterleases/172.17.0.2\" mod_revision:66064428 > success:<request_delete_range:<key:\"/registry/masterleases/172.17.0.2\" > > failure:<request_range:<key:\"/registry/masterleases/172.17.0.2\" > >>" with result "size:22" took too long (2.96177671s) to execute
I0804 21:37:19.501795      34 trace.go:116] Trace[594352561]: "Create" url:/apis/apiregistration.k8s.io/v1/apiservices,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:16.161495715 +0000 UTC m=+16.977051137) (total time: 3.340214886s):
Trace[594352561]: [3.34004087s] [3.339938729s] Object stored in database
I0804 21:37:19.502052      34 trace.go:116] Trace[515151609]: "Create" url:/apis/apiregistration.k8s.io/v1/apiservices,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:16.16268944 +0000 UTC m=+16.978244875) (total time: 3.339311007s):
Trace[515151609]: [3.339211642s] [3.339141396s] Object stored in database
I0804 21:37:19.502500      34 trace.go:116] Trace[525816533]: "Create" url:/apis/apiregistration.k8s.io/v1/apiservices,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:16.163338035 +0000 UTC m=+16.978893455) (total time: 3.339120547s):
Trace[525816533]: [3.339072758s] [3.33882124s] Object stored in database
2021-08-04 21:37:19.502097 W | etcdserver: read-only range request "key:\"/registry/clusterrolebindings/\" range_end:\"/registry/clusterrolebindings0\" " with result "range_response_count:93 size:76427" took too long (4.819542244s) to execute
I0804 21:37:19.502648      34 trace.go:116] Trace[411018770]: "Create" url:/apis/apiregistration.k8s.io/v1/apiservices,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:16.161910566 +0000 UTC m=+16.977466001) (total time: 3.340707417s):
Trace[411018770]: [3.34064877s] [3.340381945s] Object stored in database
I0804 21:37:19.505271      34 trace.go:116] Trace[1116336689]: "List etcd3" key:/clusterrolebindings,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:14.681929076 +0000 UTC m=+15.497484497) (total time: 4.823315143s):
Trace[1116336689]: [4.823315143s] [4.823315143s] END
I0804 21:37:19.507151      34 trace.go:116] Trace[1650914652]: "List" url:/apis/rbac.authorization.k8s.io/v1/clusterrolebindings,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:14.681908853 +0000 UTC m=+15.497464276) (total time: 4.825210905s):
Trace[1650914652]: [4.823403895s] [4.823394056s] Listing from storage done
2021-08-04 21:37:19.688251 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "error:context deadline exceeded" took too long (2.000020101s) to execute
2021/08/04 21:37:21 [INFO] Waiting for server to become available: the server is currently unable to handle the request
2021-08-04 21:37:21.250682 W | wal: sync duration of 4.712136779s, expected less than 1s
2021-08-04 21:37:21.286047 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (596.989153ms) to execute
2021-08-04 21:37:21.286216 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/default/kubernetes\" " with result "range_response_count:1 size:277" took too long (1.782885688s) to execute
2021-08-04 21:37:21.286254 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:aggregate-to-admin\" " with result "range_response_count:1 size:831" took too long (1.776263069s) to execute
I0804 21:37:21.287224      34 trace.go:116] Trace[2106552881]: "Get" url:/api/v1/namespaces/default/endpoints/kubernetes,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:19.502277857 +0000 UTC m=+20.317833269) (total time: 1.784861954s):
Trace[2106552881]: [1.784732787s] [1.784720903s] About to write a response
I0804 21:37:21.287414      34 trace.go:116] Trace[639215876]: "Get" url:/apis/rbac.authorization.k8s.io/v1/clusterroles/system:aggregate-to-admin,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:19.50943629 +0000 UTC m=+20.324991702) (total time: 1.777909971s):
Trace[639215876]: [1.777780207s] [1.777771877s] About to write a response
E0804 21:37:21.288642      34 controller.go:156] Unable to remove old endpoints from kubernetes service: no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service
2021/08/04 21:37:23 [INFO] Waiting for server to become available: the server is currently unable to handle the request
2021-08-04 21:37:24.036127 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (2.631871003s) to execute
2021-08-04 21:37:24.036288 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (531.605946ms) to execute
2021-08-04 21:37:24.036383 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (2.347622973s) to execute
2021-08-04 21:37:24.036426 W | etcdserver: read-only range request "key:\"/registry/clusterroles/system:certificates.k8s.io:certificatesigningrequests:nodeclient\" " with result "range_response_count:1 size:697" took too long (2.684047709s) to execute
I0804 21:37:24.037227      34 trace.go:116] Trace[2089915500]: "Get" url:/apis/rbac.authorization.k8s.io/v1/clusterroles/system:certificates.k8s.io:certificatesigningrequests:nodeclient,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:21.351771887 +0000 UTC m=+22.167327307) (total time: 2.685392047s):
Trace[2089915500]: [2.685283669s] [2.685274549s] About to write a response
2021/08/04 21:37:25 [INFO] Waiting for server to become available: the server is currently unable to handle the request
2021-08-04 21:37:26.173865 W | wal: sync duration of 1.010663311s, expected less than 1s
2021-08-04 21:37:26.577447 W | etcdserver: request "header:<ID:7587856269140755382 > txn:<compare:<target:MOD key:\"/registry/masterleases/172.17.0.2\" mod_revision:0 > success:<request_put:<key:\"/registry/masterleases/172.17.0.2\" value_size:65 lease:7587856269140755380 >> failure:<request_range:<key:\"/registry/masterleases/172.17.0.2\" > >>" with result "size:20" took too long (1.414046943s) to execute
2021-08-04 21:37:26.577802 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (1.889151848s) to execute
I0804 21:37:26.578616      34 trace.go:116] Trace[1515445373]: "GuaranteedUpdate etcd3" type:*v1.Endpoints (started: 2021-08-04 21:37:24.207813959 +0000 UTC m=+25.023369383) (total time: 2.37073092s):
Trace[1515445373]: [2.370674666s] [2.369296182s] Transaction committed
I0804 21:37:26.579767      34 registry.go:150] Registering EvenPodsSpread predicate and priority function
I0804 21:37:26.579828      34 registry.go:150] Registering EvenPodsSpread predicate and priority function
time="2021-08-04T21:37:26.581104914Z" level=info msg="Running kube-scheduler --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --port=10251 --secure-port=0"
time="2021-08-04T21:37:26.582354363Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --port=10252 --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --use-service-account-credentials=true"
I0804 21:37:26.599344      34 controllermanager.go:161] Version: v1.18.8+k3s1
I0804 21:37:26.601459      34 deprecated_insecure_serving.go:53] Serving insecurely on [::]:10252
I0804 21:37:26.601543      34 leaderelection.go:242] attempting to acquire leader lease  kube-system/kube-controller-manager...
time="2021-08-04T21:37:26.602962319Z" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --allow-untagged-cloud=true --bind-address=127.0.0.1 --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --node-status-update-frequency=1m --secure-port=0"
Flag --allow-untagged-cloud has been deprecated, This flag is deprecated and will be removed in a future release. A cluster-id will be required on cloud instances.
I0804 21:37:26.605937      34 registry.go:150] Registering EvenPodsSpread predicate and priority function
I0804 21:37:26.605995      34 registry.go:150] Registering EvenPodsSpread predicate and priority function
W0804 21:37:26.608990      34 authorization.go:47] Authorization is disabled
W0804 21:37:26.609031      34 authentication.go:40] Authentication is disabled
I0804 21:37:26.609049      34 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:10251
I0804 21:37:26.617454      34 controllermanager.go:120] Version: v1.18.8+k3s1
W0804 21:37:26.617797      34 controllermanager.go:132] detected a cluster without a ClusterID.  A ClusterID will be required in the future.  Please tag your cluster to avoid any future issues
I0804 21:37:26.618051      34 leaderelection.go:242] attempting to acquire leader lease  kube-system/cloud-controller-manager...
I0804 21:37:26.710105      34 leaderelection.go:242] attempting to acquire leader lease  kube-system/kube-scheduler...
2021-08-04 21:37:32.309865 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:578" took too long (5.597216281s) to execute
2021-08-04 21:37:32.310103 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:596" took too long (2.515031253s) to execute
2021-08-04 21:37:32.310643 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/cloud-controller-manager\" " with result "range_response_count:1 size:598" took too long (3.339977935s) to execute
2021-08-04 21:37:32.312151 W | etcdserver: read-only range request "key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitions0\" " with result "range_response_count:94 size:371930" took too long (5.085179186s) to execute
2021-08-04 21:37:32.312376 W | etcdserver: read-only range request "key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitions0\" " with result "range_response_count:94 size:371930" took too long (5.613036124s) to execute
I0804 21:37:32.313636      34 trace.go:116] Trace[8002812]: "Get" url:/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:28.970063705 +0000 UTC m=+29.785619126) (total time: 3.343496837s):
Trace[8002812]: [3.343385809s] [3.343369865s] About to write a response
I0804 21:37:32.314111      34 trace.go:116] Trace[888439527]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-scheduler,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:26.712106071 +0000 UTC m=+27.527661494) (total time: 5.60197453s):
Trace[888439527]: [5.601935537s] [5.601906392s] About to write a response
I0804 21:37:32.314563      34 trace.go:116] Trace[797559310]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:29.794445301 +0000 UTC m=+30.610000858) (total time: 2.520082755s):
Trace[797559310]: [2.520043095s] [2.520005273s] About to write a response
I0804 21:37:32.355084      34 trace.go:116] Trace[1054663085]: "List etcd3" key:/apiextensions.k8s.io/customresourcedefinitions,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:26.698650722 +0000 UTC m=+27.514206143) (total time: 5.656379067s):
Trace[1054663085]: [5.656379067s] [5.656379067s] END
I0804 21:37:32.357938      34 trace.go:116] Trace[230938352]: "List etcd3" key:/apiextensions.k8s.io/customresourcedefinitions,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:27.226322528 +0000 UTC m=+28.041877951) (total time: 5.131586633s):
Trace[230938352]: [5.131586633s] [5.131586633s] END
I0804 21:37:32.389680      34 trace.go:116] Trace[218574836]: "List" url:/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:26.698624802 +0000 UTC m=+27.514180223) (total time: 5.690977144s):
Trace[218574836]: [5.65651178s] [5.656497967s] Listing from storage done
I0804 21:37:32.394255      34 trace.go:116] Trace[656729179]: "List" url:/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:37:27.226292467 +0000 UTC m=+28.041848128) (total time: 5.167910097s):
Trace[656729179]: [5.131684375s] [5.131667043s] Listing from storage done
time="2021-08-04T21:37:32.704580271Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-1.81.0.tgz"
time="2021-08-04T21:37:32.705554059Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/rolebindings.yaml"
time="2021-08-04T21:37:32.705841893Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/ccm.yaml"
time="2021-08-04T21:37:32.706252789Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml"
time="2021-08-04T21:37:32.807415573Z" level=info msg="Starting k3s.cattle.io/v1, Kind=Addon controller"
time="2021-08-04T21:37:32.808126502Z" level=info msg="Waiting for master node  startup: resource name may not be empty"
I0804 21:37:32.808298      34 leaderelection.go:242] attempting to acquire leader lease  kube-system/k3s...
time="2021-08-04T21:37:32.808793201Z" level=info msg="Node token is available at /var/lib/rancher/k3s/server/token"
time="2021-08-04T21:37:32.808881530Z" level=info msg="To join node to cluster: k3s agent -s https://172.17.0.2:6443 -t ${NODE_TOKEN}"
I0804 21:37:32.820640      34 leaderelection.go:252] successfully acquired lease kube-system/k3s
2021-08-04 21:37:32.831621 I | http: TLS handshake error from 127.0.0.1:57276: remote error: tls: bad certificate
time="2021-08-04T21:37:32.842580282Z" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
time="2021-08-04T21:37:32.842871546Z" level=info msg="Run: k3s kubectl"
time="2021-08-04T21:37:32.843161127Z" level=info msg="k3s is up and running"
time="2021-08-04T21:37:32.843755579Z" level=info msg="module overlay was already loaded"
time="2021-08-04T21:37:32.843972121Z" level=info msg="module nf_conntrack was already loaded"
time="2021-08-04T21:37:32.844183405Z" level=info msg="module br_netfilter was already loaded"
2021-08-04 21:37:32.848156 I | http: TLS handshake error from 127.0.0.1:57284: remote error: tls: bad certificate
2021-08-04 21:37:32.860744 I | http: TLS handshake error from 127.0.0.1:57292: remote error: tls: bad certificate
time="2021-08-04T21:37:32.908211332Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
time="2021-08-04T21:37:32.908714383Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
time="2021-08-04T21:37:32.908918759Z" level=info msg="Waiting for containerd startup: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: connection refused\""
time="2021-08-04T21:37:33.912961440Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
time="2021-08-04T21:37:34.916906088Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
time="2021-08-04T21:37:35.919852761Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
2021-08-04 21:37:36.636905 W | wal: sync duration of 3.817543913s, expected less than 1s
2021-08-04 21:37:36.637947 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/features/unsupported-storage-drivers\" " with result "range_response_count:1 size:746" took too long (3.787092708s) to execute
2021-08-04 21:37:36.638035 W | etcdserver: read-only range request "key:\"/registry/secrets/kube-system/k3s-serving\" " with result "range_response_count:1 size:2278" took too long (2.972115236s) to execute
2021-08-04 21:37:36.638451 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/k3s\" " with result "range_response_count:1 size:508" took too long (3.815319618s) to execute
2021-08-04 21:37:36.638613 W | etcdserver: read-only range request "key:\"/registry/helm.cattle.io/helmcharts/\" range_end:\"/registry/helm.cattle.io/helmcharts0\" limit:500 " with result "range_response_count:0 size:7" took too long (3.747223605s) to execute
2021-08-04 21:37:36.639001 W | etcdserver: read-only range request "key:\"/registry/helm.cattle.io/helmcharts/\" range_end:\"/registry/helm.cattle.io/helmcharts0\" limit:10000 " with result "range_response_count:0 size:7" took too long (3.748573313s) to execute
2021-08-04 21:37:36.639157 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/cloud-controller-manager\" " with result "range_response_count:1 size:598" took too long (768.580393ms) to execute
2021-08-04 21:37:36.639367 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/features\" range_end:\"/registry/management.cattle.io/featuret\" count_only:true " with result "range_response_count:0 size:9" took too long (3.774884651s) to execute
2021-08-04 21:37:36.639490 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/features/\" range_end:\"/registry/management.cattle.io/features0\" limit:10000 " with result "range_response_count:3 size:2168" took too long (3.775248775s) to execute
2021-08-04 21:37:36.639683 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:596" took too long (1.254538349s) to execute
2021-08-04 21:37:36.640013 W | etcdserver: read-only range request "key:\"/registry/helm.cattle.io/helmcharts\" range_end:\"/registry/helm.cattle.io/helmchartt\" count_only:true " with result "range_response_count:0 size:7" took too long (3.749531567s) to execute
2021-08-04 21:37:36.640157 W | etcdserver: read-only range request "key:\"/registry/namespaces/default\" " with result "range_response_count:1 size:979" took too long (2.433121496s) to execute
2021-08-04 21:37:36.640382 W | etcdserver: read-only range request "key:\"/registry/minions/local-node\" " with result "range_response_count:1 size:5788" took too long (2.828664605s) to execute
2021-08-04 21:37:36.641162 W | etcdserver: read-only range request "key:\"/registry/clusterrolebindings/\" range_end:\"/registry/clusterrolebindings0\" " with result "range_response_count:93 size:76427" took too long (3.789060318s) to execute
I0804 21:37:36.642937      34 trace.go:116] Trace[639411826]: "Get" url:/api/v1/namespaces/kube-system/configmaps/k3s,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:32.822842571 +0000 UTC m=+33.638397984) (total time: 3.819963244s):
Trace[639411826]: [3.819687907s] [3.81967925s] About to write a response
I0804 21:37:36.644576      34 trace.go:116] Trace[998888116]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:35.383941584 +0000 UTC m=+36.199497005) (total time: 1.260458442s):
Trace[998888116]: [1.260238306s] [1.260184764s] About to write a response
I0804 21:37:36.645925      34 trace.go:116] Trace[1092890937]: "Get" url:/api/v1/namespaces/kube-system/secrets/k3s-serving,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:33.664906593 +0000 UTC m=+34.480462016) (total time: 2.980965321s):
Trace[1092890937]: [2.980494564s] [2.980476599s] About to write a response
I0804 21:37:36.646771      34 trace.go:116] Trace[524276838]: "Get" url:/apis/management.cattle.io/v3/features/unsupported-storage-drivers,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:37:32.849778256 +0000 UTC m=+33.665333677) (total time: 3.79692944s):
Trace[524276838]: [3.796244694s] [3.796208567s] About to write a response
I0804 21:37:36.647469      34 trace.go:116] Trace[287172946]: "List etcd3" key:/helm.cattle.io/helmcharts,resourceVersion:0,limit:500,continue: (started: 2021-08-04 21:37:32.891084571 +0000 UTC m=+33.706639995) (total time: 3.756295281s):
Trace[287172946]: [3.756295281s] [3.756295281s] END
I0804 21:37:36.660746      34 trace.go:116] Trace[2145214703]: "List" url:/apis/helm.cattle.io/v1/helmcharts,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:32.891016693 +0000 UTC m=+33.706572114) (total time: 3.769693647s):
Trace[2145214703]: [3.769442889s] [3.769386376s] Listing from storage done
I0804 21:37:36.650985      34 trace.go:116] Trace[540723056]: "List etcd3" key:/management.cattle.io/features,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:32.863845789 +0000 UTC m=+33.679401214) (total time: 3.787034937s):
Trace[540723056]: [3.787034937s] [3.787034937s] END
I0804 21:37:36.651176      34 trace.go:116] Trace[497601668]: "Get" url:/api/v1/nodes/local-node,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:33.810957711 +0000 UTC m=+34.626513132) (total time: 2.840121026s):
Trace[497601668]: [2.831916973s] [2.831870589s] About to write a response
I0804 21:37:36.646829      34 trace.go:116] Trace[971510249]: "Get" url:/api/v1/namespaces/default,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:34.206293866 +0000 UTC m=+35.021849287) (total time: 2.439715117s):
Trace[971510249]: [2.439424838s] [2.439406079s] About to write a response
I0804 21:37:36.652621      34 trace.go:116] Trace[1018010203]: "List etcd3" key:/clusterrolebindings,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:32.851756489 +0000 UTC m=+33.667311912) (total time: 3.8008273s):
Trace[1018010203]: [3.8008273s] [3.8008273s] END
I0804 21:37:36.662409      34 trace.go:116] Trace[144746405]: "List" url:/apis/rbac.authorization.k8s.io/v1/clusterrolebindings,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:32.85170656 +0000 UTC m=+33.667261995) (total time: 3.810670603s):
Trace[144746405]: [3.810453438s] [3.810411542s] Listing from storage done
I0804 21:37:36.658235      34 trace.go:116] Trace[633053414]: "List etcd3" key:/helm.cattle.io/helmcharts,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:32.889684779 +0000 UTC m=+33.705240200) (total time: 3.768480761s):
Trace[633053414]: [3.768480761s] [3.768480761s] END
I0804 21:37:36.660373      34 trace.go:116] Trace[1171465763]: "Get" url:/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:35.869836494 +0000 UTC m=+36.685391916) (total time: 790.483409ms):
Trace[1171465763]: [790.436109ms] [790.419577ms] About to write a response
time="2021-08-04T21:37:36.726032130Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChart controller"
time="2021-08-04T21:37:36.726053828Z" level=info msg="Starting batch/v1, Kind=Job controller"
time="2021-08-04T21:37:36.826372914Z" level=info msg="Starting /v1, Kind=Node controller"
time="2021-08-04T21:37:36.826481743Z" level=info msg="Starting /v1, Kind=Endpoints controller"
time="2021-08-04T21:37:36.826676764Z" level=info msg="Starting /v1, Kind=Secret controller"
time="2021-08-04T21:37:36.826402935Z" level=info msg="Starting /v1, Kind=Pod controller"
time="2021-08-04T21:37:36.827585211Z" level=info msg="Starting /v1, Kind=Service controller"
time="2021-08-04T21:37:36.922511022Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
time="2021-08-04T21:37:37.926925825Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
time="2021-08-04T21:37:38.929364949Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
2021-08-04 21:37:38.989786 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/default/kubernetes\" " with result "range_response_count:1 size:277" took too long (2.272139827s) to execute
2021-08-04 21:37:38.989949 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/features/istio-virtual-service-ui\" " with result "range_response_count:1 size:798" took too long (2.268829771s) to execute
2021-08-04 21:37:38.990310 W | etcdserver: read-only range request "key:\"/registry/clusterrolebindings/cloud-controller-manager\" " with result "range_response_count:1 size:1741" took too long (2.301310945s) to execute
2021-08-04 21:37:38.990349 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/k3s\" " with result "range_response_count:1 size:508" took too long (326.464561ms) to execute
2021-08-04 21:37:38.990799 W | etcdserver: read-only range request "key:\"/registry/secrets/kube-system/k3s-serving\" " with result "range_response_count:1 size:2278" took too long (2.160348777s) to execute
I0804 21:37:38.991918      34 trace.go:116] Trace[2100397584]: "GuaranteedUpdate etcd3" type:*rbac.ClusterRoleBinding (started: 2021-08-04 21:37:36.683774389 +0000 UTC m=+37.499329814) (total time: 2.307993884s):
Trace[2100397584]: [2.307993884s] [2.307776771s] END
I0804 21:37:38.992340      34 trace.go:116] Trace[1752332777]: "Get" url:/api/v1/namespaces/kube-system/secrets/k3s-serving,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:36.829759538 +0000 UTC m=+37.645314950) (total time: 2.162490056s):
Trace[1752332777]: [2.161956812s] [2.161940351s] About to write a response
I0804 21:37:38.992758      34 trace.go:116] Trace[1282650318]: "Patch" url:/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:36.677196971 +0000 UTC m=+37.492752394) (total time: 2.315510914s):
Trace[1282650318]: [2.31514241s] [2.304268246s] Object stored in database
I0804 21:37:38.992980      34 trace.go:116] Trace[52405334]: "GuaranteedUpdate etcd3" type:*unstructured.Unstructured (started: 2021-08-04 21:37:36.719511387 +0000 UTC m=+37.535066811) (total time: 2.273426946s):
Trace[52405334]: [2.273426946s] [2.273315458s] END
I0804 21:37:38.993663      34 trace.go:116] Trace[55329941]: "Get" url:/api/v1/namespaces/default/endpoints/kubernetes,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:36.716868855 +0000 UTC m=+37.532424276) (total time: 2.276747806s):
Trace[55329941]: [2.276583296s] [2.276568454s] About to write a response
I0804 21:37:38.994273      34 trace.go:116] Trace[200967240]: "Update" url:/apis/management.cattle.io/v3/features/istio-virtual-service-ui,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:37:36.718916971 +0000 UTC m=+37.534472392) (total time: 2.27532111s):
Trace[200967240]: [2.27495189s] [2.274436657s] Object stored in database
I0804 21:37:39.001302      34 controller.go:606] quota admission added evaluator for: addons.k3s.cattle.io
time="2021-08-04T21:37:39.530524546Z" level=info msg="Active TLS secret k3s-serving (ver=2198) (count 8): map[field.cattle.io/projectId:local:p-g9nfj listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-172.17.0.2:172.17.0.2 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/hash:1042dae54ecd440e586d608fc064d597df082ce324a7538cbdf3d291a8a6e360]"
time="2021-08-04T21:37:39.932698360Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
2021-08-04 21:37:40.283479 W | wal: sync duration of 1.282800091s, expected less than 1s
time="2021-08-04T21:37:40.935302541Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
time="2021-08-04T21:37:41.937513795Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
2021-08-04 21:37:42.394045 W | wal: sync duration of 2.110258443s, expected less than 1s
2021-08-04 21:37:42.809048 W | etcdserver: read-only range request "key:\"/registry/k3s.cattle.io/addons/kube-system/ccm\" " with result "range_response_count:1 size:650" took too long (3.807087376s) to execute
2021-08-04 21:37:42.809556 W | etcdserver: request "header:<ID:7587856269140755430 > lease_revoke:<id:694d7b1318db03b4>" with result "size:32" took too long (2.525584174s) to execute
2021-08-04 21:37:42.810132 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:578" took too long (3.140276148s) to execute
2021-08-04 21:37:42.810552 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/features/multi-cluster-management\" " with result "range_response_count:0 size:7" took too long (3.278506434s) to execute
2021-08-04 21:37:42.810855 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:596" took too long (1.987817402s) to execute
2021-08-04 21:37:42.811309 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/cloud-controller-manager\" " with result "range_response_count:1 size:598" took too long (738.384626ms) to execute
2021-08-04 21:37:42.811709 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/k3s\" " with result "range_response_count:1 size:508" took too long (1.806378647s) to execute
I0804 21:37:42.812982      34 trace.go:116] Trace[1977221883]: "Get" url:/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:42.07224807 +0000 UTC m=+42.887803491) (total time: 740.595703ms):
Trace[1977221883]: [740.44856ms] [740.430433ms] About to write a response
I0804 21:37:42.813571      34 trace.go:116] Trace[585636766]: "GuaranteedUpdate etcd3" type:*unstructured.Unstructured (started: 2021-08-04 21:37:38.999793191 +0000 UTC m=+39.815348656) (total time: 3.81374111s):
Trace[585636766]: [3.81374111s] [3.813513632s] END
I0804 21:37:42.813755      34 trace.go:116] Trace[1836211634]: "Get" url:/apis/management.cattle.io/v3/features/multi-cluster-management,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:37:39.531246135 +0000 UTC m=+40.346801570) (total time: 3.282466322s):
Trace[1836211634]: [3.282466322s] [3.28237029s] END
I0804 21:37:42.813771      34 trace.go:116] Trace[569524558]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-scheduler,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:39.66928526 +0000 UTC m=+40.484840672) (total time: 3.144443148s):
Trace[569524558]: [3.1443828s] [3.144303707s] About to write a response
I0804 21:37:42.814263      34 trace.go:116] Trace[953909209]: "Update" url:/apis/k3s.cattle.io/v1/namespaces/kube-system/addons/ccm,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:38.998847587 +0000 UTC m=+39.814403001) (total time: 3.815353195s):
Trace[953909209]: [3.814871499s] [3.814068006s] Object stored in database
I0804 21:37:42.815158      34 trace.go:116] Trace[30939452]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:40.82235717 +0000 UTC m=+41.637912582) (total time: 1.992731885s):
Trace[30939452]: [1.992670395s] [1.992611884s] About to write a response
I0804 21:37:42.817907      34 trace.go:116] Trace[2107692972]: "Get" url:/api/v1/namespaces/kube-system/configmaps/k3s,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:41.004752954 +0000 UTC m=+41.820308366) (total time: 1.813062479s):
Trace[2107692972]: [1.812970694s] [1.812954413s] About to write a response
I0804 21:37:42.830733      34 controller.go:606] quota admission added evaluator for: endpoints
time="2021-08-04T21:37:42.939771545Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
2021-08-04 21:37:43.821109 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/features/fleet\" " with result "range_response_count:1 size:642" took too long (992.902412ms) to execute
I0804 21:37:43.823179      34 trace.go:116] Trace[1635767301]: "GuaranteedUpdate etcd3" type:*core.Endpoints (started: 2021-08-04 21:37:42.82918093 +0000 UTC m=+43.644736388) (total time: 993.921556ms):
Trace[1635767301]: [993.742176ms] [992.104698ms] Transaction committed
I0804 21:37:43.823472      34 trace.go:116] Trace[1652850135]: "Update" url:/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:42.825541657 +0000 UTC m=+43.641097088) (total time: 997.891268ms):
Trace[1652850135]: [997.752741ms] [994.381631ms] Object stored in database
I0804 21:37:43.823705      34 trace.go:116] Trace[1381373005]: "GuaranteedUpdate etcd3" type:*unstructured.Unstructured (started: 2021-08-04 21:37:42.823063924 +0000 UTC m=+43.638619361) (total time: 1.000577374s):
Trace[1381373005]: [1.000577374s] [999.057409ms] END
I0804 21:37:43.824191      34 trace.go:116] Trace[681760686]: "Update" url:/apis/management.cattle.io/v3/features/fleet,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:37:42.820926501 +0000 UTC m=+43.636481913) (total time: 1.003215832s):
Trace[681760686]: [1.002875757s] [1.002304788s] Object stored in database
time="2021-08-04T21:37:43.942376446Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
time="2021-08-04T21:37:44.952239750Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
time="2021-08-04T21:37:45.955077486Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
2021-08-04 21:37:46.562497 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:596" took too long (1.371474497s) to execute
2021-08-04 21:37:46.562869 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/cloud-controller-manager\" " with result "range_response_count:1 size:505" took too long (2.728039538s) to execute
2021-08-04 21:37:46.564115 W | etcdserver: read-only range request "key:\"/registry/serviceaccounts/\" range_end:\"/registry/serviceaccounts0\" " with result "range_response_count:56 size:22828" took too long (3.729322347s) to execute
2021-08-04 21:37:46.567035 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:578" took too long (1.669619884s) to execute
2021-08-04 21:37:46.567844 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/k3s\" " with result "range_response_count:1 size:508" took too long (1.737553857s) to execute
2021-08-04 21:37:46.570167 W | etcdserver: read-only range request "key:\"/registry/namespaces/default\" " with result "range_response_count:1 size:979" took too long (2.362019217s) to execute
2021-08-04 21:37:46.572912 W | etcdserver: read-only range request "key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitions0\" " with result "range_response_count:94 size:371930" took too long (2.69941809s) to execute
I0804 21:37:46.575033      34 trace.go:116] Trace[856340912]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-scheduler,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:44.896682775 +0000 UTC m=+45.712238197) (total time: 1.678274285s):
Trace[856340912]: [1.678171763s] [1.678131485s] About to write a response
I0804 21:37:46.575634      34 trace.go:116] Trace[371082943]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:45.189358214 +0000 UTC m=+46.004913638) (total time: 1.386246084s):
Trace[371082943]: [1.386147897s] [1.386109761s] About to write a response
I0804 21:37:46.575100      34 trace.go:116] Trace[860374852]: "Get" url:/api/v1/namespaces/kube-system/configmaps/k3s,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:44.82952788 +0000 UTC m=+45.645083292) (total time: 1.745510625s):
Trace[860374852]: [1.745327012s] [1.745310695s] About to write a response
I0804 21:37:46.576384      34 trace.go:116] Trace[1247262942]: "Get" url:/api/v1/namespaces/default,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:44.206659122 +0000 UTC m=+45.022214534) (total time: 2.369684125s):
Trace[1247262942]: [2.369600232s] [2.369566106s] About to write a response
I0804 21:37:46.575113      34 trace.go:116] Trace[859539008]: "Get" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:43.827362595 +0000 UTC m=+44.642918044) (total time: 2.747595524s):
Trace[859539008]: [2.747515479s] [2.747500909s] About to write a response
I0804 21:37:46.577637      34 trace.go:116] Trace[2106966267]: "List etcd3" key:/serviceaccounts,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:42.833655318 +0000 UTC m=+43.649210734) (total time: 3.743868453s):
Trace[2106966267]: [3.743868453s] [3.743868453s] END
I0804 21:37:46.578125      34 trace.go:116] Trace[1183369946]: "List" url:/api/v1/serviceaccounts,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:42.833571719 +0000 UTC m=+43.649127135) (total time: 3.74447372s):
Trace[1183369946]: [3.744100001s] [3.744040361s] Listing from storage done
I0804 21:37:46.584223      34 controller.go:606] quota admission added evaluator for: leases.coordination.k8s.io
I0804 21:37:46.635981      34 trace.go:116] Trace[671553149]: "List etcd3" key:/apiextensions.k8s.io/customresourcedefinitions,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:43.872513881 +0000 UTC m=+44.688069301) (total time: 2.763417332s):
Trace[671553149]: [2.763417332s] [2.763417332s] END
I0804 21:37:46.673109      34 trace.go:116] Trace[987805534]: "List" url:/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:37:43.87245374 +0000 UTC m=+44.688009152) (total time: 2.800589674s):
Trace[987805534]: [2.763708001s] [2.763666529s] Listing from storage done
time="2021-08-04T21:37:46.957363947Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
2021-08-04 21:37:47.651956 W | wal: sync duration of 1.069484576s, expected less than 1s
time="2021-08-04T21:37:47.959560858Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
time="2021-08-04T21:37:48.961917924Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
2021-08-04 21:37:48.973215 W | etcdserver: read-only range request "key:\"/registry/services/specs/default/kubernetes\" " with result "range_response_count:1 size:601" took too long (2.388276377s) to execute
2021-08-04 21:37:48.974528 W | etcdserver: request "header:<ID:7587856269140755448 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/cloud-controller-manager\" mod_revision:66064418 > success:<request_put:<key:\"/registry/leases/kube-system/cloud-controller-manager\" value_size:420 >> failure:<request_range:<key:\"/registry/leases/kube-system/cloud-controller-manager\" > >>" with result "size:20" took too long (1.321380863s) to execute
I0804 21:37:48.975476      34 trace.go:116] Trace[1362383775]: "GuaranteedUpdate etcd3" type:*coordination.Lease (started: 2021-08-04 21:37:46.582827829 +0000 UTC m=+47.398383275) (total time: 2.392580707s):
Trace[1362383775]: [2.392549091s] [2.390875921s] Transaction committed
I0804 21:37:48.975612      34 trace.go:116] Trace[1085845936]: "GuaranteedUpdate etcd3" type:*core.Endpoints (started: 2021-08-04 21:37:46.589131574 +0000 UTC m=+47.404687052) (total time: 2.386414612s):
Trace[1085845936]: [2.386347579s] [2.384931016s] Transaction committed
I0804 21:37:48.975723      34 trace.go:116] Trace[453555065]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:46.582655481 +0000 UTC m=+47.398210916) (total time: 2.393028416s):
Trace[453555065]: [2.392911068s] [2.392817087s] Object stored in database
I0804 21:37:48.975811      34 trace.go:116] Trace[908395425]: "Update" url:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:46.588625581 +0000 UTC m=+47.404181049) (total time: 2.387142655s):
Trace[908395425]: [2.387050844s] [2.386648244s] Object stored in database
I0804 21:37:48.976625      34 leaderelection.go:252] successfully acquired lease kube-system/cloud-controller-manager
I0804 21:37:48.976872      34 trace.go:116] Trace[14994418]: "Get" url:/api/v1/namespaces/default/services/kubernetes,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:46.583478725 +0000 UTC m=+47.399034153) (total time: 2.393348346s):
Trace[14994418]: [2.393237414s] [2.393226856s] About to write a response
I0804 21:37:48.977478      34 event.go:278] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"cloud-controller-manager", UID:"fd91f4b0-8dd9-4044-a8ea-168aa027562f", APIVersion:"v1", ResourceVersion:"66064445", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' 9919a18ed6be_a4b48ee7-1cce-4c2d-a1b1-9c53af697dac became leader
I0804 21:37:48.979873      34 event.go:278] Event(v1.ObjectReference{Kind:"Lease", Namespace:"kube-system", Name:"cloud-controller-manager", UID:"d5f8f35d-69f3-4b5c-87dc-d2939592bf33", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"66064447", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' 9919a18ed6be_a4b48ee7-1cce-4c2d-a1b1-9c53af697dac became leader
I0804 21:37:48.990624      34 node_controller.go:110] Sending events to api server.
I0804 21:37:48.990917      34 controllermanager.go:247] Started "cloud-node"
I0804 21:37:48.996380      34 node_lifecycle_controller.go:78] Sending events to api server
I0804 21:37:48.996524      34 controllermanager.go:247] Started "cloud-node-lifecycle"
E0804 21:37:49.001842      34 core.go:90] Failed to start service controller: the cloud provider does not support external load balancers
W0804 21:37:49.001877      34 controllermanager.go:244] Skipping "service"
W0804 21:37:49.001902      34 core.go:108] configure-cloud-routes is set, but cloud provider does not support routes. Will not configure cloud provider routes.
W0804 21:37:49.001925      34 controllermanager.go:244] Skipping "route"
2021-08-04 21:37:49.509751 W | wal: sync duration of 1.857382904s, expected less than 1s
2021-08-04 21:37:49.513368 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:485" took too long (2.928286634s) to execute
2021-08-04 21:37:49.513758 W | etcdserver: read-only range request "key:\"/registry/clusterroles/\" range_end:\"/registry/clusterroles0\" " with result "range_response_count:110 size:98467" took too long (2.915075205s) to execute
I0804 21:37:49.514071      34 trace.go:116] Trace[1204267153]: "Get" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:46.581802212 +0000 UTC m=+47.397357624) (total time: 2.932181503s):
Trace[1204267153]: [2.932043593s] [2.932025566s] About to write a response
2021-08-04 21:37:49.515075 W | etcdserver: read-only range request "key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range_end:\"/registry/apiextensions.k8s.io/customresourcedefinitions0\" " with result "range_response_count:94 size:371930" took too long (2.807497488s) to execute
I0804 21:37:49.522327      34 trace.go:116] Trace[1259579285]: "List etcd3" key:/clusterroles,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:46.59785323 +0000 UTC m=+47.413408673) (total time: 2.924416153s):
Trace[1259579285]: [2.924416153s] [2.924416153s] END
I0804 21:37:49.525361      34 trace.go:116] Trace[1954029444]: "List" url:/apis/rbac.authorization.k8s.io/v1beta1/clusterroles,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:46.597684822 +0000 UTC m=+47.413240254) (total time: 2.927638069s):
Trace[1954029444]: [2.925592217s] [2.925455893s] Listing from storage done
I0804 21:37:49.572752      34 trace.go:116] Trace[648852747]: "List etcd3" key:/apiextensions.k8s.io/customresourcedefinitions,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:46.706841246 +0000 UTC m=+47.522396681) (total time: 2.865862694s):
Trace[648852747]: [2.865862694s] [2.865862694s] END
I0804 21:37:49.608585      34 trace.go:116] Trace[1153063155]: "List" url:/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:37:46.706815563 +0000 UTC m=+47.522370984) (total time: 2.901688262s):
Trace[1153063155]: [2.865986145s] [2.86597282s] Listing from storage done
time="2021-08-04T21:37:49.964769464Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
2021-08-04 21:37:50.588658 W | wal: sync duration of 1.078519604s, expected less than 1s
time="2021-08-04T21:37:50.968077240Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
time="2021-08-04T21:37:51.970042317Z" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
2021-08-04 21:37:52.605060 W | wal: sync duration of 2.016087054s, expected less than 1s
2021-08-04 21:37:52.606107 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (3.601541418s) to execute
2021-08-04 21:37:52.606795 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:499" took too long (3.62683514s) to execute
2021-08-04 21:37:52.607095 W | etcdserver: read-only range request "key:\"/registry/masterleases/172.17.0.2\" " with result "range_response_count:1 size:135" took too long (3.626888643s) to execute
2021-08-04 21:37:52.607273 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/cloud-controller-manager\" " with result "range_response_count:1 size:598" took too long (3.626901915s) to execute
2021-08-04 21:37:52.607529 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/k3s\" " with result "range_response_count:1 size:508" took too long (4.016678738s) to execute
2021-08-04 21:37:52.607756 W | etcdserver: request "header:<ID:7587856269140755454 > txn:<compare:<target:MOD key:\"/registry/events/kube-system/cloud-controller-manager.16983835ca845a8a\" mod_revision:0 > success:<request_put:<key:\"/registry/events/kube-system/cloud-controller-manager.16983835ca845a8a\" value_size:685 lease:7587856269140755451 >> failure:<>>" with result "size:20" took too long (2.017459936s) to execute
I0804 21:37:52.608420      34 trace.go:116] Trace[524787921]: "Get" url:/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:48.979201597 +0000 UTC m=+49.794757018) (total time: 3.629098278s):
Trace[524787921]: [3.628975516s] [3.628967725s] About to write a response
2021-08-04 21:37:52.608692 W | etcdserver: read-only range request "key:\"/registry/namespaces/cattle-system\" " with result "range_response_count:1 size:947" took too long (2.718625195s) to execute
2021-08-04 21:37:52.610563 W | etcdserver: read-only range request "key:\"/registry/clusterrolebindings/\" range_end:\"/registry/clusterrolebindings0\" " with result "range_response_count:93 size:76427" took too long (3.067115223s) to execute
I0804 21:37:52.608872      34 trace.go:116] Trace[1263633179]: "Get" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:48.978869877 +0000 UTC m=+49.794425298) (total time: 3.629904178s):
Trace[1263633179]: [3.629729534s] [3.629700684s] About to write a response
I0804 21:37:52.608871      34 trace.go:116] Trace[1382440039]: "Get" url:/api/v1/namespaces/kube-system/configmaps/k3s,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:48.589288244 +0000 UTC m=+49.404843656) (total time: 4.019538158s):
Trace[1382440039]: [4.01939998s] [4.019383354s] About to write a response
I0804 21:37:52.611115      34 trace.go:116] Trace[2092361357]: "Get" url:/api/v1/namespaces/cattle-system,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:37:49.889358664 +0000 UTC m=+50.704914086) (total time: 2.72172475s):
Trace[2092361357]: [2.721221729s] [2.721176912s] About to write a response
I0804 21:37:52.617419      34 trace.go:116] Trace[695588863]: "Create" url:/api/v1/namespaces/kube-system/events,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/cloud-controller-manager,client:127.0.0.1 (started: 2021-08-04 21:37:48.980822323 +0000 UTC m=+49.796377771) (total time: 3.636563457s):
Trace[695588863]: [3.636451223s] [3.636184104s] Object stored in database
I0804 21:37:52.617560      34 trace.go:116] Trace[2015473193]: "List etcd3" key:/clusterrolebindings,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:49.542805089 +0000 UTC m=+50.358360504) (total time: 3.074713246s):
Trace[2015473193]: [3.074713246s] [3.074713246s] END
I0804 21:37:52.618435      34 trace.go:116] Trace[1681050472]: "List" url:/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:49.542713411 +0000 UTC m=+50.358268832) (total time: 3.075688003s):
Trace[1681050472]: [3.074904157s] [3.07483841s] Listing from storage done
2021-08-04 21:37:55.033885 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (3.024543154s) to execute
2021-08-04 21:37:55.034043 W | etcdserver: request "header:<ID:7587856269140755459 > lease_grant:<ttl:15-second id:694d7b1318db0402>" with result "size:44" took too long (2.044208443s) to execute
2021-08-04 21:37:55.034175 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:578" took too long (3.095815112s) to execute
I0804 21:37:55.035187      34 trace.go:116] Trace[790643612]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-scheduler,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:51.937327392 +0000 UTC m=+52.752882815) (total time: 3.097777845s):
Trace[790643612]: [3.097620136s] [3.097569326s] About to write a response
I0804 21:37:55.035902      34 trace.go:116] Trace[2139749666]: "GuaranteedUpdate etcd3" type:*coordination.Lease (started: 2021-08-04 21:37:52.614200261 +0000 UTC m=+53.429755698) (total time: 2.421653508s):
Trace[2139749666]: [2.421588605s] [2.420805636s] Transaction committed
I0804 21:37:55.036144      34 trace.go:116] Trace[1275953149]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:52.614008588 +0000 UTC m=+53.429564000) (total time: 2.422085247s):
Trace[1275953149]: [2.421978007s] [2.421890594s] Object stored in database
I0804 21:37:55.036184      34 trace.go:116] Trace[58036118]: "Create" url:/api/v1/namespaces/kube-system/events,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/cloud-controller-manager,client:127.0.0.1 (started: 2021-08-04 21:37:52.619908529 +0000 UTC m=+53.435463941) (total time: 2.416240241s):
Trace[58036118]: [2.416119132s] [2.416032949s] Object stored in database
I0804 21:37:55.035902      34 trace.go:116] Trace[118603121]: "GuaranteedUpdate etcd3" type:*core.ConfigMap (started: 2021-08-04 21:37:52.614335651 +0000 UTC m=+53.429891075) (total time: 2.421517272s):
Trace[118603121]: [2.421453211s] [2.420368723s] Transaction committed
I0804 21:37:55.037027      34 trace.go:116] Trace[826815172]: "Update" url:/api/v1/namespaces/kube-system/configmaps/k3s,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:52.614076233 +0000 UTC m=+53.429631668) (total time: 2.422785863s):
Trace[826815172]: [2.422656394s] [2.422455953s] Object stored in database
I0804 21:37:55.037597      34 leaderelection.go:252] successfully acquired lease kube-system/kube-controller-manager
I0804 21:37:55.038205      34 event.go:278] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-controller-manager", UID:"34608d2c-8b9a-43eb-add2-fff8918dde72", APIVersion:"v1", ResourceVersion:"66064448", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' 9919a18ed6be_027e99c7-3ef1-4b3e-8542-9fa710ff284b became leader
I0804 21:37:55.038282      34 event.go:278] Event(v1.ObjectReference{Kind:"Lease", Namespace:"kube-system", Name:"kube-controller-manager", UID:"3dccda3d-2870-45f9-8a09-4cde7ba5860f", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"66064451", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' 9919a18ed6be_027e99c7-3ef1-4b3e-8542-9fa710ff284b became leader
2021-08-04 21:37:55.070007 W | wal: sync duration of 2.080558102s, expected less than 1s
2021-08-04 21:37:55.071262 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/settings/first-login\" " with result "range_response_count:1 size:563" took too long (2.406623176s) to execute
2021-08-04 21:37:55.071898 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/settings\" range_end:\"/registry/management.cattle.io/settingt\" count_only:true " with result "range_response_count:0 size:9" took too long (2.408721931s) to execute
2021-08-04 21:37:55.072100 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/cloud-controller-manager\" " with result "range_response_count:1 size:503" took too long (2.454310559s) to execute
2021-08-04 21:37:55.072788 W | etcdserver: read-only range request "key:\"/registry/configmaps/\" range_end:\"/registry/configmaps0\" " with result "range_response_count:13 size:31906" took too long (2.446116513s) to execute
2021-08-04 21:37:55.073886 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/settings/\" range_end:\"/registry/management.cattle.io/settings0\" limit:10000 " with result "range_response_count:81 size:61785" took too long (2.410736437s) to execute
I0804 21:37:55.075152      34 trace.go:116] Trace[2133912374]: "Get" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:52.616895003 +0000 UTC m=+53.432450413) (total time: 2.458170838s):
Trace[2133912374]: [2.458001221s] [2.457991937s] About to write a response
I0804 21:37:55.076722      34 trace.go:116] Trace[599529242]: "List etcd3" key:/configmaps,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:52.626067858 +0000 UTC m=+53.441623279) (total time: 2.450562363s):
Trace[599529242]: [2.450562363s] [2.450562363s] END
I0804 21:37:55.077231      34 trace.go:116] Trace[1268639727]: "List" url:/api/v1/configmaps,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:52.62600942 +0000 UTC m=+53.441564843) (total time: 2.451168308s):
Trace[1268639727]: [2.45084102s] [2.45078953s] Listing from storage done
I0804 21:37:55.078358      34 trace.go:116] Trace[501745253]: "Get" url:/apis/management.cattle.io/v3/settings/first-login,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:37:52.664181541 +0000 UTC m=+53.479736976) (total time: 2.414127308s):
Trace[501745253]: [2.413791396s] [2.413736791s] About to write a response
I0804 21:37:55.099638      34 trace.go:116] Trace[1973102802]: "List etcd3" key:/management.cattle.io/settings,resourceVersion:,limit:10000,continue: (started: 2021-08-04 21:37:52.662441915 +0000 UTC m=+53.477997417) (total time: 2.437157971s):
Trace[1973102802]: [2.437157971s] [2.437157971s] END
2021-08-04 21:37:57.733665 W | wal: sync duration of 2.484694298s, expected less than 1s
2021-08-04 21:37:58.139329 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:485" took too long (3.100140445s) to execute
2021-08-04 21:37:58.139526 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:596" took too long (3.099902094s) to execute
2021-08-04 21:37:58.139686 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (3.130943605s) to execute
2021-08-04 21:37:58.139974 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (3.09185573s) to execute
2021-08-04 21:37:58.140263 W | etcdserver: request "header:<ID:7587856269140755468 > lease_grant:<ttl:15-second id:694d7b1318db040b>" with result "size:44" took too long (2.891141235s) to execute
I0804 21:37:58.140841      34 trace.go:116] Trace[678366839]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:55.039271 +0000 UTC m=+55.854826415) (total time: 3.101479744s):
Trace[678366839]: [3.101367064s] [3.101348701s] About to write a response
I0804 21:37:58.140849      34 trace.go:116] Trace[1617861981]: "Get" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:55.038685035 +0000 UTC m=+55.854240456) (total time: 3.102079734s):
Trace[1617861981]: [3.101948176s] [3.101912946s] About to write a response
I0804 21:37:58.142520      34 trace.go:116] Trace[1569664990]: "GuaranteedUpdate etcd3" type:*core.Endpoints (started: 2021-08-04 21:37:55.078480408 +0000 UTC m=+55.894035848) (total time: 3.063974924s):
Trace[1569664990]: [3.06392986s] [3.063155993s] Transaction committed
2021-08-04 21:37:58.142937 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/k3s\" " with result "range_response_count:1 size:508" took too long (1.100610035s) to execute
I0804 21:37:58.143335      34 trace.go:116] Trace[1152121597]: "Update" url:/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-08-04 21:37:55.078227074 +0000 UTC m=+55.893782511) (total time: 3.065077734s):
Trace[1152121597]: [3.064971769s] [3.064776062s] Object stored in database
2021-08-04 21:37:58.143340 W | etcdserver: read-only range request "key:\"/registry/deployments/\" range_end:\"/registry/deployments0\" " with result "range_response_count:6 size:20789" took too long (3.056785017s) to execute
2021-08-04 21:37:58.143554 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (134.638856ms) to execute
2021-08-04 21:37:58.144096 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/settings/ingress-ip-domain\" " with result "range_response_count:1 size:571" took too long (3.060277762s) to execute
I0804 21:37:58.146047      34 trace.go:116] Trace[1143224240]: "Create" url:/api/v1/namespaces/kube-system/events,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/kube-controller-manager,client:127.0.0.1 (started: 2021-08-04 21:37:55.078760864 +0000 UTC m=+55.894316290) (total time: 3.067243352s):
Trace[1143224240]: [3.067152858s] [3.067073174s] Object stored in database
I0804 21:37:58.146363      34 trace.go:116] Trace[2036443019]: "List etcd3" key:/deployments,resourceVersion:,limit:0,continue: (started: 2021-08-04 21:37:55.086026815 +0000 UTC m=+55.901582235) (total time: 3.060299124s):
Trace[2036443019]: [3.060299124s] [3.060299124s] END
I0804 21:37:58.146700      34 trace.go:116] Trace[654677306]: "Get" url:/api/v1/namespaces/kube-system/configmaps/k3s,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:57.041518991 +0000 UTC m=+57.857074414) (total time: 1.10514262s):
Trace[654677306]: [1.105072972s] [1.105052329s] About to write a response
I0804 21:37:58.147089      34 trace.go:116] Trace[1677292233]: "GuaranteedUpdate etcd3" type:*v1.Endpoints (started: 2021-08-04 21:37:48.978073768 +0000 UTC m=+49.793629240) (total time: 9.168985657s):
Trace[1677292233]: [3.630117482s] [3.630117482s] initial value restored
Trace[1677292233]: [6.056748875s] [2.426631393s] Transaction prepared
Trace[1677292233]: [9.16403596s] [3.067305911s] Transaction prepared
I0804 21:37:58.147865      34 trace.go:116] Trace[1119830032]: "Get" url:/apis/management.cattle.io/v3/settings/ingress-ip-domain,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2021-08-04 21:37:55.083463505 +0000 UTC m=+55.899018940) (total time: 3.064365801s):
Trace[1119830032]: [3.064088485s] [3.064049397s] About to write a response
I0804 21:37:58.148678      34 trace.go:116] Trace[1685972099]: "List" url:/apis/apps/v1/deployments,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531,client:127.0.0.1 (started: 2021-08-04 21:37:55.08587952 +0000 UTC m=+55.901434960) (total time: 3.062754218s):
Trace[1685972099]: [3.060578298s] [3.060525856s] Listing from storage done
I0804 21:37:58.861487      34 plugins.go:100] No cloud provider specified.
I0804 21:37:58.870508      34 shared_informer.go:223] Waiting for caches to sync for tokens
I0804 21:37:58.970940      34 shared_informer.go:230] Caches are synced for tokens
E0804 21:37:58.977693      34 leaderelection.go:356] Failed to update lock: resource name may not be empty
I0804 21:37:58.977774      34 leaderelection.go:277] failed to renew lease kube-system/cloud-controller-manager: timed out waiting for the condition
F0804 21:37:58.977833      34 controllermanager.go:213] leaderelection lost
2021-08-04 21:37:59.021444 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/cloud-controller-manager\" " with result "error:context canceled" took too long (866.34191ms) to execute
2021-08-04 21:37:59.192158 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/settings/system-namespaces\" " with result "error:context canceled" took too long (1.035233864s) to execute
WARNING: 2021/08/04 21:37:59 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:37:59.194113 W | etcdserver: read-only range request "key:\"/registry/serviceaccounts/kube-system/certificate-controller\" " with result "error:context canceled" took too long (323.973882ms) to execute
WARNING: 2021/08/04 21:37:59 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021-08-04 21:37:59.194368 W | etcdserver: read-only range request "key:\"/registry/services/specs/\" range_end:\"/registry/services/specs0\" " with result "error:context canceled" took too long (1.026166666s) to execute
WARNING: 2021/08/04 21:37:59 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2021/08/04 21:37:59 [FATAL] k3s exited with: exit status 255

seco511 · August 22, 2021, 3:10am

I had the same problem till now

raymodart · November 18, 2021, 5:52am

I have the same situation

wcoateRR · November 18, 2021, 2:42pm

@Jade_Baxter - Looking at the top of the logs you posted I saw several “waiting” or "“took too long” or “deadline exceeded” messages. If your SSD is massively bogged down with I/O, possibly you’re doing this with VMs on the same physical device and have massively overloaded your hypervisor?

I know a bit over a decade ago I had an application with Oracle DB & WebLogic both required and on a physical box I could install both, but on my VM server if I installed both on a single VM it’d fail every time but installing on two separate VMs was fine (that was ESX 3 or ESXi 4 for hypervisor as I recall).

VMs are great and convenient, but sometimes applications don’t always know how to handle the entire machine getting paged out of CPU for a while and don’t have their timeouts written to accommodate that, and I/O delays can be murder for Linux (which hypervisors tend to be based upon or similar to). Maybe try getting another hypervisor & splitting your load?

Jade_Baxter · November 18, 2021, 4:49pm

So what it turned out to be is consumer ssds were unable to keep up. Even having 4 in zfs they couldnt keep up. 4 Samsung 4tb ssds performed a little better than 8 smr hdds in this usecase.

As a test I moved vms to single Enterprise plp drive and problem instantly disappeared! So I added a second drive to add write headroom capacity and have not encountered this issue again. With exception to starting all rancher worker nodes at same time. So I intend to add 2 more drives to Enterprise ssd pool.

Seems that rancher is quite sensitive to bad IO perormance. If I ran the same vms with docker only it was better in the sense that rancher didn’t keep restarting pods, but best solution was to use Enterprise drives.

wcoateRR · November 18, 2021, 6:44pm

Cascading timeouts & retries is never fun, and past a certain point it can’t recover. I’ve seen a lot of places where I/O delay has a certain amount of headroom and if you go past it you’re just hosed and need to do something drastic.

Jade_Baxter · November 18, 2021, 7:07pm

Yeah I was starting to think about multiple hosts. But saw posts where claims of enterprise ssds being better.

I was getting 1% wear out on consumer ssds per week!!!

So took the gamble and made an instant difference, infact the difference was almost unreal.
1 Enterprise ssd could out perform 4xextensive consumer grade ssds. But io wait was still higher during an vm updates, so added another ssd and this made it fine during updates but will add another 2 to add more headroom.

I am moving file server duties out of host anyways, more to reduce complexity but will also reduce chances of IO pressure. Having the app server and file server duties separate.

Topic		Replies	Views
Help on issue "[FATAL] k3s exited with: exit status 255" Rancher	8	7102	June 20, 2022
Before upgrading 1.1.4 -> 1.2.0, restart rancher-server with `--restart=unless-stopped` Rancher 1.x	0	1599	December 5, 2016
Install Rancher: [FATAL] error running the jail command: exit status 2 Rancher	1	1166	April 5, 2023
Rancher 2.7 on Docker fails start after server reboot Rancher	5	6204	March 7, 2023
PANIC: Failed to start rancher:server Rancher	0	1124	March 17, 2020

New installation of rancher fatal error exit with status 1

Related Topics