Rancher admin ui never starts

Jade_Baxter · July 23, 2021, 6:11pm

Hi,

I have 1 node with rancher
1 node with control plane and etc
3 nodes as workers

Admin ui fails to start.
How can I resolve this? Or find out what went wrong.
I restarted all vms and all the pods are running. But I can’t make changes or manage cluster via ui.

Can I get some support on where to next?
Can I build another rancher vm and import current cluster?
Can I export my manifests so if I need to blow the cluster away and start again I can easily start applications.

Jade

Jade_Baxter · July 23, 2021, 6:12pm

~$ docker logs --tail 100 intelligent_euclid

I0723 18:09:04.976096 30 container_manager_linux.go:282] Creating Container Manager object based on Node Config: {RuntimeCgroupsName:/systemd SystemCgroupsName: KubeletCgroupsName:/systemd ContainerRuntime:remote CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity: Percentage:0.05} GracePeriod:0s MinReclaim:} {Signal:nodefs.available Operator:LessThan Value:{Quantity: Percentage:0.05} GracePeriod:0s MinReclaim:}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:false CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
I0723 18:09:04.976583 30 topology_manager.go:126] [topologymanager] Creating topology manager with none policy
I0723 18:09:04.976783 30 container_manager_linux.go:312] [topologymanager] Initializing Topology Manager with none policy
I0723 18:09:04.976979 30 container_manager_linux.go:317] Creating device plugin manager: true
W0723 18:09:04.977438 30 util_unix.go:103] Using “/run/k3s/containerd/containerd.sock” as endpoint is deprecated, please consider using full url format “unix:///run/k3s/containerd/containerd.sock”.
W0723 18:09:04.977879 30 util_unix.go:103] Using “/run/k3s/containerd/containerd.sock” as endpoint is deprecated, please consider using full url format “unix:///run/k3s/containerd/containerd.sock”.
I0723 18:09:04.978275 30 kubelet.go:317] Watching apiserver
time=“2021-07-23T18:09:04.982505180Z” level=info msg=“labels have already set on node: local-node”
I0723 18:09:04.976679 30 flannel.go:105] Using interface with name eth0 and address 172.17.0.2
I0723 18:09:04.998615 30 kube.go:117] Waiting 10m0s for node controller to sync
W0723 18:09:04.975737 30 proxier.go:635] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0723 18:09:05.018639 30 proxier.go:635] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
I0723 18:09:05.030189 30 kube.go:300] Starting kube subnet manager
W0723 18:09:05.030835 30 proxier.go:635] Failed to load kernel module nf_conntrack with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
I0723 18:09:05.046476 30 kuberuntime_manager.go:211] Container runtime containerd initialized, version: v1.3.3-k3s2, apiVersion: v1alpha2
I0723 18:09:05.077269 30 server.go:1124] Started kubelet
I0723 18:09:05.101638 30 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
I0723 18:09:05.102744 30 server.go:145] Starting to listen on 0.0.0.0:10250
I0723 18:09:05.107034 30 server.go:393] Adding debug handlers to kubelet server.
I0723 18:09:05.126363 30 volume_manager.go:265] Starting Kubelet Volume Manager
I0723 18:09:05.128294 30 desired_state_of_world_populator.go:139] Desired state populator starts to run
E0723 18:09:05.131853 30 cri_stats_provider.go:375] Failed to get the info of the filesystem with mountpoint “/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs”: unable to find data in memory cache.
E0723 18:09:05.131995 30 kubelet.go:1306] Image garbage collection failed once. Stats initialization may not have completed yet: invalid capacity 0 on image filesystem
I0723 18:09:05.133377 30 node.go:136] Successfully retrieved node IP: 172.17.0.2
I0723 18:09:05.133429 30 server_others.go:187] Using iptables Proxier.
I0723 18:09:05.135171 30 network_policy_controller.go:149] Starting network policy controller
I0723 18:09:05.143957 30 server.go:583] Version: v1.18.8+k3s1
I0723 18:09:05.144905 30 conntrack.go:103] Set sysctl ‘net/netfilter/nf_conntrack_max’ to 163840
F0723 18:09:05.144986 30 server.go:497] open /proc/sys/net/netfilter/nf_conntrack_max: permission denied
2021/07/23 18:09:05 [FATAL] k3s exited with: exit status 1
2021/07/23 18:09:06 [INFO] Rancher version v2.5.7 (c824d91cd) is starting
2021/07/23 18:09:06 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Agent:false Features:}
2021/07/23 18:09:06 [INFO] Listening on /tmp/log.sock
2021/07/23 18:09:06 [INFO] Running etcd --data-dir=management-state/etcd --heartbeat-interval=500 --election-timeout=5000
2021-07-23 18:09:06.993927 W | pkg/flags: unrecognized environment variable ETCD_URL_arm64=https://github.com/etcd-io/etcd/releases/download/v3.4.3/etcd-v3.4.3-linux-arm64.tar.gz
2021-07-23 18:09:06.994063 W | pkg/flags: unrecognized environment variable ETCD_URL_amd64=https://github.com/etcd-io/etcd/releases/download/v3.4.3/etcd-v3.4.3-linux-amd64.tar.gz
2021-07-23 18:09:06.994077 W | pkg/flags: unrecognized environment variable ETCD_UNSUPPORTED_ARCH=amd64
2021-07-23 18:09:06.994087 W | pkg/flags: unrecognized environment variable ETCD_URL=ETCD_URL_amd64
[WARNING] Deprecated ‘–logger=capnslog’ flag is set; use ‘–logger=zap’ flag instead
2021-07-23 18:09:06.994178 I | etcdmain: etcd Version: 3.4.3
2021-07-23 18:09:06.994190 I | etcdmain: Git SHA: 3cf2f69b5
2021-07-23 18:09:06.994198 I | etcdmain: Go Version: go1.12.12
2021-07-23 18:09:06.994206 I | etcdmain: Go OS/Arch: linux/amd64
2021-07-23 18:09:06.994215 I | etcdmain: setting maximum number of CPUs to 5, total number of available CPUs is 5
2021-07-23 18:09:06.994366 N | etcdmain: the server is already initialized as member before, starting as etcd member…
[WARNING] Deprecated ‘–logger=capnslog’ flag is set; use ‘–logger=zap’ flag instead
2021-07-23 18:09:06.995768 I | embed: name = default
2021-07-23 18:09:06.995802 I | embed: data dir = management-state/etcd
2021-07-23 18:09:06.995814 I | embed: member dir = management-state/etcd/member
2021-07-23 18:09:06.995823 I | embed: heartbeat = 500ms
2021-07-23 18:09:06.995831 I | embed: election = 5000ms
2021-07-23 18:09:06.995839 I | embed: snapshot count = 100000
2021-07-23 18:09:06.995883 I | embed: advertise client URLs = http://localhost:2379
2021-07-23 18:09:06.995899 I | embed: initial advertise peer URLs = http://localhost:2380
2021-07-23 18:09:06.995911 I | embed: initial cluster =
2021-07-23 18:09:07.176287 I | etcdserver: recovered store from snapshot at index 680006892021-07-23 18:09:07.184543 I | mvcc: restore compact to 64981055
2021-07-23 18:09:08.195619 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 68060400
raft2021/07/23 18:09:08 INFO: 8e9e05c52164694d switched to configuration voters=(10276657743932975437)
raft2021/07/23 18:09:08 INFO: 8e9e05c52164694d became follower at term 3601
raft2021/07/23 18:09:08 INFO: newRaft 8e9e05c52164694d [peers: [8e9e05c52164694d], term: 3601, commit: 68060400, applied: 68000689, lastindex: 68060400, lastterm: 3601]
2021-07-23 18:09:08.206590 I | etcdserver/api: enabled capabilities for version 3.4
2021-07-23 18:09:08.206628 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32 from store
2021-07-23 18:09:08.206643 I | etcdserver/membership: set the cluster version to 3.4 from store
2021-07-23 18:09:08.218843 I | mvcc: restore compact to 64981055
2021-07-23 18:09:08.388087 W | auth: simple token is not cryptographically signed
2021-07-23 18:09:08.445779 I | etcdserver: starting server… [version: 3.4.3, cluster version: 3.4]
2021-07-23 18:09:08.446238 I | etcdserver: 8e9e05c52164694d as single-node; fast-forwarding 9 ticks (election ticks 10)
2021-07-23 18:09:08.452685 I | embed: listening for peers on 127.0.0.1:2380
raft2021/07/23 18:09:12 INFO: 8e9e05c52164694d is starting a new election at term 3601
raft2021/07/23 18:09:12 INFO: 8e9e05c52164694d became candidate at term 3602
raft2021/07/23 18:09:12 INFO: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 3602
raft2021/07/23 18:09:12 INFO: 8e9e05c52164694d became leader at term 3602
raft2021/07/23 18:09:12 INFO: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 3602
2021-07-23 18:09:12.719814 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
2021-07-23 18:09:12.719882 I | embed: ready to serve client requests
2021-07-23 18:09:12.721140 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
2021/07/23 18:09:12 [INFO] Waiting for server to become available: Get “https://127.0.0.1:6443/version?timeout=15m0s”: dial tcp 127.0.0.1:6443: connect: connection refused
time=“2021-07-23T18:09:13.239969282Z” level=info msg=“Starting k3s v1.18.8+k3s1 (6b595318)”
time=“2021-07-23T18:09:13.240851676Z” level=info msg=“Cluster bootstrap already complete”
time=“2021-07-23T18:09:13.276462037Z” level=info msg=“Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=unknown --authorization-mode=Node,RBAC --basic-auth-file=/var/lib/rancher/k3s/server/cred/passwd --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=http://localhost:2379 --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=k3s --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key”
Flag --basic-auth-file has been deprecated, Basic authentication mode is deprecated and will be removed in a future release. It is not recommended for production environments.
I0723 18:09:13.277696 35 server.go:645] external host was not specified, using 172.17.0.2
I0723 18:09:13.278299 35 server.go:162] Version: v1.18.8+k3s1
I0723 18:09:13.288550 35 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0723 18:09:13.288771 35 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0723 18:09:13.290445 35 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0723 18:09:13.290482 35 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0723 18:09:13.343108 35 master.go:270] Using reconciler: lease
I0723 18:09:13.403724 35 rest.go:113] the default service ipfamily for this cluster is: IPv4
W0723 18:09:14.009759 35 genericapiserver.go:409] Skipping API batch/v2alpha1 because it has no resources.
W0723 18:09:14.035626 35 genericapiserver.go:409] Skipping API discovery.k8s.io/v1alpha1 because it has no resources.
W0723 18:09:14.065731 35 genericapiserver.go:409] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0723 18:09:14.114137 35 genericapiserver.go:409] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0723 18:09:14.139940 35 genericapiserver.go:409] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0723 18:09:14.186191 35 genericapiserver.go:409] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0723 18:09:14.219755 35 genericapiserver.go:409] Skipping API apps/v1beta2 because it has no resources.
W0723 18:09:14.219999 35 genericapiserver.go:409] Skipping API apps/v1beta1 because it has no resources.
I0723 18:09:14.245346 35 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0723 18:09:14.245459 35 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.

superseb · July 24, 2021, 11:04am

Change /proc/sys/net/netfilter/nf_conntrack_max on the host to 163840 and it should be able to start.

Jade_Baxter · July 24, 2021, 11:32am

is there any reason this has happened? did I do something, or was this number wrong to start with?

just trying to understand how/why this happened.

Thanks in advance Jade

Jade_Baxter · July 24, 2021, 11:40am

how do I set this value so its persists between boots? because I restarted vm and value went back to 262144. I set it back to 163840 and rancher started again.

Jade_Baxter · July 25, 2021, 5:43am

Now Im getting this:

github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints.restfulListResource.func1(0xc002ff9f80, 0xc00061ae70)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1157 +0x91
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc002ff9f80, 0xc00061ae70)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:384 +0x282
github.com/rancher/k3s/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc001750000, 0x4c4f9c0, 0xc00061ae00, 0xc00a109900)
/go/src/github.com/rancher/k3s/vendor/github.com/emicklei/go-restful/container.go:288 +0xa84
github.com/rancher/k3s/vendor/github.com/emicklei/go-restful.(*Container).Dispatch(…)
/go/src/github.com/rancher/k3s/vendor/github.com/emicklei/go-restful/container.go:199
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x44ffbc5, 0xe, 0xc001750000, 0xc001428540, 0x4c4f9c0, 0xc00061ae00, 0xc00a109900)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/handler.go:146 +0x539
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc006881380, 0x4c4f9c0, 0xc00061ae00, 0xc00a109900)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x20f
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc008fdf500, 0x4c4f9c0, 0xc00061ae00, 0xc00a109900)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8c
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x4503a32, 0xf, 0xc00334ec60, 0xc008fdf500, 0x4c4f9c0, 0xc00061ae00, 0xc00a109900)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x74d
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x4c4f9c0, 0xc00061ae00, 0xc00a109900)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x563
net/http.HandlerFunc.ServeHTTP(0xc008f1fe00, 0x4c4f9c0, 0xc00061ae00, 0xc00a109900)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x4c4f9c0, 0xc00061ae00, 0xc00a109900)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:149 +0x582
net/http.HandlerFunc.ServeHTTP(0xc0090037a0, 0x4c4f9c0, 0xc00061ae00, 0xc00a109900)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x4c4f9c0, 0xc00061ae00, 0xc00a109900)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x231d
net/http.HandlerFunc.ServeHTTP(0xc008f1fe40, 0x4c4f9c0, 0xc00061ae00, 0xc00a109900)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x4c4f9c0, 0xc00061ae00, 0xc00a109700)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:70 +0x672
net/http.HandlerFunc.ServeHTTP(0xc008faf1d0, 0x4c4f9c0, 0xc00061ae00, 0xc00a109700)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc008ff3ce0, 0x4c4f9c0, 0xc00061ae00, 0xc00a109700)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x42a
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters.WithWaitGroup.func1(0x4c4f9c0, 0xc00061ae00, 0xc00a109700)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:59 +0x137
net/http.HandlerFunc.ServeHTTP(0xc0090037d0, 0x4c4f9c0, 0xc00061ae00, 0xc00a109700)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1(0x4c4f9c0, 0xc00061ae00, 0xc00a109600)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39 +0x269
net/http.HandlerFunc.ServeHTTP(0xc009003800, 0x4c4f9c0, 0xc00061ae00, 0xc00a109600)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithWarningRecorder.func1(0x4c4f9c0, 0xc00061ae00, 0xc00a109500)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters/warning.go:35 +0x1a7
net/http.HandlerFunc.ServeHTTP(0xc008ff3d00, 0x4c4f9c0, 0xc00061ae00, 0xc00a109500)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1(0x4c4f9c0, 0xc00061ae00, 0xc00a109500)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters/cachecontrol.go:31 +0xa8
net/http.HandlerFunc.ServeHTTP(0xc008ff3d20, 0x4c4f9c0, 0xc00061ae00, 0xc00a109500)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/httplog.WithLogging.func1(0x4c41e00, 0xc00d185118, 0xc009dd2a00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:91 +0x2f2
net/http.HandlerFunc.ServeHTTP(0xc008ff3d40, 0x4c41e00, 0xc00d185118, 0xc009dd2a00)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1(0x4c41e00, 0xc00d185118, 0xc009dd2a00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:51 +0xe6
net/http.HandlerFunc.ServeHTTP(0xc008ff3d60, 0x4c41e00, 0xc00d185118, 0xc009dd2a00)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0xc009003830, 0x4c41e00, 0xc00d185118, 0xc009dd2a00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/handler.go:189 +0x51
net/http.serverHandler.ServeHTTP(0xc00c169880, 0x4c41e00, 0xc00d185118, 0xc009dd2a00)
/usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.initALPNRequest.ServeHTTP(0x4c77340, 0xc0070f1080, 0xc0095d1500, 0xc00c169880, 0x4c41e00, 0xc00d185118, 0xc009dd2a00)
/usr/local/go/src/net/http/server.go:3415 +0x8d
github.com/rancher/k3s/vendor/golang.org/x/net/http2.(*serverConn).runHandler(0xc003812d80, 0xc00d185118, 0xc009dd2a00, 0xc00c399b60)
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/server.go:2152 +0x8b
created by github.com/rancher/k3s/vendor/golang.org/x/net/http2.(*serverConn).processHeaders
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/server.go:1882 +0x505

goroutine 2698 [select, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc0088f9200, 0x45d964b800, 0x0, 0xc0088f91a0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:588 +0x17b
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:571 +0x8c

goroutine 2483 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc0068815d0, 0xc000000001)
/usr/local/go/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc0068815c0)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/rancher/k3s/vendor/k8s.io/client-go/util/workqueue.(*Type).Get(0xc00758a480, 0x0, 0x0, 0x3d6dd00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/util/workqueue/queue.go:145 +0x89
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicServingCertificateController).processNextWorkItem(0xc00c31fb00, 0x203000)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates/tlsconfig.go:263 +0x66
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicServingCertificateController).runWorker(0xc00c31fb00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates/tlsconfig.go:258 +0x2b
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00ac8c180)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00ac8c180, 0x4bb9920, 0xc0070f0f30, 0x46eb701, 0xc00d6836e0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00ac8c180, 0x3b9aca00, 0x0, 0x1, 0xc00d6836e0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc00ac8c180, 0x3b9aca00, 0xc00d6836e0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicServingCertificateController).Run
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates/tlsconfig.go:247 +0x1b3

goroutine 2484 [select, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00ac8c1a0, 0x4bb9920, 0xc0070f0f00, 0xc00305b601, 0xc00d6836e0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x149
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00ac8c1a0, 0xdf8475800, 0x0, 0x100000000000001, 0xc00d6836e0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc00ac8c1a0, 0xdf8475800, 0xc00d6836e0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicServingCertificateController).Run
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates/tlsconfig.go:250 +0x22b

goroutine 2983 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc00929f220, 0xc000000000)
/usr/local/go/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc00929f210)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/rancher/k3s/vendor/golang.org/x/net/http2.(*pipe).Read(0xc00929f208, 0xc007fad8ec, 0x4, 0x4, 0x0, 0x0, 0x0)
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/pipe.go:65 +0x97
github.com/rancher/k3s/vendor/golang.org/x/net/http2.transportResponseBody.Read(0xc00929f1e0, 0xc007fad8ec, 0x4, 0x4, 0x0, 0x0, 0x0)
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/transport.go:2108 +0xaf
io.ReadAtLeast(0x7f4be12bf328, 0xc00929f1e0, 0xc007fad8ec, 0x4, 0x4, 0x4, 0x203002, 0x203002, 0x203002)
/usr/local/go/src/io/io.go:314 +0x87
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/framer.(*lengthDelimitedFrameReader).Read(0xc00c392660, 0xc00bfb8800, 0x400, 0x400, 0xc009d24501, 0x7f4be119f198, 0x38)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:76 +0x275
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc0099067d0, 0x0, 0x4c1f4c0, 0xc009ffeb00, 0x10, 0xc000197230, 0xc009d246b0, 0x15c775dad, 0xc0375c3fc42a9939)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0x89
github.com/rancher/k3s/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc00c392680, 0xc012b5cdcc, 0xc009d24730, 0xc00019e018, 0x4c77340, 0xc0070f1080, 0xc005cf3f80)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x6e
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc009ffeac0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:104 +0x147
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/watch.NewStreamWatcher
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:71 +0xbe

goroutine 2122 [chan receive, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc000416930, 0xc006e36060)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:628 +0x53
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:56 +0x2e
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc012b1d880, 0xc007a43840)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

goroutine 2123 [chan receive, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*controller).Run.func1(0xc00d683620, 0xc009f422d0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/controller.go:127 +0x34
created by github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*controller).Run
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/controller.go:126 +0xa5

goroutine 2124 [select, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc001e5a0d0, 0xc0375c3fc262ee8c, 0x15aafb2f6, 0x6ef39e0, 0x4c1f9c0, 0xc0077afa00, 0xc001d65b88, 0xc00323b920, 0xc00d683620, 0x0, …)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/reflector.go:451 +0x1a5
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc001e5a0d0, 0xc00d683620, 0x0, 0x0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/reflector.go:415 +0x657
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/reflector.go:209 +0x38
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc005635ee0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc001d65ee0, 0x4bb9900, 0xc0017be9b0, 0x1, 0xc00d683620)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run(0xc001e5a0d0, 0xc00d683620)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/reflector.go:208 +0x196
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:56 +0x2e
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc012b1d8d0, 0xc007a43900)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

goroutine 2593 [select, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2(0xc001e5a0d0, 0xc00d683620, 0xc007131aa0, 0xc00323b920)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/reflector.go:361 +0x16f
created by github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/reflector.go:355 +0x2a5

goroutine 2730 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc001c5ae00, 0xc000000000)
/usr/local/go/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc001c5adf0)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/rancher/k3s/vendor/golang.org/x/net/http2.(*pipe).Read(0xc001c5ade8, 0xc0076046d0, 0x4, 0x4, 0x0, 0x0, 0x0)
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/pipe.go:65 +0x97
github.com/rancher/k3s/vendor/golang.org/x/net/http2.transportResponseBody.Read(0xc001c5adc0, 0xc0076046d0, 0x4, 0x4, 0x0, 0x0, 0x0)
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/transport.go:2108 +0xaf
io.ReadAtLeast(0x7f4be12bf328, 0xc001c5adc0, 0xc0076046d0, 0x4, 0x4, 0x4, 0x203001, 0x203001, 0x203001)
/usr/local/go/src/io/io.go:314 +0x87
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/framer.(*lengthDelimitedFrameReader).Read(0xc00c1142e0, 0xc0030ff400, 0x400, 0x400, 0x1000, 0x7f4be1249208, 0x38)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:76 +0x275
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc0021f97c0, 0x0, 0x4c1f4c0, 0xc0077afa40, 0x4bb53e0, 0xc000197230, 0x803d7f, 0x80731f, 0xc008fc3180)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0x89
github.com/rancher/k3s/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc00c114300, 0xc009b6114c, 0x2, 0xc00019e018, 0xc001f83700, 0x18, 0x18)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x6e
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc0077afa00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:104 +0x147
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/watch.NewStreamWatcher
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:71 +0xbe

goroutine 2137 [chan receive, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/kube-aggregator/pkg/controllers/autoregister.(*autoRegisterController).Run(0xc0091cde80, 0x5, 0xc00d683620)
/go/src/github.com/rancher/k3s/vendor/k8s.io/kube-aggregator/pkg/controllers/autoregister/autoregister_controller.go:164 +0x2c8
github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app.createAggregatorServer.func1.1(0xc008f1fc80, 0xc0071cbd10, 0xc0091cde80, 0xc000900900, 0xc00d683620)
/go/src/github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app/aggregator.go:142 +0x76
created by github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app.createAggregatorServer.func1
/go/src/github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app/aggregator.go:135 +0xb6

goroutine 7236 [chan receive, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/etcd3.(*watchChan).startWatching(0xc01223b5f0, 0xc01aef2300)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:227 +0x21a
created by github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/etcd3.(*watchChan).run
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:143 +0x7f

goroutine 4423 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc0131e5380, 0xc000000000)
/usr/local/go/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc0131e5370)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/rancher/k3s/vendor/golang.org/x/net/http2.(*pipe).Read(0xc0131e5368, 0xc00fc36c6c, 0x4, 0x4, 0x0, 0x0, 0x0)
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/pipe.go:65 +0x97
github.com/rancher/k3s/vendor/golang.org/x/net/http2.transportResponseBody.Read(0xc0131e5340, 0xc00fc36c6c, 0x4, 0x4, 0x0, 0x0, 0x0)
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/transport.go:2108 +0xaf
io.ReadAtLeast(0x7f4be12bf328, 0xc0131e5340, 0xc00fc36c6c, 0x4, 0x4, 0x4, 0x203005, 0x203005, 0x203005)
/usr/local/go/src/io/io.go:314 +0x87
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/framer.(*lengthDelimitedFrameReader).Read(0xc016b5a260, 0xc016b5c000, 0x400, 0x400, 0x657571696e75, 0x7f4be05a4f60, 0x38)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:76 +0x275
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc004db0f50, 0x0, 0x4c1f4c0, 0xc01636cd80, 0x69222c2268637461, 0x72657571223a226e, 0x6572222c5d7d2279, 0x227365736e6f7073, 0x3a22303032227b3a)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0x89
github.com/rancher/k3s/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc016b5a280, 0x736970612e676b70, 0x31762e6174656d2e, 0x227375746174532e, 0x22313034222c7d7d, 0x7263736564227b3a, 0x3a226e6f69747069)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x6e
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc01636cd40)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:104 +0x147
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/watch.NewStreamWatcher
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:71 +0xbe

goroutine 4375 [select, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers.(*WatchServer).ServeHTTP(0xc015565680, 0x4c4f800, 0xc012a47e68, 0xc01637b500)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers/watch.go:213 +0xc88
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers.serveWatch(0x4c20880, 0xc006d18e70, 0xc001422820, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, …)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers/watch.go:138 +0xa76
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1.2()
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers/get.go:269 +0xd8
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/metrics.RecordLongRunning(0xc01637b500, 0xc014aaf340, 0x44eeb67, 0x9, 0xc0153cfea0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:340 +0x289
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1(0x4c4f800, 0xc012a47e68, 0xc01637b500)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers/get.go:268 +0xba7
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints.restfulListResource.func1(0xc010000a80, 0xc006d18e00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1157 +0x91
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc010000a80, 0xc006d18e00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:384 +0x282
github.com/rancher/k3s/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc001750000, 0x4c4f9c0, 0xc006d18d90, 0xc01637b500)
/go/src/github.com/rancher/k3s/vendor/github.com/emicklei/go-restful/container.go:288 +0xa84
github.com/rancher/k3s/vendor/github.com/emicklei/go-restful.(*Container).Dispatch(…)
/go/src/github.com/rancher/k3s/vendor/github.com/emicklei/go-restful/container.go:199
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x44ffbc5, 0xe, 0xc001750000, 0xc001428540, 0x4c4f9c0, 0xc006d18d90, 0xc01637b500)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/handler.go:146 +0x539
github.com/rancher/k3s/vendor/k8s.io/kube-aggregator/pkg/apiserver.(*proxyHandler).ServeHTTP(0xc0019ea9b0, 0x4c4f9c0, 0xc006d18d90, 0xc01637b500)
/go/src/github.com/rancher/k3s/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:121 +0x183
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc00bd94c40, 0x4c4f9c0, 0xc006d18d90, 0xc01637b500)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x3db
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc008fdf500, 0x4c4f9c0, 0xc006d18d90, 0xc01637b500)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8c
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x4503a32, 0xf, 0xc00334ec60, 0xc008fdf500, 0x4c4f9c0, 0xc006d18d90, 0xc01637b500)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x74d
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x4c4f9c0, 0xc006d18d90, 0xc01637b500)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x563
net/http.HandlerFunc.ServeHTTP(0xc008f1fe00, 0x4c4f9c0, 0xc006d18d90, 0xc01637b500)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x4c4f9c0, 0xc006d18d90, 0xc01637b500)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:149 +0x582
net/http.HandlerFunc.ServeHTTP(0xc0090037a0, 0x4c4f9c0, 0xc006d18d90, 0xc01637b500)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x4c4f9c0, 0xc006d18d90, 0xc01637b500)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x231d
net/http.HandlerFunc.ServeHTTP(0xc008f1fe40, 0x4c4f9c0, 0xc006d18d90, 0xc01637b500)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x4c4f9c0, 0xc006d18d90, 0xc01637b300)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:70 +0x672
net/http.HandlerFunc.ServeHTTP(0xc008faf1d0, 0x4c4f9c0, 0xc006d18d90, 0xc01637b300)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc008ff3ce0, 0x4c4f9c0, 0xc006d18d90, 0xc01637b300)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x42a
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters.WithWaitGroup.func1(0x4c4f9c0, 0xc006d18d90, 0xc01637b300)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:59 +0x137
net/http.HandlerFunc.ServeHTTP(0xc0090037d0, 0x4c4f9c0, 0xc006d18d90, 0xc01637b300)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1(0x4c4f9c0, 0xc006d18d90, 0xc01637b200)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39 +0x269
net/http.HandlerFunc.ServeHTTP(0xc009003800, 0x4c4f9c0, 0xc006d18d90, 0xc01637b200)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithWarningRecorder.func1(0x4c4f9c0, 0xc006d18d90, 0xc01637b100)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters/warning.go:35 +0x1a7
net/http.HandlerFunc.ServeHTTP(0xc008ff3d00, 0x4c4f9c0, 0xc006d18d90, 0xc01637b100)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1(0x4c4f9c0, 0xc006d18d90, 0xc01637b100)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters/cachecontrol.go:31 +0xa8
net/http.HandlerFunc.ServeHTTP(0xc008ff3d20, 0x4c4f9c0, 0xc006d18d90, 0xc01637b100)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/httplog.WithLogging.func1(0x4c41e00, 0xc012a47e30, 0xc01637ae00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:91 +0x2f2
net/http.HandlerFunc.ServeHTTP(0xc008ff3d40, 0x4c41e00, 0xc012a47e30, 0xc01637ae00)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1(0x4c41e00, 0xc012a47e30, 0xc01637ae00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:51 +0xe6
net/http.HandlerFunc.ServeHTTP(0xc008ff3d60, 0x4c41e00, 0xc012a47e30, 0xc01637ae00)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0xc009003830, 0x4c41e00, 0xc012a47e30, 0xc01637ae00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/handler.go:189 +0x51
net/http.serverHandler.ServeHTTP(0xc00c169880, 0x4c41e00, 0xc012a47e30, 0xc01637ae00)
/usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.initALPNRequest.ServeHTTP(0x4c77340, 0xc008b27470, 0xc00da00a80, 0xc00c169880, 0x4c41e00, 0xc012a47e30, 0xc01637ae00)
/usr/local/go/src/net/http/server.go:3415 +0x8d
github.com/rancher/k3s/vendor/golang.org/x/net/http2.(*serverConn).runHandler(0xc006418600, 0xc012a47e30, 0xc01637ae00, 0xc01636a0c0)
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/server.go:2152 +0x8b
created by github.com/rancher/k3s/vendor/golang.org/x/net/http2.(*serverConn).processHeaders
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/server.go:1882 +0x505

goroutine 2143 [IO wait]:
internal/poll.runtime_pollWait(0x7f4be19e36a8, 0x72, 0x4bc2600)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc00c356598, 0x72, 0x4bc2600, 0x6c7fe88, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(…)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00c356580, 0xc0097c2000, 0xdd8f, 0xdd8f, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
net.(*netFD).Read(0xc00c356580, 0xc0097c2000, 0xdd8f, 0xdd8f, 0x203000, 0x7ff3fb, 0xc00d684160)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc006189490, 0xc0097c2000, 0xdd8f, 0xdd8f, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:182 +0x8e
crypto/tls.(*atLeastReader).Read(0xc0032b0ea0, 0xc0097c2000, 0xdd8f, 0xdd8f, 0x267, 0xdd8a, 0xc0056f3708)
/usr/local/go/src/crypto/tls/conn.go:779 +0x62
bytes.(*Buffer).ReadFrom(0xc00d684280, 0x4bb5440, 0xc0032b0ea0, 0x40cf85, 0x3f50580, 0x43cecc0)
/usr/local/go/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc00d684000, 0x4bbe580, 0xc006189490, 0x5, 0xc006189490, 0x256)
/usr/local/go/src/crypto/tls/conn.go:801 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc00d684000, 0x0, 0x0, 0x94c83e)
/usr/local/go/src/crypto/tls/conn.go:608 +0x115
crypto/tls.(*Conn).readRecord(…)
/usr/local/go/src/crypto/tls/conn.go:576
crypto/tls.(*Conn).Read(0xc00d684000, 0xc000e90000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1252 +0x15f
bufio.(*Reader).Read(0xc006eec420, 0xc0077c2ab8, 0x9, 0x9, 0xc0056f3cc0, 0x47b39c, 0xc00ad4b640)
/usr/local/go/src/bufio/bufio.go:227 +0x222
io.ReadAtLeast(0x4bb5200, 0xc006eec420, 0xc0077c2ab8, 0x9, 0x9, 0x9, 0xc000210a00, 0xc00ad4b630, 0x46f31e0)
/usr/local/go/src/io/io.go:314 +0x87
io.ReadFull(…)
/usr/local/go/src/io/io.go:333
github.com/rancher/k3s/vendor/golang.org/x/net/http2.readFrameHeader(0xc0077c2ab8, 0x9, 0x9, 0x4bb5200, 0xc006eec420, 0x0, 0x0, 0xc000add440, 0xc0056f3dd0)
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/frame.go:237 +0x89
github.com/rancher/k3s/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0077c2a80, 0xc017add440, 0x0, 0x0, 0x0)
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/frame.go:492 +0xa5
github.com/rancher/k3s/vendor/golang.org/x/net/http2.(*clientConnReadLoop).run(0xc0056f3fa8, 0x0, 0x0)
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/transport.go:1819 +0xd8
github.com/rancher/k3s/vendor/golang.org/x/net/http2.(*ClientConn).readLoop(0xc00354af00)
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/transport.go:1741 +0x6f
created by github.com/rancher/k3s/vendor/golang.org/x/net/http2.(*Transport).newClientConn
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/transport.go:705 +0x6c5

goroutine 2685 [select, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/cacher.(*cacheWatcher).process(0xc0003fc460, 0x4c77300, 0xc0038c3d40, 0x6f2cc90, 0x0, 0x0, 0x110b)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/cacher/cacher.go:1376 +0x2a5
created by github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/cacher.(*Cacher).Watch
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/cacher/cacher.go:519 +0x585

goroutine 2129 [select, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0005dbf80)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:742 +0x157
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000416990, 0xc0081edfc0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

goroutine 2514 [chan receive, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1()
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:772 +0x5d
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc009160760)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc009224f60, 0x4bb9920, 0xc00846b1a0, 0x3e3d801, 0xc006e36420)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc009160760, 0x3b9aca00, 0x0, 0x63696877202c7401, 0xc006e36420)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).run(0xc0001a6180)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x95
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000416990, 0xc0081edfd0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

goroutine 2515 [select, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0001a6180)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:742 +0x157
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000416990, 0xc0081edfe0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

goroutine 2516 [chan receive, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1()
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:772 +0x5d
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc009161760)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc009225f60, 0x4bb9920, 0xc00846b1d0, 0x3e3d801, 0xc006e36480)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc009161760, 0x3b9aca00, 0x0, 0x1, 0xc006e36480)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).run(0xc0001a6280)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x95
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000416990, 0xc0081edff0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

goroutine 2517 [select, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0001a6280)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:742 +0x157
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000416990, 0xc00acd0010)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

goroutine 2518 [chan receive, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1()
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:772 +0x5d
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc009162760)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc009398f60, 0x4bb9920, 0xc00846b200, 0x3e3d801, 0xc006e364e0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc009162760, 0x3b9aca00, 0x0, 0x1, 0xc006e364e0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).run(0xc0001a6680)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x95
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000416990, 0xc00acd0020)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

goroutine 2519 [select, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0001a6680)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:742 +0x157
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000416990, 0xc00acd0040)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

goroutine 2520 [chan receive, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1()
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:772 +0x5d
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc009163760)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc009399f60, 0x4bb9920, 0xc00846b230, 0x3e3d801, 0xc006e36540)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc009163760, 0x3b9aca00, 0x0, 0x6176612065687401, 0xc006e36540)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).run(0xc0001a6b00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x95
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000416990, 0xc00acd0060)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

goroutine 2521 [select, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0001a6b00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:742 +0x157
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000416990, 0xc00acd0070)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

goroutine 2522 [chan receive, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1()
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:772 +0x5d
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00937e760)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00939af60, 0x4bb9920, 0xc00846b260, 0x3e3d801, 0xc006e365a0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00937e760, 0x3b9aca00, 0x0, 0x2279726575712201, 0xc006e365a0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).run(0xc0001a6c00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x95
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000416990, 0xc00acd0090)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

goroutine 2523 [select, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0001a6c00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:742 +0x157
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000416990, 0xc00acd00b0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

goroutine 2524 [chan receive, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1()
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:772 +0x5d
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00937f760)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00939bf60, 0x4bb9920, 0xc00846b290, 0x3e3d801, 0xc006e36600)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00937f760, 0x3b9aca00, 0x0, 0x222c222e676e6901, 0xc006e36600)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).run(0xc0001a6e00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x95
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000416990, 0xc00acd00d0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

goroutine 2525 [select, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0001a6e00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:742 +0x157
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000416990, 0xc00acd00e0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

goroutine 2526 [chan receive, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1()
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:772 +0x5d
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc009380760)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc009394f60, 0x4bb9920, 0xc00846b2c0, 0x3e3d801, 0xc006e36660)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc009380760, 0x3b9aca00, 0x0, 0x696e696665642f01, 0xc006e36660)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).run(0xc009332200)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x95
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000416990, 0xc00acd0100)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

goroutine 2527 [select, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).pop(0xc009332200)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:742 +0x157
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000416990, 0xc00acd0110)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

goroutine 4469 [select, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc0122fd450, 0xc0375c41c99e91bf, 0x33ec1a62d, 0x6ef39e0, 0x4c20980, 0xc00ed5a750, 0xc003df7b90, 0xc008c594a0, 0xc016f52ae0, 0x0, …)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/reflector.go:451 +0x1a5
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0122fd450, 0xc016f52ae0, 0x0, 0x0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/reflector.go:415 +0x657
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/cacher.(*Cacher).startCaching(0xc0136a0000, 0xc016f52ae0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/cacher/cacher.go:418 +0x110
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/cacher.NewCacherFromConfig.func1.1()
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/cacher/cacher.go:385 +0x65
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00d0e8788)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc003df7f88, 0x4bb9920, 0xc0100e1e30, 0x1, 0xc016f52ae0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00d0e8788, 0x3b9aca00, 0x0, 0x4c41e01, 0xc016f52ae0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/cacher.NewCacherFromConfig.func1(0xc0136a0000, 0xc016f52ae0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/cacher/cacher.go:382 +0xc7
created by github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/cacher.NewCacherFromConfig
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/cacher/cacher.go:379 +0x90e

goroutine 2530 [IO wait]:
internal/poll.runtime_pollWait(0x7f4be1924460, 0x72, 0x4bc2600)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc00c357298, 0x72, 0x4bc2600, 0x6c7fe88, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(…)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00c357280, 0xc00163a700, 0x60c, 0x60c, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
net.(*netFD).Read(0xc00c357280, 0xc00163a700, 0x60c, 0x60c, 0x203000, 0x7ff3fb, 0xc0095d1660)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc00d185030, 0xc00163a700, 0x60c, 0x60c, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:182 +0x8e
crypto/tls.(*atLeastReader).Read(0xc0032b1000, 0xc00163a700, 0x60c, 0x60c, 0x1e, 0x607, 0xc002129800)
/usr/local/go/src/crypto/tls/conn.go:779 +0x62
bytes.(*Buffer).ReadFrom(0xc0095d1780, 0x4bb5440, 0xc0032b1000, 0x40cf85, 0x3f50580, 0x43cecc0)
/usr/local/go/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc0095d1500, 0x4bbe580, 0xc00d185030, 0x5, 0xc00d185030, 0xd)
/usr/local/go/src/crypto/tls/conn.go:801 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc0095d1500, 0x0, 0x0, 0xc017124180)
/usr/local/go/src/crypto/tls/conn.go:608 +0x115
crypto/tls.(*Conn).readRecord(…)
/usr/local/go/src/crypto/tls/conn.go:576
crypto/tls.(*Conn).Read(0xc0095d1500, 0xc00aa16038, 0x9, 0x9, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1252 +0x15f
io.ReadAtLeast(0x7f4be12befc0, 0xc0095d1500, 0xc00aa16038, 0x9, 0x9, 0x9, 0x2, 0x1120bb0a95101, 0x0)
/usr/local/go/src/io/io.go:314 +0x87
io.ReadFull(…)
/usr/local/go/src/io/io.go:333
github.com/rancher/k3s/vendor/golang.org/x/net/http2.readFrameHeader(0xc00aa16038, 0x9, 0x9, 0x7f4be12befc0, 0xc0095d1500, 0x0, 0xc000000000, 0xc002129f28, 0xc006e36780)
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/frame.go:237 +0x89
github.com/rancher/k3s/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00aa16000, 0xc002129ee0, 0x2, 0x0, 0x1)
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/frame.go:492 +0xa5
github.com/rancher/k3s/vendor/golang.org/x/net/http2.(*serverConn).readFrames(0xc003812d80)
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/server.go:745 +0xa9
created by github.com/rancher/k3s/vendor/golang.org/x/net/http2.(*serverConn).serve
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/server.go:851 +0x354

goroutine 2927 [select]:
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers.(*WatchServer).ServeHTTP(0xc00c6a0280, 0x4c4f800, 0xc006189f10, 0xc00a1da200)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers/watch.go:213 +0xc88
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers.serveWatch(0x4c20880, 0xc00045d030, 0xc0033149c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, …)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers/watch.go:138 +0xa76
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1.2()
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers/get.go:269 +0xd8
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/metrics.RecordLongRunning(0xc00a1da200, 0xc0019d2000, 0x44eeb67, 0x9, 0xc00a2d80d0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:340 +0x289
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1(0x4c4f800, 0xc006189f10, 0xc00a1da200)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers/get.go:268 +0xba7
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints.restfulListResource.func1(0xc00308f170, 0xc00045cfc0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1157 +0x91
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc00308f170, 0xc00045cfc0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:384 +0x282
github.com/rancher/k3s/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc001750000, 0x4c4f9c0, 0xc00045cf50, 0xc00a1da200)
/go/src/github.com/rancher/k3s/vendor/github.com/emicklei/go-restful/container.go:288 +0xa84
github.com/rancher/k3s/vendor/github.com/emicklei/go-restful.(*Container).Dispatch(…)
/go/src/github.com/rancher/k3s/vendor/github.com/emicklei/go-restful/container.go:199
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x44ffbc5, 0xe, 0xc001750000, 0xc001428540, 0x4c4f9c0, 0xc00045cf50, 0xc00a1da200)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/handler.go:146 +0x539
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc006881380, 0x4c4f9c0, 0xc00045cf50, 0xc00a1da200)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x20f
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc008fdf500, 0x4c4f9c0, 0xc00045cf50, 0xc00a1da200)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8c
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x4503a32, 0xf, 0xc00334ec60, 0xc008fdf500, 0x4c4f9c0, 0xc00045cf50, 0xc00a1da200)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x74d
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x4c4f9c0, 0xc00045cf50, 0xc00a1da200)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x563
net/http.HandlerFunc.ServeHTTP(0xc008f1fe00, 0x4c4f9c0, 0xc00045cf50, 0xc00a1da200)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x4c4f9c0, 0xc00045cf50, 0xc00a1da200)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:149 +0x582
net/http.HandlerFunc.ServeHTTP(0xc0090037a0, 0x4c4f9c0, 0xc00045cf50, 0xc00a1da200)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x4c4f9c0, 0xc00045cf50, 0xc00a1da200)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x231d
net/http.HandlerFunc.ServeHTTP(0xc008f1fe40, 0x4c4f9c0, 0xc00045cf50, 0xc00a1da200)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x4c4f9c0, 0xc00045cf50, 0xc00a1da000)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:70 +0x672
net/http.HandlerFunc.ServeHTTP(0xc008faf1d0, 0x4c4f9c0, 0xc00045cf50, 0xc00a1da000)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc008ff3ce0, 0x4c4f9c0, 0xc00045cf50, 0xc00a1da000)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x42a
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters.WithWaitGroup.func1(0x4c4f9c0, 0xc00045cf50, 0xc00a1da000)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:59 +0x137
net/http.HandlerFunc.ServeHTTP(0xc0090037d0, 0x4c4f9c0, 0xc00045cf50, 0xc00a1da000)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1(0x4c4f9c0, 0xc00045cf50, 0xc009cfdf00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39 +0x269
net/http.HandlerFunc.ServeHTTP(0xc009003800, 0x4c4f9c0, 0xc00045cf50, 0xc009cfdf00)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithWarningRecorder.func1(0x4c4f9c0, 0xc00045cf50, 0xc009cfde00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters/warning.go:35 +0x1a7
net/http.HandlerFunc.ServeHTTP(0xc008ff3d00, 0x4c4f9c0, 0xc00045cf50, 0xc009cfde00)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1(0x4c4f9c0, 0xc00045cf50, 0xc009cfde00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters/cachecontrol.go:31 +0xa8
net/http.HandlerFunc.ServeHTTP(0xc008ff3d20, 0x4c4f9c0, 0xc00045cf50, 0xc009cfde00)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/httplog.WithLogging.func1(0x4c41e00, 0xc006189ef0, 0xc009cfdb00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:91 +0x2f2
net/http.HandlerFunc.ServeHTTP(0xc008ff3d40, 0x4c41e00, 0xc006189ef0, 0xc009cfdb00)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1(0x4c41e00, 0xc006189ef0, 0xc009cfdb00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:51 +0xe6
net/http.HandlerFunc.ServeHTTP(0xc008ff3d60, 0x4c41e00, 0xc006189ef0, 0xc009cfdb00)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0xc009003830, 0x4c41e00, 0xc006189ef0, 0xc009cfdb00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/handler.go:189 +0x51
net/http.serverHandler.ServeHTTP(0xc00c169880, 0x4c41e00, 0xc006189ef0, 0xc009cfdb00)
/usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.initALPNRequest.ServeHTTP(0x4c77340, 0xc0070f1080, 0xc0095d1500, 0xc00c169880, 0x4c41e00, 0xc006189ef0, 0xc009cfdb00)
/usr/local/go/src/net/http/server.go:3415 +0x8d
github.com/rancher/k3s/vendor/golang.org/x/net/http2.(*serverConn).runHandler(0xc003812d80, 0xc006189ef0, 0xc009cfdb00, 0xc00c3a8640)
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/server.go:2152 +0x8b
created by github.com/rancher/k3s/vendor/golang.org/x/net/http2.(*serverConn).processHeaders
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/server.go:1882 +0x505

goroutine 2974 [select, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/cacher.(*cacheWatcher).process(0xc000591ea0, 0x4c77300, 0xc00601ccc0, 0x6f2cc90, 0x0, 0x0, 0x110b)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/cacher/cacher.go:1376 +0x2a5
created by github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/cacher.(*Cacher).Watch
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/cacher/cacher.go:519 +0x585

goroutine 3077 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc001426950, 0x71206c6100000004)
/usr/local/go/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc001426940)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/rancher/k3s/vendor/k8s.io/client-go/util/workqueue.(*Type).Get(0xc00035cea0, 0x0, 0x0, 0x656a626f20646500)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/util/workqueue/queue.go:145 +0x89
github.com/rancher/k3s/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer.(*CRDFinalizer).processNextWorkItem(0xc001db90e0, 0x203000)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer/crd_finalizer.go:287 +0x66
github.com/rancher/k3s/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer.(*CRDFinalizer).runWorker(0xc001db90e0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer/crd_finalizer.go:281 +0x2b
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0093a6370)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0093a6370, 0x4bb9920, 0xc0062c2c30, 0x6465746165726301, 0xc00d683620)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0093a6370, 0x3b9aca00, 0x0, 0x6465757373692001, 0xc00d683620)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc0093a6370, 0x3b9aca00, 0xc00d683620)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by github.com/rancher/k3s/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer.(*CRDFinalizer).Run
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer/crd_finalizer.go:274 +0x1c7

goroutine 3210 [chan receive, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/etcd3.(*watchChan).startWatching(0xc010c5ca90, 0xc00fefb620)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:227 +0x21a
created by github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/etcd3.(*watchChan).run
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:143 +0x7f

goroutine 2696 [select, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.WaitFor(0xc006744fa0, 0xc00aed53c0, 0xc0088f9140, 0x0, 0x0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:539 +0x11d
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.PollUntil(0x45d964b800, 0xc00aed53c0, 0xc000182660, 0x0, 0x0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:492 +0xc5
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0x45d964b800, 0xc00aed53c0, 0xc000182660, 0xc0080b8778, 0xc0043dec01)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:511 +0xb3
created by github.com/rancher/k3s/vendor/k8s.io/client-go/transport.(*dynamicClientCert).Run
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/transport/cert_rotation.go:142 +0x213

goroutine 2942 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc007f2b900, 0x2)
/usr/local/go/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc007f2b8f0)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/rancher/k3s/vendor/golang.org/x/net/http2.(*pipe).Read(0xc007f2b8e8, 0xc023b287bc, 0x4, 0x4, 0x0, 0x0, 0x0)
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/pipe.go:65 +0x97
github.com/rancher/k3s/vendor/golang.org/x/net/http2.transportResponseBody.Read(0xc007f2b8c0, 0xc023b287bc, 0x4, 0x4, 0x0, 0x0, 0x0)
/go/src/github.com/rancher/k3s/vendor/golang.org/x/net/http2/transport.go:2108 +0xaf
io.ReadAtLeast(0x7f4be12bf328, 0xc007f2b8c0, 0xc023b287bc, 0x4, 0x4, 0x4, 0x203007, 0x203007, 0x203007)
/usr/local/go/src/io/io.go:314 +0x87
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/framer.(*lengthDelimitedFrameReader).Read(0xc00c285ca0, 0xc009bbfc00, 0x400, 0x400, 0xc007d14de0, 0x7f4be025e7e0, 0x38)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:76 +0x275
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc009964640, 0x0, 0x4c1f4c0, 0xc01c27bf40, 0x0, 0x0, 0x4665a0, 0xc002f51e00, 0xc007d14ec8)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0x89
github.com/rancher/k3s/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc00c285cc0, 0xc007d14f68, 0x8, 0x4c1dcc0, 0xc023b32b40, 0x0, 0x0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x6e
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc009c71a00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:104 +0x147
created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/watch.NewStreamWatcher
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:71 +0xbe

goroutine 2874 [chan receive, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x4c1fa00, 0xc00314b320, 0xc007a8cf60)
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/record/event.go:301 +0xaa
created by github.com/rancher/k3s/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/record/event.go:299 +0x6e

goroutine 2944 [select, 1 minutes]:
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers.(*WatchServer).ServeHTTP(0xc00c4e1580, 0x4c4f800, 0xc00161ba40, 0xc009c75a00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers/watch.go:213 +0xc88
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers.serveWatch(0x4c20880, 0xc0002b13b0, 0xc007481ba0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, …)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers/watch.go:138 +0xa76
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1.2()
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers/get.go:269 +0xd8
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/metrics.RecordLongRunning(0xc009c75a00, 0xc001b78b00, 0x44eeb67, 0x9, 0xc009e700d0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:340 +0x289
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1(0x4c4f800, 0xc00161ba40, 0xc009c75a00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/handlers/get.go:268 +0xba7
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints.restfulListResource.func1(0xc002ec5b00, 0xc0002b12d0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1157 +0x91
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc002ec5b00, 0xc0002b12d0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:384 +0x282
github.com/rancher/k3s/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc001750000, 0x4c4f9c0, 0xc0002b1260, 0xc009c75a00)
/go/src/github.com/rancher/k3s/vendor/github.com/emicklei/go-restful/container.go:288 +0xa84
github.com/rancher/k3s/vendor/github.com/emicklei/go-restful.(*Container).Dispatch(…)
/go/src/github.com/rancher/k3s/vendor/github.com/emicklei/go-restful/container.go:199
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x44ffbc5, 0xe, 0xc001750000, 0xc001428540, 0x4c4f9c0, 0xc0002b1260, 0xc009c75a00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/handler.go:146 +0x539
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc006881380, 0x4c4f9c0, 0xc0002b1260, 0xc009c75a00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x20f
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc008fdf500, 0x4c4f9c0, 0xc0002b1260, 0xc009c75a00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8c
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x4503a32, 0xf, 0xc00334ec60, 0xc008fdf500, 0x4c4f9c0, 0xc0002b1260, 0xc009c75a00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x74d
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x4c4f9c0, 0xc0002b1260, 0xc009c75a00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x563
net/http.HandlerFunc.ServeHTTP(0xc008f1fe00, 0x4c4f9c0, 0xc0002b1260, 0xc009c75a00)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x4c4f9c0, 0xc0002b1260, 0xc009c75a00)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:149 +0x582
net/http.HandlerFunc.ServeHTTP(0xc0090037a0, 0x4c4f9c0, 0xc0002b1260, 0xc009c75a00)
/usr/local/go/I0725 05:43:02.134453 32 event.go:291] “Event occurred” object=“kube-system/kube-controller-manager” kind=“Lease” apiVersion=“coordination.k8s.io/v1” type=“Normal” reason=“LeaderElection” message=“7a87998423f0_18a1acef-3b30-48bb-9ece-7f6c020e9f0e stopped leading”
I0725 05:43:02.417786 32 resource_quota_controller.go:291] Shutting down resource quota controller
2021-07-25 05:43:03.253155 W | etcdserver: read-only range request “key:”/registry/fleet.cattle.io/bundledeployments/cluster-fleet-local-local-1a3d67d0a899/" range_end:"/registry/fleet.cattle.io/bundledeployments/cluster-fleet-local-local-1a3d67d0a8990" limit:10000 " with result “error:context canceled” took too long (4.354190954s) to execute
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
2021-07-25 05:43:03.264542 W | etcdserver: read-only range request “key:”/registry/masterleases/172.17.0.2" " with result “error:context canceled” took too long (4.929239817s) to execute
W0725 05:43:03.269166 8 reflector.go:437] pkg/mod/github.com/rancher/client-go@v1.20.0-rancher.1/tools/cache/reflector.go:168: watch of *v1beta1.Ingress ended with: very short watch: pkg/mod/github.com/rancher/client-go@v1.20.0-rancher.1/tools/cache/reflector.go:168: Unexpected watch close - watch lasted less than a second and no items received
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
2021-07-25 05:43:03.271623 W | etcdserver: read-only range request “key:”/registry/limitranges/" range_end:"/registry/limitranges0" count_only:true " with result “error:context canceled” took too long (2.188841142s) to execute
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
2021-07-25 05:43:03.271697 W | etcdserver: read-only range request “key:”/registry/configmaps/fleet-system/fleet-agent-lock" " with result “error:context canceled” took too long (4.371901892s) to execute
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
2021-07-25 05:43:03.271760 W | etcdserver: read-only range request “key:”/registry/services/endpoints/kube-system/kube-scheduler" " with result “error:context canceled” took too long (2.368504747s) to execute
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
2021-07-25 05:43:03.271811 W | etcdserver: read-only range request “key:”/registry/minions/" range_end:"/registry/minions0" limit:10000 " with result “error:context canceled” took too long (4.367315035s) to execute
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
2021-07-25 05:43:03.271914 W | etcdserver: read-only range request “key:”/registry/serviceaccounts/" range_end:"/registry/serviceaccounts0" limit:10000 " with result “error:context canceled” took too long (4.367527434s) to execute
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
2021-07-25 05:43:03.272325 W | etcdserver: read-only range request “key:”/registry/leases/" range_end:"/registry/leases0" count_only:true " with result “error:context canceled” took too long (4.635671089s) to execute
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
2021-07-25 05:43:03.272533 W | etcdserver: read-only range request “key:”/registry/leases/kube-system/cloud-controller-manager" " with result “error:context canceled” took too long (4.369007725s) to execute
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
2021-07-25 05:43:03.272960 W | etcdserver: read-only range request “key:”/registry/events/" range_end:"/registry/events0" count_only:true " with result “error:context canceled” took too long (170.334137ms) to execute
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
2021-07-25 05:43:03.273058 W | etcdserver: read-only range request “key:”/registry/configmaps/fleet-system/gitjob" " with result “error:context canceled” took too long (4.766141785s) to execute
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
2021-07-25 05:43:03.273248 W | etcdserver: read-only range request “key:”/registry/configmaps/kube-system/k3s" " with result “error:context canceled” took too long (4.001771613s) to execute
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
2021-07-25 05:43:03.274663 W | etcdserver: read-only range request “key:”/registry/configmaps/kube-system/cattle-controllers" " with result “error:context canceled” took too long (4.000731456s) to execute
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
2021-07-25 05:43:03.274727 W | etcdserver: read-only range request “key:”/registry/configmaps/kube-system/rancher-controller-lock" " with result “error:context canceled” took too long (2.509634484s) to execute
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
2021-07-25 05:43:03.274779 W | etcdserver: read-only range request “key:”/registry/management.cattle.io/features/" range_end:"/registry/management.cattle.io/features0" count_only:true " with result “error:context canceled” took too long (1.255016314s) to execute
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
2021-07-25 05:43:03.274846 W | etcdserver: read-only range request “key:”/registry/project.cattle.io/pipelinesettings/" range_end:"/registry/project.cattle.io/pipelinesettings0" count_only:true " with result “error:context canceled” took too long (3.265262424s) to execute
WARNING: 2021/07/25 05:43:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”
W0725 05:43:03.278369 8 reflector.go:437] pkg/mod/github.com/rancher/client-go@v1.20.0-rancher.1/tools/cache/reflector.go:168: watch of *v1.APIService ended with: very short watch: pkg/mod/github.com/rancher/client-go@v1.20.0-rancher.1/tools/cache/reflector.go:168: Unexpected watch close - watch lasted less than a second and no items received

Jade_Baxter · July 27, 2021, 5:34am

anyone got any ideas?

Its seems to be getting worse… still wont start:
2021-07-27 05:32:27.698777 W | etcdserver: request “header:<ID:7587856077371510539 > txn:<compare:<target:MOD key:”/registry/management.cattle.io/authconfigs/adfs" mod_revision:0 > success:<request_put:<key:"/registry/management.cattle.io/authconfigs/adfs" value_size:490 >> failure:<>>" with result “size:7” took too long (733.395994ms) to execute
2021-07-27 05:32:27.699610 W | etcdserver: read-only range request “key:”/registry/leases/kube-system/cloud-controller-manager" " with result “range_response_count:1 size:501” took too long (667.812669ms) to execute
I0727 05:32:27.700334 33 trace.go:116] Trace[1978284593]: “GuaranteedUpdate etcd3” type:*core.Endpoints (started: 2021-07-27 05:32:27.143289664 +0000 UTC m=+41.418685107) (total time: 556.996077ms):
Trace[1978284593]: [556.958416ms] [554.380888ms] Transaction committed
I0727 05:32:27.700633 33 trace.go:116] Trace[139995136]: “Get” url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-07-27 05:32:26.990935193 +0000 UTC m=+41.266330601) (total time: 709.647527ms):
Trace[139995136]: [709.321218ms] [709.311143ms] About to write a response
I0727 05:32:27.700644 33 trace.go:116] Trace[659955287]: “GuaranteedUpdate etcd3” type:*core.Endpoints (started: 2021-07-27 05:32:27.061840511 +0000 UTC m=+41.337235979) (total time: 638.780267ms):
Trace[659955287]: [638.757192ms] [637.493054ms] Transaction committed
I0727 05:32:27.702418 33 trace.go:116] Trace[468509230]: “Update” url:/api/v1/namespaces/kube-system/endpoints/kube-scheduler,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-07-27 05:32:27.061400267 +0000 UTC m=+41.336795675) (total time: 640.984827ms):
Trace[468509230]: [640.905623ms] [640.587317ms] Object stored in database
I0727 05:32:27.700826 33 trace.go:116] Trace[190611735]: “Update” url:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,user-agent:k3s/v1.18.8+k3s1 (linux/amd64) kubernetes/6b59531/leader-election,client:127.0.0.1 (started: 2021-07-27 05:32:27.142868259 +0000 UTC m=+41.418263666) (total time: 557.926783ms):
Trace[190611735]: [557.53963ms] [557.202831ms] Object stored in database
I0727 05:32:27.704811 33 controller.go:606] quota admission added evaluator for: leases.coordination.k8s.io
2021/07/27 05:32:30 [ERROR] error syncing ‘rancher-charts’: handler helm-clusterrepo-ensure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721 reset --hard FETCH_HEAD error: exit status 128, detail: fatal: Unable to create ‘/var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721/.git/index.lock’: File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by ‘git commit’. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
, requeuing

Fraser_Goffin · July 31, 2021, 9:49am

It would probably help if you didn’t put such a large log into the post. Either take the time to pick out the entries which seem most relevant or, if you really have no idea what is or isn’t, then at least put it into a scrollable code window.

Jade_Baxter · July 31, 2021, 9:25pm

Thanks I will try that next time, I cant edit the post now.

Any ideas? I limping along quite badly, I have no control over rancher.
I have had to stop a rancher pod and start a docker container manually because I needed to make a change.

Jade_Baxter · July 31, 2021, 9:38pm

I have tried installing every version of rancher from 2.5.9 → 2.5.0.

and I cant get a single version to work!, tried installing docker 19.03 and tried all the versions of rancher.
Still no go - not sure whats going on here.

vincent · July 31, 2021, 11:34pm

The conntrack issue probably came from updating your kernel and hitting this.

The long log starts in the middle of a stack trace so that’s unhelpful to know what’s going on (which is usually at the beginning).

The rest looks like assorted things are unhappy(timeouts, etcd queries took too long) because storage is too slow or inadequate memory; What hardware this running on?

Jade_Baxter · August 1, 2021, 12:12am

Looks like disk load is from rancher… checking proxmox all of the disk load has references to rancher.

Jade

Jade_Baxter · August 2, 2021, 7:56am

Im running ubuntu 20.04.2 LTS

sudo apt install linux-image-5.4.0-77-generic linux-headers-5.4.0-77-generic
Rebooted grub to 5.4.0-77

2021-08-01 01:08:21.501181 W | etcdserver: read-only range request “key:”/registry/priorityclasses/" range_end:"/registry/priorityclasses0" limit:10000 " with result “range_response_count:2 size:798” took too long (451.74293ms) to execute
2021-08-01 01:08:21.501326 W | etcdserver: read-only range request “key:”/registry/configmaps" range_end:"/registry/configmapt" count_only:true " with result “range_response_count:0 size:9” took too long (725.996104ms) to execute
2021-08-01 01:08:21.501486 W | etcdserver: read-only range request “key:”/registry/storageclasses" range_end:"/registry/storageclasset" count_only:true " with result “range_response_count:0 size:7” took too long (448.364833ms) to execute
2021-08-01 01:08:21.501646 W | etcdserver: read-only range request “key:”/registry/persistentvolumeclaims" range_end:"/registry/persistentvolumeclaimt" count_only:true " with result “range_response_count:0 size:7” took too long (728.906715ms) to execute
2021-08-01 01:08:21.501789 W | etcdserver: read-only range request “key:”/registry/priorityclasses" range_end:"/registry/priorityclasset" count_only:true " with result “range_response_count:0 size:9” took too long (450.500341ms) to execute
2021-08-01 01:08:21.501865 W | etcdserver: read-only range request “key:”/registry/storageclasses/" range_end:"/registry/storageclasses0" limit:10000 " with result “range_response_count:0 size:7” took too long (449.348101ms) to execute
2021-08-01 01:08:21.502057 W | etcdserver: read-only range request “key:”/registry/priorityclasses/" range_end:"/registry/priorityclasses0" limit:10000 " with result “range_response_count:2 size:798” took too long (451.071064ms) to execute
2021-08-01 01:08:21.502165 W | etcdserver: read-only range request “key:”/registry/ingress" range_end:"/registry/ingrest" count_only:true " with result “range_response_count:0 size:7” took too long (490.851926ms) to execute
2021-08-01 01:08:21.502379 W | etcdserver: read-only range request “key:”/registry/events" range_end:"/registry/eventt" count_only:true " with result “range_response_count:0 size:10” took too long (393.117659ms) to execute
2021-08-01 01:08:21.505149 W | etcdserver: read-only range request “key:”/registry/secrets/" range_end:"/registry/secrets0" limit:10000 " with result “range_response_count:73 size:452420” took too long (735.087828ms) to execute
I0801 01:08:21.505715 32 trace.go:116] Trace[1997685313]: “List etcd3” key:/persistentvolumeclaims,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:20.772120925 +0000 UTC m=+0.495775051) (total time: 733.520931ms):
Trace[1997685313]: [733.520931ms] [733.520931ms] END
2021-08-01 01:08:21.505777 W | etcdserver: read-only range request “key:”/registry/persistentvolumes" range_end:"/registry/persistentvolumet" count_only:true " with result “range_response_count:0 size:7” took too long (731.064058ms) to execute
2021-08-01 01:08:21.505950 W | etcdserver: read-only range request “key:”/registry/priorityclasses" range_end:"/registry/priorityclasset" count_only:true " with result “range_response_count:0 size:9” took too long (453.286855ms) to execute
2021-08-01 01:08:21.506048 W | etcdserver: read-only range request “key:”/registry/runtimeclasses/" range_end:"/registry/runtimeclasses0" limit:10000 " with result “range_response_count:0 size:7” took too long (483.946564ms) to execute
2021-08-01 01:08:21.506083 W | etcdserver: read-only range request “key:”/registry/minions" range_end:"/registry/miniont" count_only:true " with result “range_response_count:0 size:9” took too long (716.647506ms) to execute
2021-08-01 01:08:21.506152 W | etcdserver: read-only range request “key:”/registry/ingress/" range_end:"/registry/ingress0" limit:10000 " with result “range_response_count:0 size:7” took too long (491.445266ms) to execute
2021-08-01 01:08:21.506176 W | etcdserver: read-only range request “key:”/registry/csidrivers/" range_end:"/registry/csidrivers0" limit:10000 " with result “range_response_count:0 size:7” took too long (420.367926ms) to execute
2021-08-01 01:08:21.506262 W | etcdserver: read-only range request “key:”/registry/ingress/" range_end:"/registry/ingress0" limit:10000 " with result “range_response_count:0 size:7” took too long (488.175542ms) to execute
2021-08-01 01:08:21.506290 W | etcdserver: read-only range request “key:”/registry/deployments/" range_end:"/registry/deployments0" limit:10000 " with result “range_response_count:6 size:20795” took too long (418.732013ms) to execute
2021-08-01 01:08:21.506350 W | etcdserver: read-only range request “key:”/registry/services/endpoints" range_end:"/registry/services/endpointt" count_only:true " with result “range_response_count:0 size:9” took too long (724.948753ms) to execute
2021-08-01 01:08:21.506422 W | etcdserver: read-only range request “key:”/registry/deployments" range_end:"/registry/deploymentt" count_only:true " with result “range_response_count:0 size:9” took too long (418.909671ms) to execute
2021-08-01 01:08:21.506489 W | etcdserver: read-only range request “key:”/registry/ingress" range_end:"/registry/ingrest" count_only:true " with result “range_response_count:0 size:7” took too long (488.31061ms) to execute
2021-08-01 01:08:21.506520 W | etcdserver: read-only range request “key:”/registry/statefulsets/" range_end:"/registry/statefulsets0" limit:10000 " with result “range_response_count:0 size:7” took too long (417.31457ms) to execute
2021-08-01 01:08:21.506544 W | etcdserver: read-only range request “key:”/registry/statefulsets" range_end:"/registry/statefulsett" count_only:true " with result “range_response_count:0 size:7” took too long (416.995938ms) to execute
2021-08-01 01:08:21.506566 W | etcdserver: read-only range request “key:”/registry/services/endpoints/" range_end:"/registry/services/endpoints0" limit:10000 " with result “range_response_count:7 size:3719” took too long (725.330105ms) to execute
2021-08-01 01:08:21.506632 W | etcdserver: read-only range request “key:”/registry/daemonsets/" range_end:"/registry/daemonsets0" limit:10000 " with result “range_response_count:0 size:7” took too long (403.504254ms) to execute
2021-08-01 01:08:21.506657 W | etcdserver: read-only range request “key:”/registry/daemonsets" range_end:"/registry/daemonsett" count_only:true " with result “range_response_count:0 size:7” took too long (403.220439ms) to execute
2021-08-01 01:08:21.506718 W | etcdserver: read-only range request “key:”/registry/endpointslices/" range_end:"/registry/endpointslices0" limit:10000 " with result “range_response_count:4 size:3352” took too long (493.297487ms) to execute
2021-08-01 01:08:21.506785 W | etcdserver: read-only range request “key:”/registry/replicasets" range_end:"/registry/replicasett" count_only:true " with result “range_response_count:0 size:9” took too long (401.383012ms) to execute
2021-08-01 01:08:21.506810 W | etcdserver: read-only range request “key:”/registry/runtimeclasses" range_end:"/registry/runtimeclasset" count_only:true " with result “range_response_count:0 size:7” took too long (484.502291ms) to execute
2021-08-01 01:08:21.506905 W | etcdserver: read-only range request “key:”/registry/namespaces" range_end:"/registry/namespacet" count_only:true " with result “range_response_count:0 size:9” took too long (725.86097ms) to execute
2021-08-01 01:08:21.506936 W | etcdserver: read-only range request “key:”/registry/endpointslices" range_end:"/registry/endpointslicet" count_only:true " with result “range_response_count:0 size:9” took too long (493.469838ms) to execute
2021-08-01 01:08:21.506994 W | etcdserver: read-only range request “key:”/registry/replicasets/" range_end:"/registry/replicasets0" limit:10000 " with result “range_response_count:7 size:22377” took too long (401.439556ms) to execute
2021-08-01 01:08:21.507062 W | etcdserver: read-only range request “key:”/registry/apiextensions.k8s.io/customresourcedefinitions/" range_end:"/registry/apiextensions.k8s.io/customresourcedefinitions0" limit:10000 " with result “range_response_count:94 size:371930” took too long (776.599833ms) to execute
2021-08-01 01:08:21.507131 W | etcdserver: read-only range request “key:”/registry/ingressclasses/" range_end:"/registry/ingressclasses0" limit:10000 " with result “range_response_count:0 size:7” took too long (486.117364ms) to execute
2021-08-01 01:08:21.507262 W | etcdserver: read-only range request “key:”/registry/controllerrevisions/" range_end:"/registry/controllerrevisions0" limit:10000 " with result “range_response_count:0 size:7” took too long (400.769336ms) to execute
2021-08-01 01:08:21.507317 W | etcdserver: read-only range request “key:”/registry/controllerrevisions" range_end:"/registry/controllerrevisiont" count_only:true " with result “range_response_count:0 size:7” took too long (400.594971ms) to execute
2021-08-01 01:08:21.507386 W | etcdserver: read-only range request “key:”/registry/apiextensions.k8s.io/customresourcedefinitions" range_end:"/registry/apiextensions.k8s.io/customresourcedefinitiont" count_only:true " with result “range_response_count:0 size:9” took too long (776.823169ms) to execute
2021-08-01 01:08:21.507454 W | etcdserver: read-only range request “key:”/registry/leases/" range_end:"/registry/leases0" limit:10000 " with result “range_response_count:4 size:2064” took too long (495.338967ms) to execute
2021-08-01 01:08:21.507484 W | etcdserver: read-only range request “key:”/registry/storageclasses/" range_end:"/registry/storageclasses0" limit:10000 " with result “range_response_count:0 size:7” took too long (445.266391ms) to execute
2021-08-01 01:08:21.507506 W | etcdserver: read-only range request “key:”/registry/validatingwebhookconfigurations/" range_end:"/registry/validatingwebhookconfigurations0" limit:10000 " with result “range_response_count:1 size:5738” took too long (399.569498ms) to execute
2021-08-01 01:08:21.507567 W | etcdserver: read-only range request “key:”/registry/leases" range_end:"/registry/leaset" count_only:true " with result “range_response_count:0 size:9” took too long (497.258316ms) to execute
2021-08-01 01:08:21.507596 W | etcdserver: read-only range request “key:”/registry/storageclasses" range_end:"/registry/storageclasset" count_only:true " with result “range_response_count:0 size:7” took too long (445.346082ms) to execute
2021-08-01 01:08:21.507656 W | etcdserver: read-only range request “key:”/registry/validatingwebhookconfigurations" range_end:"/registry/validatingwebhookconfigurationt" count_only:true " with result “range_response_count:0 size:9” took too long (399.209022ms) to execute
2021-08-01 01:08:21.507683 W | etcdserver: read-only range request “key:”/registry/leases" range_end:"/registry/leaset" count_only:true " with result “range_response_count:0 size:9” took too long (495.286908ms) to execute
2021-08-01 01:08:21.507749 W | etcdserver: read-only range request “key:”/registry/csidrivers" range_end:"/registry/csidrivert" count_only:true " with result “range_response_count:0 size:7” took too long (446.652329ms) to execute
2021-08-01 01:08:21.507855 W | etcdserver: read-only range request “key:”/registry/ingressclasses" range_end:"/registry/ingressclasset" count_only:true " with result “range_response_count:0 size:7” took too long (486.278265ms) to execute
2021-08-01 01:08:21.507889 W | etcdserver: read-only range request “key:”/registry/csidrivers/" range_end:"/registry/csidrivers0" limit:10000 " with result “range_response_count:0 size:7” took too long (446.717012ms) to execute
2021-08-01 01:08:21.507947 W | etcdserver: read-only range request “key:”/registry/mutatingwebhookconfigurations/" range_end:"/registry/mutatingwebhookconfigurations0" limit:10000 " with result “range_response_count:1 size:3815” took too long (398.374943ms) to execute
2021-08-01 01:08:21.508083 W | etcdserver: read-only range request “key:”/registry/volumeattachments" range_end:"/registry/volumeattachmentt" count_only:true " with result “range_response_count:0 size:7” took too long (424.203703ms) to execute
2021-08-01 01:08:21.508181 W | etcdserver: read-only range request “key:”/registry/csinodes" range_end:"/registry/csinodet" count_only:true " with result “range_response_count:0 size:9” took too long (447.84671ms) to execute
2021-08-01 01:08:21.508311 W | etcdserver: read-only range request “key:”/registry/mutatingwebhookconfigurations" range_end:"/registry/mutatingwebhookconfigurationt" count_only:true " with result “range_response_count:0 size:9” took too long (397.852941ms) to execute
2021-08-01 01:08:21.508425 W | etcdserver: read-only range request “key:”/registry/volumeattachments/" range_end:"/registry/volumeattachments0" limit:10000 " with result “range_response_count:0 size:7” took too long (424.122768ms) to execute
2021-08-01 01:08:21.508594 W | etcdserver: read-only range request “key:”/registry/certificatesigningrequests/" range_end:"/registry/certificatesigningrequests0" limit:10000 " with result “range_response_count:0 size:7” took too long (499.151662ms) to execute
2021-08-01 01:08:21.508697 W | etcdserver: read-only range request “key:”/registry/validatingwebhookconfigurations/" range_end:"/registry/validatingwebhookconfigurations0" limit:10000 " with result “range_response_count:1 size:5738” took too long (397.075376ms) to execute
2021-08-01 01:08:21.508822 W | etcdserver: read-only range request “key:”/registry/csinodes/" range_end:"/registry/csinodes0" limit:10000 " with result “range_response_count:1 size:458” took too long (422.424879ms) to execute
2021-08-01 01:08:21.508925 W | etcdserver: read-only range request “key:”/registry/certificatesigningrequests" range_end:"/registry/certificatesigningrequestt" count_only:true " with result “range_response_count:0 size:7” took too long (499.262941ms) to execute
2021-08-01 01:08:21.509024 W | etcdserver: read-only range request “key:”/registry/validatingwebhookconfigurations" range_end:"/registry/validatingwebhookconfigurationt" count_only:true " with result “range_response_count:0 size:9” took too long (396.701344ms) to execute
2021-08-01 01:08:21.509123 W | etcdserver: read-only range request “key:”/registry/cronjobs" range_end:"/registry/cronjobt" count_only:true " with result “range_response_count:0 size:7” took too long (500.908716ms) to execute
2021-08-01 01:08:21.509197 W | etcdserver: read-only range request “key:”/registry/csinodes" range_end:"/registry/csinodet" count_only:true " with result “range_response_count:0 size:9” took too long (422.411248ms) to execute
2021-08-01 01:08:21.509317 W | etcdserver: read-only range request “key:”/registry/mutatingwebhookconfigurations" range_end:"/registry/mutatingwebhookconfigurationt" count_only:true " with result “range_response_count:0 size:9” took too long (395.990761ms) to execute
2021-08-01 01:08:21.509388 W | etcdserver: read-only range request “key:”/registry/cronjobs/" range_end:"/registry/cronjobs0" limit:10000 " with result “range_response_count:0 size:7” took too long (501.023111ms) to execute
2021-08-01 01:08:21.509420 W | etcdserver: read-only range request “key:”/registry/csidrivers" range_end:"/registry/csidrivert" count_only:true " with result “range_response_count:0 size:7” took too long (421.778964ms) to execute
2021-08-01 01:08:21.509458 W | etcdserver: read-only range request “key:”/registry/jobs" range_end:"/registry/jobt" count_only:true " with result “range_response_count:0 size:7” took too long (501.997339ms) to execute
2021-08-01 01:08:21.509511 W | etcdserver: read-only range request “key:”/registry/mutatingwebhookconfigurations/" range_end:"/registry/mutatingwebhookconfigurations0" limit:10000 " with result “range_response_count:1 size:3815” took too long (395.517828ms) to execute
2021-08-01 01:08:21.509536 W | etcdserver: read-only range request “key:”/registry/horizontalpodautoscalers" range_end:"/registry/horizontalpodautoscalert" count_only:true " with result “range_response_count:0 size:7” took too long (503.602617ms) to execute
2021-08-01 01:08:21.509617 W | etcdserver: read-only range request “key:”/registry/csinodes/" range_end:"/registry/csinodes0" limit:10000 " with result “range_response_count:1 size:458” took too long (448.356584ms) to execute
2021-08-01 01:08:21.509643 W | etcdserver: read-only range request “key:”/registry/jobs/" range_end:"/registry/jobs0" limit:10000 " with result “range_response_count:0 size:7” took too long (502.496473ms) to execute
2021-08-01 01:08:21.509783 W | etcdserver: read-only range request “key:”/registry/volumeattachments/" range_end:"/registry/volumeattachments0" limit:10000 " with result “range_response_count:0 size:7” took too long (449.84202ms) to execute
2021-08-01 01:08:21.509833 W | etcdserver: read-only range request “key:”/registry/leases/" range_end:"/registry/leases0" limit:10000 " with result “range_response_count:4 size:2064” took too long (497.778847ms) to execute
2021-08-01 01:08:21.509860 W | etcdserver: read-only range request “key:”/registry/configmaps/" range_end:"/registry/configmaps0" limit:10000 " with result “range_response_count:13 size:31906” took too long (728.493243ms) to execute
2021-08-01 01:08:21.509898 W | etcdserver: read-only range request “key:”/registry/clusterroles" range_end:"/registry/clusterrolet" count_only:true " with result “range_response_count:0 size:9” took too long (459.223665ms) to execute
2021-08-01 01:08:21.509987 W | etcdserver: read-only range request “key:”/registry/resourcequotas" range_end:"/registry/resourcequotat" count_only:true " with result “range_response_count:0 size:7” took too long (734.931229ms) to execute
2021-08-01 01:08:21.510130 W | etcdserver: read-only range request “key:”/registry/rolebindings" range_end:"/registry/rolebindingt" count_only:true " with result “range_response_count:0 size:9” took too long (461.523106ms) to execute
2021-08-01 01:08:21.510195 W | etcdserver: read-only range request “key:”/registry/podtemplates/" range_end:"/registry/podtemplates0" limit:10000 " with result “range_response_count:0 size:7” took too long (739.903048ms) to execute
2021-08-01 01:08:21.510235 W | etcdserver: read-only range request “key:”/registry/resourcequotas/" range_end:"/registry/resourcequotas0" limit:10000 " with result “range_response_count:0 size:7” took too long (735.422869ms) to execute
2021-08-01 01:08:21.510333 W | etcdserver: read-only range request “key:”/registry/horizontalpodautoscalers/" range_end:"/registry/horizontalpodautoscalers0" limit:10000 " with result “range_response_count:0 size:7” took too long (504.353957ms) to execute
2021-08-01 01:08:21.510419 W | etcdserver: read-only range request “key:”/registry/horizontalpodautoscalers" range_end:"/registry/horizontalpodautoscalert" count_only:true " with result “range_response_count:0 size:7” took too long (505.899984ms) to execute
2021-08-01 01:08:21.510634 W | etcdserver: read-only range request “key:”/registry/clusterroles/" range_end:"/registry/clusterroles0" limit:10000 " with result “range_response_count:110 size:98467” took too long (459.702448ms) to execute
2021-08-01 01:08:21.510774 W | etcdserver: read-only range request “key:”/registry/persistentvolumes/" range_end:"/registry/persistentvolumes0" limit:10000 " with result “range_response_count:0 size:7” took too long (733.205784ms) to execute
2021-08-01 01:08:21.510911 W | etcdserver: read-only range request “key:”/registry/roles/" range_end:"/registry/roles0" limit:10000 " with result “range_response_count:46 size:44987” took too long (463.637357ms) to execute
2021-08-01 01:08:21.511011 W | etcdserver: read-only range request “key:”/registry/horizontalpodautoscalers/" range_end:"/registry/horizontalpodautoscalers0" limit:10000 " with result “range_response_count:0 size:7” took too long (506.697783ms) to execute
2021-08-01 01:08:21.511040 W | etcdserver: read-only range request “key:”/registry/limitranges/" range_end:"/registry/limitranges0" limit:10000 " with result “range_response_count:0 size:7” took too long (736.515722ms) to execute
2021-08-01 01:08:21.511083 W | etcdserver: read-only range request “key:”/registry/horizontalpodautoscalers/" range_end:"/registry/horizontalpodautoscalers0" limit:10000 " with result “range_response_count:0 size:7” took too long (509.062031ms) to execute
2021-08-01 01:08:21.511143 W | etcdserver: read-only range request “key:”/registry/limitranges" range_end:"/registry/limitranget" count_only:true " with result “range_response_count:0 size:7” took too long (736.648715ms) to execute
2021-08-01 01:08:21.511170 W | etcdserver: read-only range request “key:”/registry/roles" range_end:"/registry/rolet" count_only:true " with result “range_response_count:0 size:9” took too long (464.401665ms) to execute
2021-08-01 01:08:21.511216 W | etcdserver: read-only range request “key:”/registry/horizontalpodautoscalers" range_end:"/registry/horizontalpodautoscalert" count_only:true " with result “range_response_count:0 size:7” took too long (509.379041ms) to execute
2021-08-01 01:08:21.511254 W | etcdserver: read-only range request “key:”/registry/clusterrolebindings/" range_end:"/registry/clusterrolebindings0" limit:10000 " with result “range_response_count:93 size:76427” took too long (459.334821ms) to execute
2021-08-01 01:08:21.511279 W | etcdserver: read-only range request “key:”/registry/secrets" range_end:"/registry/secrett" count_only:true " with result “range_response_count:0 size:9” took too long (734.668957ms) to execute
2021-08-01 01:08:21.511304 W | etcdserver: read-only range request “key:”/registry/clusterrolebindings" range_end:"/registry/clusterrolebindingt" count_only:true " with result “range_response_count:0 size:9” took too long (459.545588ms) to execute
2021-08-01 01:08:21.511325 W | etcdserver: read-only range request “key:”/registry/podsecuritypolicy/" range_end:"/registry/podsecuritypolicy0" limit:10000 " with result “range_response_count:0 size:7” took too long (474.429329ms) to execute
2021-08-01 01:08:21.511378 W | etcdserver: read-only range request “key:”/registry/rolebindings" range_end:"/registry/rolebindingt" count_only:true " with result “range_response_count:0 size:9” took too long (467.645897ms) to execute
2021-08-01 01:08:21.511413 W | etcdserver: read-only range request “key:”/registry/services/specs" range_end:"/registry/services/spect" count_only:true " with result “range_response_count:0 size:9” took too long (710.130359ms) to execute
2021-08-01 01:08:21.512389 W | etcdserver: read-only range request “key:”/registry/clusterroles/" range_end:"/registry/clusterroles0" limit:10000 " with result “range_response_count:110 size:98467” took too long (470.735439ms) to execute
2021-08-01 01:08:21.512448 W | etcdserver: read-only range request “key:”/registry/services/specs/" range_end:"/registry/services/specs0" limit:10000 " with result “range_response_count:4 size:4725” took too long (711.241014ms) to execute
2021-08-01 01:08:21.512518 W | etcdserver: read-only range request “key:”/registry/roles" range_end:"/registry/rolet" count_only:true " with result “range_response_count:0 size:9” took too long (473.484054ms) to execute
2021-08-01 01:08:21.512607 W | etcdserver: read-only range request “key:”/registry/apiextensions.k8s.io/customresourcedefinitions" range_end:"/registry/apiextensions.k8s.io/customresourcedefinitiont" count_only:true " with result “range_response_count:0 size:9” took too long (778.828747ms) to execute
2021-08-01 01:08:21.512896 W | etcdserver: read-only range request “key:”/registry/roles/" range_end:"/registry/roles0" limit:10000 " with result “range_response_count:46 size:44987” took too long (473.913048ms) to execute
2021-08-01 01:08:21.512954 W | etcdserver: read-only range request “key:”/registry/rolebindings/" range_end:"/registry/rolebindings0" limit:10000 " with result “range_response_count:97 size:78255” took too long (463.8741ms) to execute
2021-08-01 01:08:21.513100 W | etcdserver: read-only range request “key:”/registry/podsecuritypolicy" range_end:"/registry/podsecuritypolicz" count_only:true " with result “range_response_count:0 size:7” took too long (475.604939ms) to execute
2021-08-01 01:08:21.513167 W | etcdserver: read-only range request “key:”/registry/serviceaccounts" range_end:"/registry/serviceaccountt" count_only:true " with result “range_response_count:0 size:9” took too long (713.083563ms) to execute
2021-08-01 01:08:21.513426 W | etcdserver: read-only range request “key:”/registry/pods" range_end:"/registry/podt" count_only:true " with result “range_response_count:0 size:9” took too long (714.654546ms) to execute
2021-08-01 01:08:21.513618 W | etcdserver: read-only range request “key:”/registry/poddisruptionbudgets" range_end:"/registry/poddisruptionbudgett" count_only:true " with result “range_response_count:0 size:7” took too long (476.839945ms) to execute
2021-08-01 01:08:21.513666 W | etcdserver: read-only range request “key:”/registry/serviceaccounts/" range_end:"/registry/serviceaccounts0" limit:10000 " with result “range_response_count:56 size:22828” took too long (712.781091ms) to execute
2021-08-01 01:08:21.513694 W | etcdserver: read-only range request “key:”/registry/networkpolicies" range_end:"/registry/networkpoliciet" count_only:true " with result “range_response_count:0 size:9” took too long (493.521656ms) to execute
2021-08-01 01:08:21.513923 W | etcdserver: read-only range request “key:”/registry/pods/" range_end:"/registry/pods0" limit:10000 " with result “range_response_count:6 size:20082” took too long (715.335291ms) to execute
2021-08-01 01:08:21.514080 W | etcdserver: read-only range request “key:”/registry/networkpolicies/" range_end:"/registry/networkpolicies0" limit:10000 " with result “range_response_count:1 size:1443” took too long (493.806765ms) to execute
2021-08-01 01:08:21.514153 W | etcdserver: read-only range request “key:”/registry/poddisruptionbudgets/" range_end:"/registry/poddisruptionbudgets0" limit:10000 " with result “range_response_count:0 size:7” took too long (478.774899ms) to execute
2021-08-01 01:08:21.514264 W | etcdserver: read-only range request “key:”/registry/minions/" range_end:"/registry/minions0" limit:10000 " with result “range_response_count:1 size:5788” took too long (720.819814ms) to execute
W0801 01:08:21.514301 32 genericapiserver.go:409] Skipping API apps/v1beta2 because it has no resources.
W0801 01:08:21.514335 32 genericapiserver.go:409] Skipping API apps/v1beta1 because it has no resources.
2021-08-01 01:08:21.515636 W | etcdserver: read-only range request “key:”/registry/apiextensions.k8s.io/customresourcedefinitions/" range_end:"/registry/apiextensions.k8s.io/customresourcedefinitions0" limit:10000 " with result “range_response_count:94 size:371930” took too long (780.77398ms) to execute
2021-08-01 01:08:21.515928 W | etcdserver: read-only range request “key:”/registry/podtemplates" range_end:"/registry/podtemplatet" count_only:true " with result “range_response_count:0 size:7” took too long (743.656729ms) to execute
2021-08-01 01:08:21.516182 W | etcdserver: read-only range request “key:”/registry/clusterrolebindings" range_end:"/registry/clusterrolebindingt" count_only:true " with result “range_response_count:0 size:9” took too long (468.743329ms) to execute
2021-08-01 01:08:21.516239 W | etcdserver: read-only range request “key:”/registry/controllers/" range_end:"/registry/controllers0" limit:10000 " with result “range_response_count:0 size:7” took too long (708.903849ms) to execute
2021-08-01 01:08:21.516266 W | etcdserver: read-only range request “key:”/registry/controllers" range_end:"/registry/controllert" count_only:true " with result “range_response_count:0 size:7” took too long (709.003333ms) to execute
2021-08-01 01:08:21.516477 W | etcdserver: read-only range request “key:”/registry/namespaces/" range_end:"/registry/namespaces0" limit:10000 " with result “range_response_count:21 size:22755” took too long (731.131615ms) to execute
I0801 01:08:21.516644 32 trace.go:116] Trace[1419425125]: “List etcd3” key:/services/endpoints,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:20.777288618 +0000 UTC m=+0.500942743) (total time: 739.329398ms):
Trace[1419425125]: [739.329398ms] [739.329398ms] END
2021-08-01 01:08:21.516975 W | etcdserver: read-only range request “key:”/registry/rolebindings/" range_end:"/registry/rolebindings0" limit:10000 " with result “range_response_count:97 size:78255” took too long (471.266148ms) to execute
2021-08-01 01:08:21.517173 W | etcdserver: read-only range request “key:”/registry/clusterrolebindings/" range_end:"/registry/clusterrolebindings0" limit:10000 " with result “range_response_count:93 size:76427” took too long (470.38681ms) to execute
I0801 01:08:21.517275 32 trace.go:116] Trace[1840079445]: “List etcd3” key:/ingress,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:21.014009824 +0000 UTC m=+0.737663958) (total time: 503.2111ms):
Trace[1840079445]: [503.2111ms] [503.2111ms] END
I0801 01:08:21.517293 32 trace.go:116] Trace[1812478350]: “List etcd3” key:/endpointslices,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:21.008824313 +0000 UTC m=+0.732478442) (total time: 508.443966ms):
Trace[1812478350]: [508.443966ms] [508.443966ms] END
2021-08-01 01:08:21.517292 W | etcdserver: read-only range request “key:”/registry/clusterroles" range_end:"/registry/clusterrolet" count_only:true " with result “range_response_count:0 size:9” took too long (471.394105ms) to execute
2021-08-01 01:08:21.517977 W | etcdserver: read-only range request “key:”/registry/events" range_end:"/registry/eventt" count_only:true " with result “range_response_count:0 size:10” took too long (745.1094ms) to execute
I0801 01:08:21.518064 32 trace.go:116] Trace[603322748]: “List etcd3” key:/ingress,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:21.010871728 +0000 UTC m=+0.734525856) (total time: 507.146455ms):
Trace[603322748]: [507.146455ms] [507.146455ms] END
I0801 01:08:21.518486 32 trace.go:116] Trace[1684298317]: “List etcd3” key:/runtimeclasses,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:21.01821384 +0000 UTC m=+0.741867969) (total time: 500.215703ms):
Trace[1684298317]: [500.215703ms] [500.215703ms] END
I0801 01:08:21.522313 32 trace.go:116] Trace[1860115605]: “List etcd3” key:/ingressclasses,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:21.016658561 +0000 UTC m=+0.740312700) (total time: 505.591195ms):
Trace[1860115605]: [505.591195ms] [505.591195ms] END
I0801 01:08:21.524418 32 trace.go:116] Trace[854958597]: “List etcd3” key:/leases,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:21.007508814 +0000 UTC m=+0.731163015) (total time: 516.872511ms):
Trace[854958597]: [516.872511ms] [516.872511ms] END
I0801 01:08:21.525816 32 trace.go:116] Trace[1958944791]: “List etcd3” key:/networkpolicies,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:21.012532516 +0000 UTC m=+0.736186646) (total time: 513.259201ms):
Trace[1958944791]: [513.259201ms] [513.259201ms] END
I0801 01:08:21.528747 32 trace.go:116] Trace[1945680244]: “List etcd3” key:/services/specs,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:20.794421354 +0000 UTC m=+0.518075483) (total time: 734.289935ms):
Trace[1945680244]: [734.289935ms] [734.289935ms] END
I0801 01:08:21.522822 32 trace.go:116] Trace[1255389899]: “List etcd3” key:/leases,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:21.00550178 +0000 UTC m=+0.729155921) (total time: 517.280034ms):
Trace[1255389899]: [517.280034ms] [517.280034ms] END
I0801 01:08:21.530712 32 trace.go:116] Trace[999109965]: “List etcd3” key:/horizontalpodautoscalers,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:20.995055212 +0000 UTC m=+0.718709339) (total time: 535.627726ms):
Trace[999109965]: [535.627726ms] [535.627726ms] END
I0801 01:08:21.531087 32 trace.go:116] Trace[1096563069]: “List etcd3” key:/pods,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:20.791017293 +0000 UTC m=+0.514671422) (total time: 740.008603ms):
Trace[1096563069]: [740.008603ms] [740.008603ms] END
I0801 01:08:21.532694 32 trace.go:116] Trace[1257380704]: “List etcd3” key:/podsecuritypolicy,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:21.030366879 +0000 UTC m=+0.754021017) (total time: 502.300684ms):
Trace[1257380704]: [502.300684ms] [502.300684ms] END
I0801 01:08:21.533153 32 trace.go:116] Trace[1721213223]: “List etcd3” key:/limitranges,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:20.767578212 +0000 UTC m=+0.491232344) (total time: 765.531337ms):
Trace[1721213223]: [765.531337ms] [765.531337ms] END
I0801 01:08:21.533580 32 trace.go:116] Trace[1509753612]: “List etcd3” key:/persistentvolumes,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:20.770963528 +0000 UTC m=+0.494617657) (total time: 762.572528ms):
Trace[1509753612]: [762.572528ms] [762.572528ms] END
I0801 01:08:21.533978 32 trace.go:116] Trace[1773808004]: “List etcd3” key:/roles,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:21.031786367 +0000 UTC m=+0.755440495) (total time: 502.164709ms):
Trace[1773808004]: [502.164709ms] [502.164709ms] END
I0801 01:08:21.534265 32 trace.go:116] Trace[304994788]: “List etcd3” key:/jobs,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:21.000923498 +0000 UTC m=+0.724577655) (total time: 533.318277ms):
Trace[304994788]: [533.318277ms] [533.318277ms] END
I0801 01:08:21.534638 32 trace.go:116] Trace[731017292]: “List etcd3” key:/podtemplates,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:20.763596433 +0000 UTC m=+0.487250553) (total time: 770.986455ms):
Trace[731017292]: [770.986455ms] [770.986455ms] END
I0801 01:08:21.534907 32 trace.go:116] Trace[748527273]: “List etcd3” key:/serviceaccounts,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:20.793010542 +0000 UTC m=+0.516664679) (total time: 741.871783ms):
Trace[748527273]: [741.871783ms] [741.871783ms] END
I0801 01:08:21.533982 32 trace.go:116] Trace[9814009]: “List etcd3” key:/horizontalpodautoscalers,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:20.999270904 +0000 UTC m=+0.722925032) (total time: 534.648299ms):
Trace[9814009]: [534.648299ms] [534.648299ms] END
I0801 01:08:21.531959 32 trace.go:116] Trace[1907229663]: “List etcd3” key:/configmaps,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:20.774882555 +0000 UTC m=+0.498536680) (total time: 757.052966ms):
Trace[1907229663]: [757.052966ms] [757.052966ms] END
I0801 01:08:21.535995 32 trace.go:116] Trace[1435680197]: “List etcd3” key:/controllers,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:20.798668735 +0000 UTC m=+0.522322863) (total time: 737.284094ms):
Trace[1435680197]: [737.284094ms] [737.284094ms] END
I0801 01:08:21.533215 32 trace.go:116] Trace[924631372]: “List etcd3” key:/resourcequotas,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:20.768368417 +0000 UTC m=+0.492022546) (total time: 764.803402ms):
Trace[924631372]: [764.803402ms] [764.803402ms] END
I0801 01:08:21.536975 32 trace.go:116] Trace[203560028]: “List etcd3” key:/cronjobs,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:21.002177458 +0000 UTC m=+0.725831609) (total time: 534.772668ms):
Trace[203560028]: [534.772668ms] [534.772668ms] END
I0801 01:08:21.532355 32 trace.go:116] Trace[160756547]: “List etcd3” key:/horizontalpodautoscalers,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:20.997398218 +0000 UTC m=+0.721052344) (total time: 534.914886ms):
Trace[160756547]: [534.914886ms] [534.914886ms] END
I0801 01:08:21.531856 32 trace.go:116] Trace[1733051045]: “List etcd3” key:/minions,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:20.785780476 +0000 UTC m=+0.509434723) (total time: 746.047645ms):
Trace[1733051045]: [746.047645ms] [746.047645ms] END
I0801 01:08:21.536061 32 trace.go:116] Trace[945335156]: “List etcd3” key:/namespaces,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:20.776162081 +0000 UTC m=+0.499816210) (total time: 759.874631ms):
Trace[945335156]: [759.874631ms] [759.874631ms] END
I0801 01:08:21.540281 32 trace.go:116] Trace[1167875465]: “List etcd3” key:/certificatesigningrequests,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:21.003709996 +0000 UTC m=+0.727364144) (total time: 536.522368ms):
Trace[1167875465]: [536.522368ms] [536.522368ms] END
I0801 01:08:21.545691 32 trace.go:116] Trace[1741186006]: “List etcd3” key:/clusterrolebindings,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:21.045039563 +0000 UTC m=+0.768693691) (total time: 500.614383ms):
Trace[1741186006]: [500.614383ms] [500.614383ms] END
I0801 01:08:21.548033 32 trace.go:116] Trace[117123178]: “List etcd3” key:/rolebindings,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:21.03358916 +0000 UTC m=+0.757243299) (total time: 514.402174ms):
Trace[117123178]: [514.402174ms] [514.402174ms] END
I0801 01:08:21.548435 32 trace.go:116] Trace[892472922]: “List etcd3” key:/rolebindings,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:21.041820472 +0000 UTC m=+0.765474600) (total time: 506.513307ms):
Trace[892472922]: [506.513307ms] [506.513307ms] END
I0801 01:08:21.559963 32 trace.go:116] Trace[174577996]: “List etcd3” key:/clusterroles,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:21.034782604 +0000 UTC m=+0.758436733) (total time: 525.135459ms):
Trace[174577996]: [525.135459ms] [525.135459ms] END
I0801 01:08:21.577369 32 trace.go:116] Trace[123850854]: “List etcd3” key:/clusterroles,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:21.044266263 +0000 UTC m=+0.767920388) (total time: 533.044631ms):
Trace[123850854]: [533.044631ms] [533.044631ms] END
I0801 01:08:21.578451 32 trace.go:116] Trace[782133060]: “List etcd3” key:/clusterrolebindings,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:21.038359206 +0000 UTC m=+0.762013336) (total time: 540.030559ms):
Trace[782133060]: [540.030559ms] [540.030559ms] END
I0801 01:08:21.588926 32 trace.go:116] Trace[1125580203]: “List etcd3” key:/secrets,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:20.769664158 +0000 UTC m=+0.493318301) (total time: 819.196956ms):
Trace[1125580203]: [819.196956ms] [819.196956ms] END
I0801 01:08:21.591370 32 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0801 01:08:21.591469 32 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0801 01:08:21.637444 32 trace.go:116] Trace[1015852569]: “List etcd3” key:/apiextensions.k8s.io/customresourcedefinitions,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:20.72645544 +0000 UTC m=+0.450109565) (total time: 910.952528ms):
Trace[1015852569]: [910.952528ms] [910.952528ms] END
I0801 01:08:21.638063 32 trace.go:116] Trace[1051982404]: “List etcd3” key:/apiextensions.k8s.io/customresourcedefinitions,resourceVersion:,limit:10000,continue: (started: 2021-08-01 01:08:20.725220352 +0000 UTC m=+0.448874476) (total time: 912.798184ms):
Trace[1051982404]: [912.798184ms] [912.798184ms] END
2021/08/01 01:08:22 [INFO] Waiting for server to become available: the server is currently unable to handle the request
2021/08/01 01:08:24 [INFO] Waiting for server to become available: the server is currently unable to handle the request
Reply

Share
Bookmark
Flag
Reply
You will receive notifications because you created this topic.
Suggested Topics
Topic	Replies	Views	Activity
Fresh installation of Rancher Server fails - Container keeps restarting 
Rancher 2.x
3	7	1h
Liveness probe failed connection refused 
Rancher 2.x
0	4	15h
Prometheus monitoring outside of cluster
Rancher 2.x
0	104	Feb 23
Air-gapped rke provisioning still uses docker.io
Rancher 2.x
3	134	Mar 9
After add catalog rancher cannot start
Rancher 2.x
0	62	27d
There are 2 new topics remaining, or browse other topics in 
Rancher 2.x

Jade_Baxter · August 6, 2021, 2:09am

I was using a consumer grade SSD for VMs that rancher was on, ever since rancher started playing up I got really bad IO waits. I think I reached the limit of small writes to the SSD.

Upgraded to enterprise SSD and all my rancher problems have disappeared!
Moved the consumer grade SSD to storage pool… all is good in the world. Just had to spend 750$ per 2TB for new SSD

Thanks for the help
Jade

Topic		Replies	Views
Issues accessing rancher UI on PROD RancherOS	3	7996	December 9, 2021
Fresh installation of Rancher Server fails - Container keeps restarting Rancher	4	3385	August 2, 2021
Cannot manage cluster using the ui Rancher	1	775	November 26, 2018
Rancher 2.7 on Docker fails start after server reboot Rancher	5	6241	March 7, 2023
Rebooted vm running rancherserver and k3s won't start / no UI Rancher	2	4996	March 31, 2021

Rancher admin ui never starts

Related topics