A lot of Issues after Server-Downtime

patrik · October 31, 2021, 6:58pm

Hello,

Yesterday our server went down. Since then we have massive problems with our cluster.

kubectl get pods --all-namespaces
NAMESPACE                  NAME                                                     READY   STATUS                  RESTARTS   AGE
default                    svclb-ingress-nginx-controller-n9vvp                     0/2     Unknown                 0          85d
default                    ingress-nginx-controller-68649d49b8-g25vh                0/1     Unknown                 0          84d
test                       order-management-api-postgresql-postgresql-0             0/1     Unknown                 0          2d16h
k3os-system                system-upgrade-controller-8bf4f84c4-fsnbq                0/1     NodeAffinity            0          84d
test                       document-management-api-mongodb-5f46494477-nqzbl         0/1     Unknown                 0          15d
test                       notification-management-api-mongodb-8458655556-m86xx     0/1     Unknown                 0          15d
test                       document-management-api-7b6664966f-2grtb                 0/1     Unknown                 0          15d
test                       order-management-api-axon-5c4f84d668-njwzh               0/1     Unknown                 0          11d
keycloak                   keycloak-postgresql-0                                    0/1     Unknown                 0          2d16h
test                       order-management-api-mongodb-774f8549cd-cnvd6            0/1     Unknown                 0          11d
test                       user-management-api-mongodb-7cfcc95977-4jr6q             0/1     Unknown                 0          2d16h
cattle-resources-system    rancher-backup-d9d984f74-h6cn7                           0/1     Unknown                 0          15d
k3os-system                system-upgrade-controller-8bf4f84c4-gbsw7                0/1     Pending                 0          108m
default                    busybox                                                  0/1     Pending                 0          89m
default                    dnsutils                                                 0/1     Pending                 0          49m
longhorn-system            engine-image-ei-0f7c4304-d5l2p                           1/1     Running                 0          35d
longhorn-system            csi-snapshotter-69f8bc8dcf-cjvnk                         1/1     Running                 0          35d
cattle-monitoring-system   pushprox-k3s-server-client-qwhvq                         1/1     Running                 0          32d
longhorn-system            csi-snapshotter-69f8bc8dcf-wcfbz                         1/1     Running                 0          35d
fleet-system               gitjob-568c57cfb9-c4mww                                  1/1     Running                 0          84d
longhorn-system            csi-attacher-75588bff58-rr57b                            1/1     Running                 0          35d
cattle-monitoring-system   pushprox-k3s-server-proxy-6946957f87-r75k8               1/1     Running                 0          32d
cattle-monitoring-system   alertmanager-rancher-monitoring-alertmanager-0           2/2     Running                 0          32d
longhorn-system            csi-attacher-75588bff58-pdmt7                            1/1     Running                 0          35d
longhorn-system            csi-provisioner-6968cf94f9-fb894                         1/1     Running                 0          35d
cert-manager               cert-manager-cainjector-595fc4d55-ljzn2                  1/1     Running                 0          84d
cert-manager               cert-manager-58bbf5798-cs5x2                             1/1     Running                 0          84d
rancher-operator-system    rancher-operator-595ddc6db9-w4jsj                        1/1     Running                 0          84d
longhorn-system            csi-provisioner-6968cf94f9-wlrql                         1/1     Running                 0          35d
kube-system                local-path-provisioner-5ff76fc89d-d4pdw                  1/1     Running                 0          84d
cattle-system              rancher-webhook-6cccfd96b5-v5v24                         1/1     Running                 0          84d
longhorn-system            csi-snapshotter-69f8bc8dcf-dh85b                         1/1     Running                 0          35d
longhorn-system            csi-attacher-75588bff58-rlk42                            1/1     Running                 0          35d
longhorn-system            csi-resizer-5c88bfd4cf-mqwj4                             1/1     Running                 0          35d
longhorn-system            longhorn-driver-deployer-5fc857649c-mgp8m                0/1     Init:0/1                0          35d
cattle-monitoring-system   rancher-monitoring-operator-9779869d-6r4mh               1/1     Running                 0          32d
longhorn-system            csi-resizer-5c88bfd4cf-tfqf2                             1/1     Running                 0          35d
fleet-system               fleet-agent-d59db746-h7nqt                               1/1     Running                 0          84d
fleet-system               fleet-controller-79554fcbf5-9zw2d                        1/1     Running                 0          84d
test                       notification-management-api-bd7d67c5d-wfs4q              1/1     Running                 0          34d
longhorn-system            csi-provisioner-6968cf94f9-vgxks                         1/1     Running                 0          35d
kube-system                metrics-server-86cbb8457f-tkq6f                          1/1     Running                 0          84d
cert-manager               cert-manager-webhook-7dc86765b4-kdsvv                    1/1     Running                 0          84d
longhorn-system            engine-image-ei-a5a44787-7q78r                           1/1     Running                 0          85d
cattle-monitoring-system   rancher-monitoring-kube-state-metrics-5f56fc44ff-vfp4c   1/1     Running                 0          32d
test                       planung-ui-959b9d47c-x8djr                               1/1     Running                 0          2d6h
longhorn-system            csi-resizer-5c88bfd4cf-ghxfz                             1/1     Running                 0          35d
longhorn-system            longhorn-manager-ll268                                   1/1     Running                 0          35d
cattle-monitoring-system   rancher-monitoring-prometheus-node-exporter-rmz2w        1/1     Running                 0          32d
kube-system                coredns-7448499f4d-plr2q                                 1/1     Running                 0          84d
cattle-monitoring-system   rancher-monitoring-grafana-6c9976ccc6-njtt6              3/3     Running                 0          32d
cattle-monitoring-system   rancher-monitoring-prometheus-adapter-58d5b96d85-wwh45   1/1     Running                 0          32d
cattle-system              rancher-6d8598477d-h8cf2                                 1/1     Running                 0          84d
cattle-monitoring-system   prometheus-rancher-monitoring-prometheus-0               2/3     CrashLoopBackOff        9          32d
keycloak                   keycloak-0                                               0/1     Init:CrashLoopBackOff   9          14d
longhorn-system            longhorn-ui-64b96684bc-8jbmk                             0/1     CrashLoopBackOff        9          35d
longhorn-system            longhorn-csi-plugin-gxqb2                                1/2     CrashLoopBackOff        9          35d
test                       user-management-api-76c74dc97b-bxjtk                     0/1     CrashLoopBackOff        9          24d
test                       order-managament-api-7d654d8d84-jvnf5                    0/1     Error                   9          11d

The log of the rancher system

2021/10/31 18:52:20 [ERROR] error syncing 'p-8mlxr/akomljen-charts': handler projectCatalog: Timeout in HTTP GET to [https://raw.githubusercontent.com/komljen/helm-charts/master/charts/index.yaml], did not respond in 30s, requeuing
2021/10/31 18:52:45 [ERROR] error syncing 'p-g7r2v/bitnami': handler projectCatalog: Timeout in HTTP GET to [https://charts.bitnami.com/bitnami/index.yaml], did not respond in 30s, requeuing
2021/10/31 18:53:00 [ERROR] error syncing 'p-g7r2v/wg-access-server': handler projectCatalog: Error in HTTP GET to [https://place1.github.io/wg-access-server/index.yaml], error: Get "https://place1.github.io/wg-access-server/index.yaml": dial tcp: lookup place1.github.io on 10.43.0.10:53: read udp 10.42.0.84:47076->10.43.0.10:53: read: connection refused, requeuing
2021/10/31 18:53:15 [ERROR] error syncing 'p-8mlxr/elastic': handler projectCatalog: Error in HTTP GET to [https://helm.elastic.co/index.yaml], error: Get "https://helm.elastic.co/index.yaml": dial tcp: lookup helm.elastic.co on 10.43.0.10:53: read udp 10.42.0.84:44027->10.43.0.10:53: read: connection refused, requeuing
2021/10/31 18:53:40 [ERROR] error syncing 'p-8mlxr/akomljen-charts': handler projectCatalog: Timeout in HTTP GET to [https://raw.githubusercontent.com/komljen/helm-charts/master/charts/index.yaml], did not respond in 30s, requeuing
2021/10/31 18:53:45 [ERROR] error syncing 'p-g7r2v/bitnami': handler projectCatalog: Error in HTTP GET to [https://charts.bitnami.com/bitnami/index.yaml], error: Get "https://charts.bitnami.com/bitnami/index.yaml": dial tcp: lookup charts.bitnami.com on 10.43.0.10:53: read udp 10.42.0.84:42944->10.43.0.10:53: read: connection refused, requeuing

If i want to read the logs of the nginx i get the following message

kubectl -n default logs -f ingress-nginx-controller-68649d49b8-g25vh
unable to retrieve container logs for containerd://4c98ecfe29a501755196e6bc6dd1268baaa604aac3bfe8ddc92990369eb22152v

I have no idea what i can do I’m running in K3OS i hope some one can help me…

patrik · November 1, 2021, 6:44am

I have tested the DNS. I think i have a problem with the DNS service. If i type

kubectl run -it --rm --restart=Never busybox2 --image=busybox:1.28 -- nslookup kubernetes.default
pod "busybox2" deleted
error: timed out waiting for the condition

Then i found this page:

But i cannot run the dnsutils pod. it does not come up…

kubectl get pods dnsutils
NAME       READY   STATUS    RESTARTS   AGE
dnsutils   0/1     Pending   0          12h

Has anyone a idea whats going wrong with my envirement?

wcoateRR · November 1, 2021, 3:02pm

I didn’t notice coredns in your pod list either. I’m new to Kubernetes & Rancher but from what I understand that’s the default DNS provider for Kubernetes clusters these days.

mattmattox · November 1, 2021, 3:33pm

@patrik How many master nodes are part of this cluster?
Also, it looks like scheduler might be broken. If you looks at /var/log/syslog or /var/log/messages do you see anything about k3s on the master node(s)?

patrik · November 1, 2021, 8:29pm

Hey,
i only have one node in my cluster.
at “/var/log/messages” i get a lot of logs of trying to connect to the vps isnt it?

Nov  1 20:15:19 upspot-cluster auth.err sshd[26421]: error: kex_exchange_identification: Connection closed by remote host
Nov  1 20:15:19 upspot-cluster auth.info sshd[26421]: Connection closed by 221.10.243.56 port 42366
Nov  1 20:16:28 upspot-cluster auth.info sshd[28661]: User root not allowed because account is locked
Nov  1 20:16:28 upspot-cluster auth.err sshd[28661]: error: Could not get shadow information for NOUSER
Nov  1 20:16:28 upspot-cluster auth.info sshd[28661]: Failed password for invalid user root from 182.61.16.130 port 35854 ssh2
Nov  1 20:16:29 upspot-cluster auth.info sshd[28661]: Received disconnect from 182.61.16.130 port 35854:11: Bye Bye [preauth]
Nov  1 20:16:29 upspot-cluster auth.info sshd[28661]: Disconnected from invalid user root 182.61.16.130 port 35854 [preauth]
Nov  1 20:17:35 upspot-cluster auth.info sshd[30866]: User root not allowed because account is locked
Nov  1 20:17:35 upspot-cluster auth.err sshd[30866]: error: Could not get shadow information for NOUSER
Nov  1 20:17:35 upspot-cluster auth.info sshd[30866]: Failed password for invalid user root from 81.70.180.77 port 34382 ssh2
Nov  1 20:17:35 upspot-cluster auth.info sshd[30866]: Received disconnect from 81.70.180.77 port 34382:11: Bye Bye [preauth]
Nov  1 20:17:35 upspot-cluster auth.info sshd[30866]: Disconnected from invalid user root 81.70.180.77 port 34382 [preauth]
Nov  1 20:18:03 upspot-cluster authpriv.notice sudo:  rancher : TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/usr/bin/su
Nov  1 20:18:03 upspot-cluster auth.notice su: + pts/0 root:root
Nov  1 20:20:01 upspot-cluster auth.info sshd[4335]: User root not allowed because account is locked
Nov  1 20:20:01 upspot-cluster auth.err sshd[4335]: error: Could not get shadow information for NOUSER
Nov  1 20:20:01 upspot-cluster auth.info sshd[4335]: Failed password for invalid user root from 179.183.27.51 port 40419 ssh2
Nov  1 20:20:02 upspot-cluster auth.info sshd[4335]: Received disconnect from 179.183.27.51 port 40419:11: Bye Bye [preauth]
Nov  1 20:20:02 upspot-cluster auth.info sshd[4335]: Disconnected from invalid user root 179.183.27.51 port 40419 [preauth]

there are a k3s-service.log too

tail -f -n 500 k3s-service.log
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.261)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.262)
Trace[725048495]: ---"Transaction committed" 19ms (20:21:00.282)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.282)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.283)
Trace[725048495]: ---"Transaction committed" 21ms (20:21:00.304)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.305)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.306)
Trace[725048495]: ---"Transaction committed" 22ms (20:21:00.328)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.328)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.329)
Trace[725048495]: ---"Transaction committed" 20ms (20:21:00.349)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.350)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.350)
Trace[725048495]: ---"Transaction committed" 23ms (20:21:00.374)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.375)
Trace[725048495]: ---"Transaction prepared" 1ms (20:21:00.376)
Trace[725048495]: ---"Transaction committed" 20ms (20:21:00.396)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.397)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.398)
Trace[725048495]: ---"Transaction committed" 20ms (20:21:00.419)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.419)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.420)
Trace[725048495]: ---"Transaction committed" 20ms (20:21:00.440)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.441)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.442)
Trace[725048495]: ---"Transaction committed" 19ms (20:21:00.461)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.462)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.463)
Trace[725048495]: ---"Transaction committed" 20ms (20:21:00.483)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.484)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.484)
Trace[725048495]: ---"Transaction committed" 20ms (20:21:00.504)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.505)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.506)
Trace[725048495]: ---"Transaction committed" 23ms (20:21:00.529)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.529)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.530)
Trace[725048495]: ---"Transaction committed" 21ms (20:21:00.552)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.552)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.553)
Trace[725048495]: ---"Transaction committed" 25ms (20:21:00.578)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.579)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.579)
Trace[725048495]: ---"Transaction committed" 19ms (20:21:00.599)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.599)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.600)
Trace[725048495]: ---"Transaction committed" 19ms (20:21:00.619)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.620)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.620)
Trace[725048495]: ---"Transaction committed" 20ms (20:21:00.641)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.641)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.642)
Trace[725048495]: ---"Transaction committed" 20ms (20:21:00.663)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.664)
Trace[725048495]: ---"Transaction prepared" 1ms (20:21:00.665)
Trace[725048495]: ---"Transaction committed" 26ms (20:21:00.691)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.691)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.692)
Trace[725048495]: ---"Transaction committed" 20ms (20:21:00.713)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.714)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.714)
Trace[725048495]: ---"Transaction committed" 26ms (20:21:00.741)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.741)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.742)
Trace[725048495]: ---"Transaction committed" 40ms (20:21:00.783)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.783)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.784)
Trace[725048495]: ---"Transaction committed" 24ms (20:21:00.809)
Trace[725048495]: ---"Retry value restored" 0ms (20:21:00.809)
Trace[725048495]: ---"Transaction prepared" 0ms (20:21:00.810)
Trace[725048495]: [33.999613243s] [33.999613243s] END
I1101 20:21:57.095689    2586 trace.go:205] Trace[663015526]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:40d23106-1428-4ccf-ad7a-75ad1d5e6a2e (01-Nov-2021 20:21:56.063) (total time: 1032ms):
Trace[663015526]: [1.032472825s] [1.032472825s] END
I1101 20:21:57.095731    2586 trace.go:205] Trace[1812652461]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:56.052) (total time: 1042ms):
Trace[1812652461]: [1.042945917s] [1.042945917s] END
E1101 20:21:57.095761    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:21:57.195670    2586 trace.go:205] Trace[546051022]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:8fe639b4-791d-40e8-8dc7-159cea99aab2 (01-Nov-2021 20:21:56.187) (total time: 1007ms):
Trace[546051022]: [1.007990561s] [1.007990561s] END
I1101 20:21:57.195707    2586 trace.go:205] Trace[1269843309]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:56.176) (total time: 1019ms):
Trace[1269843309]: [1.019633192s] [1.019633192s] END
E1101 20:21:57.195726    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:21:57.511666    2586 trace.go:205] Trace[256958804]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1alpha2,object-count:1,UID:e9c10299-81cc-4cd4-9371-ab701fd73d5b (01-Nov-2021 20:21:56.507) (total time: 1004ms):
Trace[256958804]: [1.004407612s] [1.004407612s] END
I1101 20:21:57.511711    2586 trace.go:205] Trace[1464123698]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:56.495) (total time: 1015ms):
Trace[1464123698]: [1.01579443s] [1.01579443s] END
E1101 20:21:57.511742    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1alpha2, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:21:57.671778    2586 trace.go:205] Trace[136202121]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:404eea53-8eeb-4540-a2d4-16745e209a09 (01-Nov-2021 20:21:56.667) (total time: 1004ms):
Trace[136202121]: [1.004397155s] [1.004397155s] END
I1101 20:21:57.671842    2586 trace.go:205] Trace[2141596514]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:56.656) (total time: 1015ms):
Trace[2141596514]: [1.015422224s] [1.015422224s] END
E1101 20:21:57.671885    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:21:57.767686    2586 trace.go:205] Trace[2116705438]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:9096a29a-25ce-4e2f-91a2-d86c33f26563 (01-Nov-2021 20:21:56.737) (total time: 1029ms):
Trace[2116705438]: [1.02995764s] [1.02995764s] END
I1101 20:21:57.767739    2586 trace.go:205] Trace[129292633]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:56.720) (total time: 1046ms):
Trace[129292633]: [1.046954567s] [1.046954567s] END
E1101 20:21:57.767770    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:21:57.899787    2586 trace.go:205] Trace[34497927]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:c96bdb35-b3b6-4f11-9b8b-76478ca3d8ee (01-Nov-2021 20:21:56.891) (total time: 1007ms):
Trace[34497927]: [1.007787666s] [1.007787666s] END
I1101 20:21:57.899837    2586 trace.go:205] Trace[2045424304]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:56.880) (total time: 1018ms):
Trace[2045424304]: [1.0188944s] [1.0188944s] END
E1101 20:21:57.899872    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:21:57.959690    2586 trace.go:205] Trace[69151470]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1beta1,object-count:1,UID:e54fb4a8-de4b-4487-b1a7-42470db49e88 (01-Nov-2021 20:21:56.956) (total time: 1002ms):
Trace[69151470]: [1.002944011s] [1.002944011s] END
I1101 20:21:57.959729    2586 trace.go:205] Trace[2026659538]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:56.943) (total time: 1015ms):
Trace[2026659538]: [1.015841525s] [1.015841525s] END
E1101 20:21:57.959761    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1beta1, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:21:58.023718    2586 trace.go:205] Trace[1649327103]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:12d07c3d-a851-418c-b5c9-44d95278eced (01-Nov-2021 20:21:57.019) (total time: 1003ms):
Trace[1649327103]: [1.003747827s] [1.003747827s] END
I1101 20:21:58.023764    2586 trace.go:205] Trace[244351459]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:57.008) (total time: 1015ms):
Trace[244351459]: [1.01526964s] [1.01526964s] END
E1101 20:21:58.023797    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:21:58.087696    2586 trace.go:205] Trace[1963563795]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:16348604-192c-40ca-8875-81ddde544637 (01-Nov-2021 20:21:57.056) (total time: 1030ms):
Trace[1963563795]: [1.030766751s] [1.030766751s] END
I1101 20:21:58.087739    2586 trace.go:205] Trace[90995043]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:57.040) (total time: 1047ms):
Trace[90995043]: [1.047441937s] [1.047441937s] END
E1101 20:21:58.087769    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:21:58.091706    2586 trace.go:205] Trace[989420331]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:4b19243a-046c-4202-b5a5-345ba9c043f5 (01-Nov-2021 20:21:57.058) (total time: 1033ms):
Trace[989420331]: [1.033007706s] [1.033007706s] END
I1101 20:21:58.091757    2586 trace.go:205] Trace[2062317025]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:57.040) (total time: 1051ms):
Trace[2062317025]: [1.051442913s] [1.051442913s] END
E1101 20:21:58.091784    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:21:58.279755    2586 trace.go:205] Trace[1852203447]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:a7b01b1a-46eb-46f6-9230-d16424299cb0 (01-Nov-2021 20:21:57.277) (total time: 1002ms):
Trace[1852203447]: [1.002678411s] [1.002678411s] END
I1101 20:21:58.279806    2586 trace.go:205] Trace[1149031870]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:57.264) (total time: 1014ms):
Trace[1149031870]: [1.014873262s] [1.014873262s] END
E1101 20:21:58.279841    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:21:58.663807    2586 trace.go:205] Trace[1111536816]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1alpha3,object-count:1,UID:7775fe25-aabc-4963-b943-b3da3e02af96 (01-Nov-2021 20:21:57.659) (total time: 1004ms):
Trace[1111536816]: [1.004600615s] [1.004600615s] END
I1101 20:21:58.663856    2586 trace.go:205] Trace[609901295]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:57.647) (total time: 1015ms):
Trace[609901295]: [1.015951344s] [1.015951344s] END
E1101 20:21:58.663892    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1alpha3, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:21:58.720367    2586 trace.go:205] Trace[2055414302]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:fb4df6a9-9a66-4226-a247-7da810de5b04 (01-Nov-2021 20:21:57.690) (total time: 1029ms):
Trace[2055414302]: [1.029882192s] [1.029882192s] END
I1101 20:21:58.720417    2586 trace.go:205] Trace[2043368690]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:57.680) (total time: 1040ms):
Trace[2043368690]: [1.04031709s] [1.04031709s] END
E1101 20:21:58.720438    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:21:58.780555    2586 trace.go:205] Trace[172144300]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:fff576e9-72ae-4a31-b4a0-643c135b6b76 (01-Nov-2021 20:21:57.755) (total time: 1025ms):
Trace[172144300]: [1.025304973s] [1.025304973s] END
I1101 20:21:58.780609    2586 trace.go:205] Trace[1048271773]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:57.744) (total time: 1036ms):
Trace[1048271773]: [1.036373565s] [1.036373565s] END
E1101 20:21:58.780637    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:21:58.823736    2586 trace.go:205] Trace[1261558026]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:141e323f-1231-47c0-8339-c0c10095d17e (01-Nov-2021 20:21:57.791) (total time: 1032ms):
Trace[1261558026]: [1.032164084s] [1.032164084s] END
I1101 20:21:58.823787    2586 trace.go:205] Trace[1528342616]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:57.780) (total time: 1043ms):
Trace[1528342616]: [1.043172021s] [1.043172021s] END
E1101 20:21:58.823820    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:21:59.115760    2586 trace.go:205] Trace[738756538]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:ee5d323d-5e03-4e72-8161-849917a5a4a9 (01-Nov-2021 20:21:58.107) (total time: 1007ms):
Trace[738756538]: [1.007889847s] [1.007889847s] END
I1101 20:21:59.115833    2586 trace.go:205] Trace[180615457]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:58.096) (total time: 1018ms):
Trace[180615457]: [1.018927141s] [1.018927141s] END
E1101 20:21:59.115869    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:21:59.239743    2586 trace.go:205] Trace[1655708793]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:5c47b9bf-bdb5-472f-b3e0-9420a215e5ab (01-Nov-2021 20:21:58.208) (total time: 1031ms):
Trace[1655708793]: [1.031158621s] [1.031158621s] END
I1101 20:21:59.239797    2586 trace.go:205] Trace[370591090]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:58.196) (total time: 1043ms):
Trace[370591090]: [1.043586404s] [1.043586404s] END
E1101 20:21:59.239829    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:21:59.390501    2586 scope.go:111] "RemoveContainer" containerID="8c4870b6058f013c1e70f1acf6d6a16c6f17ff1e293cc4775fd32306b0e52fcd"
I1101 20:21:59.527647    2586 trace.go:205] Trace[1381221765]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1alpha2,object-count:1,UID:a2b6b4ba-9b36-4ef5-99b3-d747a0072b2d (01-Nov-2021 20:21:58.526) (total time: 1000ms):
Trace[1381221765]: [1.000833507s] [1.000833507s] END
I1101 20:21:59.527706    2586 trace.go:205] Trace[1130807068]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:58.511) (total time: 1015ms):
Trace[1130807068]: [1.015812332s] [1.015812332s] END
E1101 20:21:59.527745    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1alpha2, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:21:59.687696    2586 trace.go:205] Trace[1236093261]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:ed6ee14b-c77c-464b-ae81-a84c70a41e3c (01-Nov-2021 20:21:58.683) (total time: 1004ms):
Trace[1236093261]: [1.004609447s] [1.004609447s] END
I1101 20:21:59.687752    2586 trace.go:205] Trace[1100548330]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:58.672) (total time: 1015ms):
Trace[1100548330]: [1.015443174s] [1.015443174s] END
E1101 20:21:59.687784    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:21:59.812329    2586 scope.go:111] "RemoveContainer" containerID="8c4870b6058f013c1e70f1acf6d6a16c6f17ff1e293cc4775fd32306b0e52fcd"
I1101 20:21:59.812987    2586 scope.go:111] "RemoveContainer" containerID="8c8871b752d107604212967bb7871a4216d1f641d99c003e5e5f7f1506a42dda"
E1101 20:21:59.813742    2586 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"prometheus\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=prometheus pod=prometheus-rancher-monitoring-prometheus-0_cattle-monitoring-system(42b6fd88-42c1-4545-bf30-556b7623645a)\"" pod="cattle-monitoring-system/prometheus-rancher-monitoring-prometheus-0" podUID=42b6fd88-42c1-4545-bf30-556b7623645a
I1101 20:21:59.815652    2586 trace.go:205] Trace[1649107762]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:b44bf213-e097-487e-a21f-4097345f2dfa (01-Nov-2021 20:21:58.796) (total time: 1019ms):
Trace[1649107762]: [1.019143811s] [1.019143811s] END
I1101 20:21:59.815687    2586 trace.go:205] Trace[1104733199]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:58.780) (total time: 1035ms):
Trace[1104733199]: [1.035298057s] [1.035298057s] END
E1101 20:21:59.815720    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:21:59.915683    2586 trace.go:205] Trace[1639473565]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:03f2dbb0-b4e3-4195-a1de-554917e65452 (01-Nov-2021 20:21:58.910) (total time: 1004ms):
Trace[1639473565]: [1.004753633s] [1.004753633s] END
I1101 20:21:59.915750    2586 trace.go:205] Trace[1840898144]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:58.900) (total time: 1015ms):
Trace[1840898144]: [1.015671782s] [1.015671782s] END
E1101 20:21:59.915788    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:00.007699    2586 trace.go:205] Trace[81252209]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1beta1,object-count:1,UID:4085dbc2-8499-4261-815b-6c5fffa2dc34 (01-Nov-2021 20:21:58.983) (total time: 1023ms):
Trace[81252209]: [1.023901576s] [1.023901576s] END
I1101 20:22:00.007742    2586 trace.go:205] Trace[2042202901]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:58.959) (total time: 1047ms):
Trace[2042202901]: [1.047887061s] [1.047887061s] END
E1101 20:22:00.007772    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1beta1, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:00.039740    2586 trace.go:205] Trace[722835624]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:465d1a71-a5e3-4405-998e-b74566c7a5cb (01-Nov-2021 20:21:59.036) (total time: 1003ms):
Trace[722835624]: [1.003647977s] [1.003647977s] END
I1101 20:22:00.039806    2586 trace.go:205] Trace[765634968]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:59.023) (total time: 1015ms):
Trace[765634968]: [1.01590632s] [1.01590632s] END
E1101 20:22:00.039842    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:00.107742    2586 trace.go:205] Trace[1213896694]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:6b2b9b07-e0ed-4fac-83b9-18775b6ccb81 (01-Nov-2021 20:21:59.101) (total time: 1005ms):
Trace[1213896694]: [1.005830139s] [1.005830139s] END
I1101 20:22:00.107795    2586 trace.go:205] Trace[1264188164]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:59.088) (total time: 1019ms):
Trace[1264188164]: [1.019199341s] [1.019199341s] END
E1101 20:22:00.107838    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:00.135680    2586 trace.go:205] Trace[161616962]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:b1367b7d-c8c0-4378-b17b-2c492f8b069e (01-Nov-2021 20:21:59.106) (total time: 1029ms):
Trace[161616962]: [1.029339712s] [1.029339712s] END
I1101 20:22:00.135721    2586 trace.go:205] Trace[1782704852]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:59.092) (total time: 1043ms):
Trace[1782704852]: [1.04348184s] [1.04348184s] END
E1101 20:22:00.135749    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:00.295770    2586 trace.go:205] Trace[497914119]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:ba6cdd6e-c2be-4d24-966e-702fe24c3dd1 (01-Nov-2021 20:21:59.290) (total time: 1004ms):
Trace[497914119]: [1.004898334s] [1.004898334s] END
I1101 20:22:00.295817    2586 trace.go:205] Trace[713566365]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:59.280) (total time: 1015ms):
Trace[713566365]: [1.015093388s] [1.015093388s] END
E1101 20:22:00.295853    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:00.679718    2586 trace.go:205] Trace[2121386886]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1alpha3,object-count:1,UID:c4490cca-aca7-434d-9ba5-94ef1b43e254 (01-Nov-2021 20:21:59.676) (total time: 1003ms):
Trace[2121386886]: [1.003057004s] [1.003057004s] END
I1101 20:22:00.679778    2586 trace.go:205] Trace[1528803957]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:59.664) (total time: 1015ms):
Trace[1528803957]: [1.015623381s] [1.015623381s] END
E1101 20:22:00.679815    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1alpha3, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:00.747715    2586 trace.go:205] Trace[1018437998]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:3318803b-94f6-474b-a75b-a1c1d54da4c4 (01-Nov-2021 20:21:59.732) (total time: 1015ms):
Trace[1018437998]: [1.015604176s] [1.015604176s] END
I1101 20:22:00.747768    2586 trace.go:205] Trace[321919791]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:59.720) (total time: 1026ms):
Trace[321919791]: [1.026987304s] [1.026987304s] END
E1101 20:22:00.747801    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:00.807716    2586 trace.go:205] Trace[951148024]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:844a79b5-ba79-454f-a649-4634b8399e69 (01-Nov-2021 20:21:59.793) (total time: 1014ms):
Trace[951148024]: [1.014346398s] [1.014346398s] END
I1101 20:22:00.807767    2586 trace.go:205] Trace[892155961]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:59.780) (total time: 1026ms):
Trace[892155961]: [1.026797816s] [1.026797816s] END
E1101 20:22:00.807798    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:00.839698    2586 trace.go:205] Trace[98556770]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:083899c2-a5dc-4afe-a1b8-3d9c5805c6ee (01-Nov-2021 20:21:59.838) (total time: 1000ms):
Trace[98556770]: [1.00075194s] [1.00075194s] END
I1101 20:22:00.839736    2586 trace.go:205] Trace[2013769399]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:21:59.824) (total time: 1015ms):
Trace[2013769399]: [1.015030608s] [1.015030608s] END
E1101 20:22:00.839772    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:01.159720    2586 trace.go:205] Trace[1940988138]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:b23ed2b5-54da-490c-9cf2-cc091c240582 (01-Nov-2021 20:22:00.131) (total time: 1028ms):
Trace[1940988138]: [1.02827191s] [1.02827191s] END
I1101 20:22:01.159785    2586 trace.go:205] Trace[2078112572]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:00.116) (total time: 1043ms):
Trace[2078112572]: [1.043128504s] [1.043128504s] END
E1101 20:22:01.159820    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:01.255741    2586 trace.go:205] Trace[98230197]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:4604df7d-f332-4c5c-9f3c-efd0b5f1a399 (01-Nov-2021 20:22:00.252) (total time: 1003ms):
Trace[98230197]: [1.003377562s] [1.003377562s] END
I1101 20:22:01.255794    2586 trace.go:205] Trace[789468099]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:00.240) (total time: 1015ms):
Trace[789468099]: [1.015570768s] [1.015570768s] END
E1101 20:22:01.255830    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:01.543696    2586 trace.go:205] Trace[2026511607]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1alpha2,object-count:1,UID:45ff3623-6ae4-4f3d-aa00-cfbc8ff84fbd (01-Nov-2021 20:22:00.539) (total time: 1003ms):
Trace[2026511607]: [1.003949621s] [1.003949621s] END
I1101 20:22:01.543737    2586 trace.go:205] Trace[569889286]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:00.528) (total time: 1015ms):
Trace[569889286]: [1.015064782s] [1.015064782s] END
E1101 20:22:01.543765    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1alpha2, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:01.703694    2586 trace.go:205] Trace[1561332728]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:023abb4c-1833-459a-add7-11af3e3d8a00 (01-Nov-2021 20:22:00.700) (total time: 1002ms):
Trace[1561332728]: [1.002903096s] [1.002903096s] END
I1101 20:22:01.703749    2586 trace.go:205] Trace[1802413549]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:00.688) (total time: 1015ms):
Trace[1802413549]: [1.015523174s] [1.015523174s] END
E1101 20:22:01.703777    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:01.835705    2586 trace.go:205] Trace[1763523796]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:452157c1-d7b6-483a-bc90-a14112c61e13 (01-Nov-2021 20:22:00.828) (total time: 1007ms):
Trace[1763523796]: [1.007384004s] [1.007384004s] END
I1101 20:22:01.835774    2586 trace.go:205] Trace[1429261092]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:00.815) (total time: 1019ms):
Trace[1429261092]: [1.019891739s] [1.019891739s] END
E1101 20:22:01.835803    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:01.959707    2586 trace.go:205] Trace[1706657551]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:3867b87b-cea3-4fb1-bbf8-f760ee1e9cb7 (01-Nov-2021 20:22:00.927) (total time: 1032ms):
Trace[1706657551]: [1.032341367s] [1.032341367s] END
I1101 20:22:01.959745    2586 trace.go:205] Trace[1073515493]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:00.916) (total time: 1042ms):
Trace[1073515493]: [1.042869022s] [1.042869022s] END
E1101 20:22:01.959774    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:02.023735    2586 trace.go:205] Trace[811502867]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1beta1,object-count:1,UID:7f9298f8-9b52-4cc3-88fc-9c98e6c36ddb (01-Nov-2021 20:22:01.019) (total time: 1004ms):
Trace[811502867]: [1.004155458s] [1.004155458s] END
I1101 20:22:02.023788    2586 trace.go:205] Trace[629528487]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:01.007) (total time: 1015ms):
Trace[629528487]: [1.015785367s] [1.015785367s] END
E1101 20:22:02.023826    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1beta1, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:02.055742    2586 trace.go:205] Trace[361633448]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:6247be22-91f9-4d84-a5d7-9e2eba760043 (01-Nov-2021 20:22:01.054) (total time: 1001ms):
Trace[361633448]: [1.001394627s] [1.001394627s] END
I1101 20:22:02.055789    2586 trace.go:205] Trace[1333196727]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:01.040) (total time: 1015ms):
Trace[1333196727]: [1.015585047s] [1.015585047s] END
E1101 20:22:02.055820    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:02.151689    2586 trace.go:205] Trace[2055673491]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:9ca52fe4-89c7-4430-95d7-84aceba84f01 (01-Nov-2021 20:22:01.147) (total time: 1004ms):
Trace[2055673491]: [1.004483613s] [1.004483613s] END
I1101 20:22:02.151763    2586 trace.go:205] Trace[29324902]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:01.135) (total time: 1015ms):
Trace[29324902]: [1.015805036s] [1.015805036s] END
E1101 20:22:02.151791    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:02.151695    2586 trace.go:205] Trace[543213160]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:780fca0f-86d0-41e3-a192-7d613a3ada35 (01-Nov-2021 20:22:01.124) (total time: 1027ms):
Trace[543213160]: [1.027180311s] [1.027180311s] END
I1101 20:22:02.151826    2586 trace.go:205] Trace[1303644108]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:01.108) (total time: 1042ms):
Trace[1303644108]: [1.042882881s] [1.042882881s] END
E1101 20:22:02.151851    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:02.311682    2586 trace.go:205] Trace[1402087108]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:0163ce14-1f8c-4879-b4d0-4137428f01a7 (01-Nov-2021 20:22:01.306) (total time: 1004ms):
Trace[1402087108]: [1.004803243s] [1.004803243s] END
I1101 20:22:02.311720    2586 trace.go:205] Trace[335864132]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:01.295) (total time: 1015ms):
Trace[335864132]: [1.015748531s] [1.015748531s] END
E1101 20:22:02.311760    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:02.385263    2586 scope.go:111] "RemoveContainer" containerID="d466a3a17cf62a62a352bffbdb5a645e77313e097d07628ae71efeb7d29120f1"
E1101 20:22:02.385978    2586 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"longhorn-csi-plugin\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=longhorn-csi-plugin pod=longhorn-csi-plugin-gxqb2_longhorn-system(2034c658-a446-4e42-bdee-aa2d161f06f9)\"" pod="longhorn-system/longhorn-csi-plugin-gxqb2" podUID=2034c658-a446-4e42-bdee-aa2d161f06f9
I1101 20:22:02.687627    2586 scope.go:111] "RemoveContainer" containerID="8c8871b752d107604212967bb7871a4216d1f641d99c003e5e5f7f1506a42dda"
E1101 20:22:02.688551    2586 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"prometheus\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=prometheus pod=prometheus-rancher-monitoring-prometheus-0_cattle-monitoring-system(42b6fd88-42c1-4545-bf30-556b7623645a)\"" pod="cattle-monitoring-system/prometheus-rancher-monitoring-prometheus-0" podUID=42b6fd88-42c1-4545-bf30-556b7623645a
I1101 20:22:02.727696    2586 trace.go:205] Trace[359641660]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1alpha3,object-count:1,UID:aeb5f695-2f30-489a-9cbc-a0e49bbc5620 (01-Nov-2021 20:22:01.697) (total time: 1030ms):
Trace[359641660]: [1.030014891s] [1.030014891s] END
I1101 20:22:02.727766    2586 trace.go:205] Trace[761240564]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:01.680) (total time: 1047ms):
Trace[761240564]: [1.047359619s] [1.047359619s] END
E1101 20:22:02.727800    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1alpha3, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:02.759722    2586 trace.go:205] Trace[516372836]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:bf9ad079-e261-4662-8bf1-8264adc30924 (01-Nov-2021 20:22:01.758) (total time: 1000ms):
Trace[516372836]: [1.000794568s] [1.000794568s] END
I1101 20:22:02.759771    2586 trace.go:205] Trace[979467272]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:01.748) (total time: 1011ms):
Trace[979467272]: [1.011246601s] [1.011246601s] END
E1101 20:22:02.759800    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:02.823723    2586 trace.go:205] Trace[2067005685]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:e7178170-a4f3-4c39-95fe-fc416edd104f (01-Nov-2021 20:22:01.820) (total time: 1003ms):
Trace[2067005685]: [1.00319637s] [1.00319637s] END
I1101 20:22:02.823779    2586 trace.go:205] Trace[1315612572]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:01.808) (total time: 1015ms):
Trace[1315612572]: [1.015672544s] [1.015672544s] END
E1101 20:22:02.823815    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:02.855686    2586 trace.go:205] Trace[678152571]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:46e7ad08-6da5-442a-8c00-143854c42e4b (01-Nov-2021 20:22:01.851) (total time: 1004ms):
Trace[678152571]: [1.004486458s] [1.004486458s] END
I1101 20:22:02.855730    2586 trace.go:205] Trace[761921965]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:01.840) (total time: 1015ms):
Trace[761921965]: [1.015206841s] [1.015206841s] END
E1101 20:22:02.855760    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:03.211728    2586 trace.go:205] Trace[1875390311]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:c7f6a482-c087-4bf0-a9e7-49db64bb4328 (01-Nov-2021 20:22:02.180) (total time: 1030ms):
Trace[1875390311]: [1.030890952s] [1.030890952s] END
I1101 20:22:03.211799    2586 trace.go:205] Trace[1497772160]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:02.160) (total time: 1051ms):
Trace[1497772160]: [1.051482533s] [1.051482533s] END
E1101 20:22:03.211833    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:03.271667    2586 trace.go:205] Trace[1524550760]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:97b6823d-f7e1-4191-b7a4-dc6bb8475f0c (01-Nov-2021 20:22:02.268) (total time: 1003ms):
Trace[1524550760]: [1.003227916s] [1.003227916s] END
I1101 20:22:03.271709    2586 trace.go:205] Trace[1124185419]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:02.256) (total time: 1014ms):
Trace[1124185419]: [1.014785098s] [1.014785098s] END
E1101 20:22:03.271729    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:03.385396    2586 scope.go:111] "RemoveContainer" containerID="4d1a051bfb5b2d910f0f6f5fbc15320dad2345b504086071c3ede407fdd17256"
E1101 20:22:03.385675    2586 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"user-management-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=user-management-api pod=user-management-api-76c74dc97b-bxjtk_test(8109999b-7caa-4583-9b85-31640b4c7100)\"" pod="test/user-management-api-76c74dc97b-bxjtk" podUID=8109999b-7caa-4583-9b85-31640b4c7100
I1101 20:22:03.385741    2586 scope.go:111] "RemoveContainer" containerID="4ea83f240c3b6e7992dd576a6f666e583f464a64e32b784e760706a043a4d865"
E1101 20:22:03.386083    2586 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"order-managament-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=order-managament-api pod=order-managament-api-7d654d8d84-jvnf5_test(66bf2f2b-f992-4250-89ff-160889c09af7)\"" pod="test/order-managament-api-7d654d8d84-jvnf5" podUID=66bf2f2b-f992-4250-89ff-160889c09af7
I1101 20:22:03.559698    2586 trace.go:205] Trace[960122620]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1alpha2,object-count:1,UID:766ccc20-fc36-4851-87d1-4ffcf6b46ace (01-Nov-2021 20:22:02.555) (total time: 1003ms):
Trace[960122620]: [1.003716889s] [1.003716889s] END
I1101 20:22:03.559735    2586 trace.go:205] Trace[1985100837]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:02.544) (total time: 1014ms):
Trace[1985100837]: [1.014978428s] [1.014978428s] END
E1101 20:22:03.559762    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1alpha2, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:03.719687    2586 trace.go:205] Trace[1336518367]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:e8067181-d3bc-4d16-836c-0f25abc86d93 (01-Nov-2021 20:22:02.714) (total time: 1005ms):
Trace[1336518367]: [1.005255452s] [1.005255452s] END
I1101 20:22:03.719745    2586 trace.go:205] Trace[1756789669]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:02.703) (total time: 1015ms):
Trace[1756789669]: [1.015777847s] [1.015777847s] END
E1101 20:22:03.719766    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:03.883706    2586 trace.go:205] Trace[1489884540]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:ca933293-b53a-4820-8415-7f36bc80f1da (01-Nov-2021 20:22:02.847) (total time: 1036ms):
Trace[1489884540]: [1.036013937s] [1.036013937s] END
I1101 20:22:03.883755    2586 trace.go:205] Trace[1469346776]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:02.836) (total time: 1047ms):
Trace[1469346776]: [1.04735612s] [1.04735612s] END
E1101 20:22:03.883786    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:03.975729    2586 trace.go:205] Trace[1108582867]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:05339e04-f4ef-4160-8497-58749d684e33 (01-Nov-2021 20:22:02.970) (total time: 1005ms):
Trace[1108582867]: [1.005251298s] [1.005251298s] END
I1101 20:22:03.975781    2586 trace.go:205] Trace[1492111274]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:02.959) (total time: 1015ms):
Trace[1492111274]: [1.015763273s] [1.015763273s] END
E1101 20:22:03.975818    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:04.075690    2586 trace.go:205] Trace[1663675922]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:b2c9d1f4-74c5-4c6c-971b-0a1f743c463a (01-Nov-2021 20:22:03.067) (total time: 1007ms):
Trace[1663675922]: [1.007916147s] [1.007916147s] END
I1101 20:22:04.075734    2586 trace.go:205] Trace[2097825984]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:03.056) (total time: 1019ms):
Trace[2097825984]: [1.019025368s] [1.019025368s] END
E1101 20:22:04.075760    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:04.080345    2586 trace.go:205] Trace[1483288149]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1beta1,object-count:1,UID:e61f2da6-d14b-4c66-8e8f-ea64ecd11738 (01-Nov-2021 20:22:03.040) (total time: 1039ms):
Trace[1483288149]: [1.039891231s] [1.039891231s] END
I1101 20:22:04.080405    2586 trace.go:205] Trace[624523915]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:03.024) (total time: 1056ms):
Trace[624523915]: [1.056260608s] [1.056260608s] END
E1101 20:22:04.080434    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1beta1, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:04.167678    2586 trace.go:205] Trace[2124297753]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:d4d95744-92fa-4e31-924c-f908c8515d9d (01-Nov-2021 20:22:03.165) (total time: 1002ms):
Trace[2124297753]: [1.002519854s] [1.002519854s] END
I1101 20:22:04.167719    2586 trace.go:205] Trace[1857686808]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:03.152) (total time: 1015ms):
Trace[1857686808]: [1.015283474s] [1.015283474s] END
E1101 20:22:04.167741    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:04.167678    2586 trace.go:205] Trace[1470354092]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:3b238b01-f20f-4b2a-af20-c86ac30abe83 (01-Nov-2021 20:22:03.164) (total time: 1003ms):
Trace[1470354092]: [1.003615453s] [1.003615453s] END
I1101 20:22:04.167762    2586 trace.go:205] Trace[938505408]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:03.152) (total time: 1015ms):
Trace[938505408]: [1.015307299s] [1.015307299s] END
E1101 20:22:04.167773    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:04.327706    2586 trace.go:205] Trace[661376791]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:38dfc8ba-b5eb-40cf-90ca-eaf7639aa374 (01-Nov-2021 20:22:03.326) (total time: 1001ms):
Trace[661376791]: [1.001158871s] [1.001158871s] END
I1101 20:22:04.327761    2586 trace.go:205] Trace[20015875]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:03.312) (total time: 1015ms):
Trace[20015875]: [1.015696991s] [1.015696991s] END
E1101 20:22:04.327789    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:04.743994    2586 trace.go:205] Trace[1468142671]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1alpha3,object-count:1,UID:45769716-ca6e-4c1c-b7be-6a277f6ae9ed (01-Nov-2021 20:22:03.739) (total time: 1004ms):
Trace[1468142671]: [1.004341129s] [1.004341129s] END
I1101 20:22:04.744077    2586 trace.go:205] Trace[1981799028]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:03.727) (total time: 1016ms):
Trace[1981799028]: [1.016158975s] [1.016158975s] END
E1101 20:22:04.744688    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1alpha3, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:04.807801    2586 trace.go:205] Trace[717756195]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:80d9a692-a828-439b-8f51-ea7a3b24b5e4 (01-Nov-2021 20:22:03.781) (total time: 1026ms):
Trace[717756195]: [1.026123183s] [1.026123183s] END
I1101 20:22:04.807849    2586 trace.go:205] Trace[1028105197]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:03.759) (total time: 1047ms):
Trace[1028105197]: [1.047932166s] [1.047932166s] END
E1101 20:22:04.807882    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:04.883773    2586 trace.go:205] Trace[1345663483]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:26d931a2-6bd4-4384-bb38-087d622cbc7f (01-Nov-2021 20:22:03.836) (total time: 1046ms):
Trace[1345663483]: [1.046761514s] [1.046761514s] END
I1101 20:22:04.883853    2586 trace.go:205] Trace[117267443]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:03.823) (total time: 1059ms):
Trace[117267443]: [1.059894656s] [1.059894656s] END
E1101 20:22:04.883877    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:04.883986    2586 trace.go:205] Trace[632590289]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:715a20b1-a153-4d81-bdde-594eba8e1419 (01-Nov-2021 20:22:03.867) (total time: 1016ms):
Trace[632590289]: [1.016663804s] [1.016663804s] END
I1101 20:22:04.884010    2586 trace.go:205] Trace[22528081]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:03.856) (total time: 1027ms):
Trace[22528081]: [1.027631386s] [1.027631386s] END
E1101 20:22:04.884022    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:05.255784    2586 trace.go:205] Trace[1472507261]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:269d2efa-75ab-4feb-be4c-b7ea68fc905d (01-Nov-2021 20:22:04.224) (total time: 1030ms):
Trace[1472507261]: [1.030828054s] [1.030828054s] END
I1101 20:22:05.255841    2586 trace.go:205] Trace[1069800561]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:04.212) (total time: 1043ms):
Trace[1069800561]: [1.04314706s] [1.04314706s] END
E1101 20:22:05.255901    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:05.287755    2586 trace.go:205] Trace[440557489]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:3c933610-24c9-4c29-86f8-6d1e3208975a (01-Nov-2021 20:22:04.285) (total time: 1002ms):
Trace[440557489]: [1.002594143s] [1.002594143s] END
I1101 20:22:05.287814    2586 trace.go:205] Trace[1671386535]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:04.271) (total time: 1015ms):
Trace[1671386535]: [1.015984804s] [1.015984804s] END
E1101 20:22:05.287838    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:05.580587    2586 trace.go:205] Trace[2127384150]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1alpha2,object-count:1,UID:f3187d82-41ba-4bdc-9a1c-557b8dc8c5de (01-Nov-2021 20:22:04.572) (total time: 1007ms):
Trace[2127384150]: [1.007779479s] [1.007779479s] END
I1101 20:22:05.580656    2586 trace.go:205] Trace[194130984]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:04.560) (total time: 1019ms):
Trace[194130984]: [1.019856027s] [1.019856027s] END
E1101 20:22:05.580693    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1alpha2, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:05.780421    2586 trace.go:205] Trace[1663892278]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:72fc13da-f170-494c-babe-2041fd2fb846 (01-Nov-2021 20:22:04.735) (total time: 1044ms):
Trace[1663892278]: [1.044500969s] [1.044500969s] END
I1101 20:22:05.780474    2586 trace.go:205] Trace[1953083093]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:04.719) (total time: 1060ms):
Trace[1953083093]: [1.060469056s] [1.060469056s] END
E1101 20:22:05.780506    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:05.931617    2586 trace.go:205] Trace[559144319]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:6bb8a392-f21a-45e2-96fe-b8ca02a89a4f (01-Nov-2021 20:22:04.898) (total time: 1033ms):
Trace[559144319]: [1.033418058s] [1.033418058s] END
I1101 20:22:05.931657    2586 trace.go:205] Trace[1384143726]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:04.884) (total time: 1047ms):
Trace[1384143726]: [1.047524601s] [1.047524601s] END
E1101 20:22:05.931713    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:06.023745    2586 trace.go:205] Trace[1696319155]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:703043ba-3c57-4e15-8b15-19412f01cc06 (01-Nov-2021 20:22:04.996) (total time: 1027ms):
Trace[1696319155]: [1.027663315s] [1.027663315s] END
I1101 20:22:06.023796    2586 trace.go:205] Trace[1159931824]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:04.980) (total time: 1043ms):
Trace[1159931824]: [1.043338807s] [1.043338807s] END
E1101 20:22:06.023829    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:06.119699    2586 trace.go:205] Trace[762321893]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:ee3fdc87-4ae5-4a9c-afb4-5e8707ba38eb (01-Nov-2021 20:22:05.096) (total time: 1022ms):
Trace[762321893]: [1.022797147s] [1.022797147s] END
I1101 20:22:06.119752    2586 trace.go:205] Trace[752222472]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:05.081) (total time: 1038ms):
Trace[752222472]: [1.03856861s] [1.03856861s] END
I1101 20:22:06.119759    2586 trace.go:205] Trace[470818461]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1beta1,object-count:1,UID:7039940f-a009-48b2-9a32-190ec3de05b1 (01-Nov-2021 20:22:05.098) (total time: 1021ms):
Trace[470818461]: [1.021211627s] [1.021211627s] END
E1101 20:22:06.119786    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:06.119789    2586 trace.go:205] Trace[1874193739]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:05.080) (total time: 1039ms):
Trace[1874193739]: [1.039208113s] [1.039208113s] END
E1101 20:22:06.119807    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1beta1, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:06.215757    2586 trace.go:205] Trace[801272316]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:1f09825b-9bda-4a2b-8b1d-efb343b62c80 (01-Nov-2021 20:22:05.194) (total time: 1020ms):
Trace[801272316]: [1.020903493s] [1.020903493s] END
I1101 20:22:06.215802    2586 trace.go:205] Trace[1961813998]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:05.168) (total time: 1047ms):
Trace[1961813998]: [1.047689005s] [1.047689005s] END
E1101 20:22:06.215844    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:06.215757    2586 trace.go:205] Trace[696343563]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:91f7d7e6-6d34-44aa-9ce5-a66546f723a6 (01-Nov-2021 20:22:05.195) (total time: 1020ms):
Trace[696343563]: [1.020434663s] [1.020434663s] END
I1101 20:22:06.215912    2586 trace.go:205] Trace[1354831829]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:05.168) (total time: 1047ms):
Trace[1354831829]: [1.04781243s] [1.04781243s] END
E1101 20:22:06.215942    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:06.343726    2586 trace.go:205] Trace[177976858]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:09b32d14-04bf-433f-a483-df03479efc95 (01-Nov-2021 20:22:05.340) (total time: 1002ms):
Trace[177976858]: [1.002938498s] [1.002938498s] END
I1101 20:22:06.343772    2586 trace.go:205] Trace[1388153951]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:05.327) (total time: 1015ms):
Trace[1388153951]: [1.015874468s] [1.015874468s] END
E1101 20:22:06.343800    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
E1101 20:22:06.385538    2586 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"theme-provider\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=theme-provider pod=keycloak-0_keycloak(29ffde13-fa8d-407b-a9a7-7c212c8627bc)\"" pod="keycloak/keycloak-0" podUID=29ffde13-fa8d-407b-a9a7-7c212c8627bc
I1101 20:22:06.793255    2586 trace.go:205] Trace[785026953]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1alpha3,object-count:1,UID:70fadff6-05b4-4071-99ac-0df48d3986a1 (01-Nov-2021 20:22:05.760) (total time: 1032ms):
Trace[785026953]: [1.032648227s] [1.032648227s] END
I1101 20:22:06.793325    2586 trace.go:205] Trace[1812717985]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:05.745) (total time: 1048ms):
Trace[1812717985]: [1.048034969s] [1.048034969s] END
E1101 20:22:06.793365    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1alpha3, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:06.823749    2586 trace.go:205] Trace[1453682022]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:628117c0-80ae-457d-8cc3-86c3c940a480 (01-Nov-2021 20:22:05.820) (total time: 1002ms):
Trace[1453682022]: [1.002747822s] [1.002747822s] END
I1101 20:22:06.823823    2586 trace.go:205] Trace[1114455418]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:05.808) (total time: 1015ms):
Trace[1114455418]: [1.015357982s] [1.015357982s] END
E1101 20:22:06.823854    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:06.919711    2586 trace.go:205] Trace[841876575]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:ec32b014-f924-4a77-9045-8cd244770e74 (01-Nov-2021 20:22:05.897) (total time: 1021ms):
Trace[841876575]: [1.021781879s] [1.021781879s] END
I1101 20:22:06.919763    2586 trace.go:205] Trace[25817878]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:05.884) (total time: 1035ms):
Trace[25817878]: [1.035659295s] [1.035659295s] END
E1101 20:22:06.919796    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:06.919712    2586 trace.go:205] Trace[1052843727]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:d955b454-67ed-4e1b-913d-738e2dc6f329 (01-Nov-2021 20:22:05.895) (total time: 1023ms):
Trace[1052843727]: [1.023705081s] [1.023705081s] END
I1101 20:22:06.919832    2586 trace.go:205] Trace[1442363574]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:05.884) (total time: 1035ms):
Trace[1442363574]: [1.035722506s] [1.035722506s] END
E1101 20:22:06.919858    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:07.283766    2586 trace.go:205] Trace[47404936]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:bd9813bf-5400-410c-84c4-24979285109a (01-Nov-2021 20:22:06.269) (total time: 1013ms):
Trace[47404936]: [1.013880426s] [1.013880426s] END
I1101 20:22:07.283835    2586 trace.go:205] Trace[1434577232]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:06.256) (total time: 1026ms):
Trace[1434577232]: [1.026832444s] [1.026832444s] END
E1101 20:22:07.283883    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:07.335728    2586 trace.go:205] Trace[1057374656]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:b0df02d7-ccd0-476f-90d8-46190b325cfa (01-Nov-2021 20:22:06.303) (total time: 1032ms):
Trace[1057374656]: [1.032175519s] [1.032175519s] END
I1101 20:22:07.335797    2586 trace.go:205] Trace[320847360]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:06.287) (total time: 1047ms):
Trace[320847360]: [1.047865186s] [1.047865186s] END
E1101 20:22:07.335827    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:07.623683    2586 trace.go:205] Trace[790659093]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1alpha2,object-count:1,UID:07b9d0b4-1830-4e06-8cc9-53b297492cde (01-Nov-2021 20:22:06.597) (total time: 1026ms):
Trace[790659093]: [1.026601691s] [1.026601691s] END
I1101 20:22:07.623721    2586 trace.go:205] Trace[676757671]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:06.581) (total time: 1042ms):
Trace[676757671]: [1.042013701s] [1.042013701s] END
E1101 20:22:07.623752    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1alpha2, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:07.819705    2586 trace.go:205] Trace[725102879]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:b8db2727-0fcc-4985-ac43-bada199f3669 (01-Nov-2021 20:22:06.795) (total time: 1024ms):
Trace[725102879]: [1.024311966s] [1.024311966s] END
I1101 20:22:07.819753    2586 trace.go:205] Trace[1293898305]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:06.780) (total time: 1039ms):
Trace[1293898305]: [1.039130268s] [1.039130268s] END
E1101 20:22:07.819783    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:07.975709    2586 trace.go:205] Trace[1697009476]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:5054f644-7fdf-4bcb-83f8-85135aae24db (01-Nov-2021 20:22:06.944) (total time: 1031ms):
Trace[1697009476]: [1.031490357s] [1.031490357s] END
I1101 20:22:07.975753    2586 trace.go:205] Trace[1843073340]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:06.932) (total time: 1042ms):
Trace[1843073340]: [1.042968869s] [1.042968869s] END
E1101 20:22:07.975780    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:08.071774    2586 trace.go:205] Trace[323084030]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:36436c67-afa2-421e-b2b8-94b3e82f5617 (01-Nov-2021 20:22:07.043) (total time: 1028ms):
Trace[323084030]: [1.028088711s] [1.028088711s] END
I1101 20:22:08.071853    2586 trace.go:205] Trace[452386147]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:07.024) (total time: 1046ms):
Trace[452386147]: [1.046999474s] [1.046999474s] END
E1101 20:22:08.071881    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:08.167686    2586 trace.go:205] Trace[1912432660]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1beta1,object-count:1,UID:d0490b9e-9456-4962-b14b-2033000124b6 (01-Nov-2021 20:22:07.134) (total time: 1032ms):
Trace[1912432660]: [1.032770831s] [1.032770831s] END
I1101 20:22:08.167726    2586 trace.go:205] Trace[1692381860]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:07.120) (total time: 1047ms):
Trace[1692381860]: [1.04746561s] [1.04746561s] END
E1101 20:22:08.167756    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1beta1, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:08.180369    2586 trace.go:205] Trace[1720113255]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:276d25c5-37e2-456f-bee5-0e84d48e59bd (01-Nov-2021 20:22:07.138) (total time: 1041ms):
Trace[1720113255]: [1.041407187s] [1.041407187s] END
I1101 20:22:08.180417    2586 trace.go:205] Trace[1077634113]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:07.120) (total time: 1060ms):
Trace[1077634113]: [1.06013887s] [1.06013887s] END
E1101 20:22:08.180452    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:08.231732    2586 trace.go:205] Trace[1322293054]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:70157acd-1f6f-4f3f-aa7b-42090fecc8d3 (01-Nov-2021 20:22:07.230) (total time: 1001ms):
Trace[1322293054]: [1.001631281s] [1.001631281s] END
I1101 20:22:08.231789    2586 trace.go:205] Trace[849656434]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:07.216) (total time: 1015ms):
Trace[849656434]: [1.015053383s] [1.015053383s] END
E1101 20:22:08.231830    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:08.263672    2586 trace.go:205] Trace[261341200]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:0519d558-e80c-43de-8e1d-8dc3e68f8b0e (01-Nov-2021 20:22:07.232) (total time: 1031ms):
Trace[261341200]: [1.031508986s] [1.031508986s] END
I1101 20:22:08.263735    2586 trace.go:205] Trace[534662244]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:07.216) (total time: 1047ms):
Trace[534662244]: [1.047010165s] [1.047010165s] END
E1101 20:22:08.263770    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:08.359769    2586 trace.go:205] Trace[1914773392]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:369732ac-996c-4f83-88ef-90c7fc439ffd (01-Nov-2021 20:22:07.354) (total time: 1005ms):
Trace[1914773392]: [1.005161416s] [1.005161416s] END
I1101 20:22:08.359838    2586 trace.go:205] Trace[22684293]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:07.343) (total time: 1015ms):
Trace[22684293]: [1.015923808s] [1.015923808s] END
E1101 20:22:08.359874    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:08.384681    2586 scope.go:111] "RemoveContainer" containerID="64ee128b9d00efa08946314840985befc7b404ee74abf2a79c988312bd14f007"
E1101 20:22:08.385022    2586 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"longhorn-ui\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=longhorn-ui pod=longhorn-ui-64b96684bc-8jbmk_longhorn-system(1f9a0b1c-9b12-4cfc-80ff-825a2e6911c7)\"" pod="longhorn-system/longhorn-ui-64b96684bc-8jbmk" podUID=1f9a0b1c-9b12-4cfc-80ff-825a2e6911c7
I1101 20:22:08.839707    2586 trace.go:205] Trace[1905383571]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1alpha3,object-count:1,UID:191a4c01-d8f8-4379-afe1-2df4de9e6ec2 (01-Nov-2021 20:22:07.807) (total time: 1032ms):
Trace[1905383571]: [1.032618269s] [1.032618269s] END
I1101 20:22:08.839764    2586 trace.go:205] Trace[410712833]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:07.793) (total time: 1046ms):
Trace[410712833]: [1.046241855s] [1.046241855s] END
E1101 20:22:08.839808    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1alpha3, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:08.875778    2586 trace.go:205] Trace[908669351]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:5c92060f-aa36-49ce-8afa-1f1f382cb0c3 (01-Nov-2021 20:22:07.839) (total time: 1036ms):
Trace[908669351]: [1.03668451s] [1.03668451s] END
I1101 20:22:08.875850    2586 trace.go:205] Trace[1888918062]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:07.823) (total time: 1051ms):
Trace[1888918062]: [1.051857387s] [1.051857387s] END
E1101 20:22:08.875891    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:08.967711    2586 trace.go:205] Trace[993938116]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:633f0e2d-4eb3-4a1e-93d2-1a008a131862 (01-Nov-2021 20:22:07.935) (total time: 1031ms):
Trace[993938116]: [1.031850996s] [1.031850996s] END
I1101 20:22:08.967777    2586 trace.go:205] Trace[1992303873]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:07.920) (total time: 1047ms):
Trace[1992303873]: [1.047736185s] [1.047736185s] END
E1101 20:22:08.967805    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:08.967711    2586 trace.go:205] Trace[1555087407]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:e3c1b134-698e-4d17-812c-e12531541906 (01-Nov-2021 20:22:07.935) (total time: 1031ms):
Trace[1555087407]: [1.031770472s] [1.031770472s] END
I1101 20:22:08.967839    2586 trace.go:205] Trace[418966467]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:07.920) (total time: 1047ms):
Trace[418966467]: [1.047800756s] [1.047800756s] END
E1101 20:22:08.967859    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:09.323774    2586 trace.go:205] Trace[233458637]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:7ab04fae-bf75-469c-a0a4-7b91bf8d924b (01-Nov-2021 20:22:08.299) (total time: 1024ms):
Trace[233458637]: [1.024300966s] [1.024300966s] END
I1101 20:22:09.323860    2586 trace.go:205] Trace[1484355924]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:08.283) (total time: 1039ms):
Trace[1484355924]: [1.039850667s] [1.039850667s] END
E1101 20:22:09.323897    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:09.383697    2586 trace.go:205] Trace[1713422660]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:a501f470-2a98-4b80-bc19-799dc8937926 (01-Nov-2021 20:22:08.352) (total time: 1031ms):
Trace[1713422660]: [1.031155451s] [1.031155451s] END
I1101 20:22:09.383756    2586 trace.go:205] Trace[1584741326]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:08.336) (total time: 1047ms):
Trace[1584741326]: [1.047089362s] [1.047089362s] END
E1101 20:22:09.383792    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:09.642833    2586 trace.go:205] Trace[1803774319]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1alpha2,object-count:1,UID:1f214686-6c21-4733-ac6e-253f826c7d90 (01-Nov-2021 20:22:08.638) (total time: 1004ms):
Trace[1803774319]: [1.00458067s] [1.00458067s] END
I1101 20:22:09.642894    2586 trace.go:205] Trace[38157110]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:08.624) (total time: 1018ms):
Trace[38157110]: [1.018748058s] [1.018748058s] END
E1101 20:22:09.642926    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1alpha2, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:09.880415    2586 trace.go:205] Trace[538071723]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:b1489008-b329-4dd5-8787-8ac5c8b255df (01-Nov-2021 20:22:08.832) (total time: 1047ms):
Trace[538071723]: [1.047400867s] [1.047400867s] END
I1101 20:22:09.880497    2586 trace.go:205] Trace[2034828561]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:08.819) (total time: 1060ms):
Trace[2034828561]: [1.060565311s] [1.060565311s] END
E1101 20:22:09.880532    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:09.991714    2586 trace.go:205] Trace[1000650514]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:881b23bd-cc6a-442e-8027-9fd02194ee97 (01-Nov-2021 20:22:08.989) (total time: 1002ms):
Trace[1000650514]: [1.002313019s] [1.002313019s] END
I1101 20:22:09.991789    2586 trace.go:205] Trace[439417717]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:08.976) (total time: 1015ms):
Trace[439417717]: [1.015362965s] [1.015362965s] END
E1101 20:22:09.991818    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:10.119774    2586 trace.go:205] Trace[933991462]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:d4d9bf12-2e06-448c-bd39-0d04905d5447 (01-Nov-2021 20:22:09.094) (total time: 1025ms):
Trace[933991462]: [1.0256248s] [1.0256248s] END
I1101 20:22:10.119835    2586 trace.go:205] Trace[237962449]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:09.080) (total time: 1039ms):
Trace[237962449]: [1.03940954s] [1.03940954s] END
E1101 20:22:10.119874    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:10.215709    2586 trace.go:205] Trace[447327644]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1beta1,object-count:1,UID:12ac1e6d-4d4b-4291-a56a-b5424837dcee (01-Nov-2021 20:22:09.202) (total time: 1013ms):
Trace[447327644]: [1.013121926s] [1.013121926s] END
I1101 20:22:10.215773    2586 trace.go:205] Trace[86183511]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:09.180) (total time: 1035ms):
Trace[86183511]: [1.035435621s] [1.035435621s] END
I1101 20:22:10.215734    2586 trace.go:205] Trace[1826437661]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:660e8493-e87f-4ef5-a43a-803365acdf7f (01-Nov-2021 20:22:09.202) (total time: 1013ms):
Trace[1826437661]: [1.013464345s] [1.013464345s] END
E1101 20:22:10.215803    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1beta1, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:10.215831    2586 trace.go:205] Trace[1601317923]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:09.180) (total time: 1035ms):
Trace[1601317923]: [1.035302848s] [1.035302848s] END
E1101 20:22:10.215882    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:10.311686    2586 trace.go:205] Trace[66383694]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:537c59e4-7f5d-424f-8c7b-fa1d48692e1d (01-Nov-2021 20:22:09.282) (total time: 1028ms):
Trace[66383694]: [1.028745516s] [1.028745516s] END
I1101 20:22:10.311755    2586 trace.go:205] Trace[238315750]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:09.232) (total time: 1079ms):
Trace[238315750]: [1.07900574s] [1.07900574s] END
E1101 20:22:10.311784    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:10.311687    2586 trace.go:205] Trace[567205342]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:afce581d-7a83-45a5-87cc-7a7181140367 (01-Nov-2021 20:22:09.296) (total time: 1014ms):
Trace[567205342]: [1.014897024s] [1.014897024s] END
I1101 20:22:10.311817    2586 trace.go:205] Trace[896571362]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:09.280) (total time: 1031ms):
Trace[896571362]: [1.031471693s] [1.031471693s] END
E1101 20:22:10.311832    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:10.411708    2586 trace.go:205] Trace[610175501]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:24f8cf88-23ce-4987-9bf8-e3c52a4a3456 (01-Nov-2021 20:22:09.396) (total time: 1015ms):
Trace[610175501]: [1.015542482s] [1.015542482s] END
I1101 20:22:10.411786    2586 trace.go:205] Trace[488614041]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:09.380) (total time: 1031ms):
Trace[488614041]: [1.031488896s] [1.031488896s] END
E1101 20:22:10.411828    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:10.891641    2586 trace.go:205] Trace[515120110]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1alpha3,object-count:1,UID:6033c111-f2b8-4c5d-8853-610d2749aeb4 (01-Nov-2021 20:22:09.855) (total time: 1036ms):
Trace[515120110]: [1.036098145s] [1.036098145s] END
I1101 20:22:10.891706    2586 trace.go:205] Trace[1914930048]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:09.839) (total time: 1051ms):
Trace[1914930048]: [1.051804386s] [1.051804386s] END
E1101 20:22:10.891751    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1alpha3, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:10.919728    2586 trace.go:205] Trace[525507542]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:7203e6d3-9306-44a2-bdda-38a7a9c5e105 (01-Nov-2021 20:22:09.896) (total time: 1023ms):
Trace[525507542]: [1.023676564s] [1.023676564s] END
I1101 20:22:10.919801    2586 trace.go:205] Trace[1889577539]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:09.880) (total time: 1039ms):
Trace[1889577539]: [1.039483465s] [1.039483465s] END
E1101 20:22:10.919835    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:11.015711    2586 trace.go:205] Trace[1713179662]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:a724eb78-228e-496d-be0b-dad663c2ed8e (01-Nov-2021 20:22:09.992) (total time: 1023ms):
Trace[1713179662]: [1.023650826s] [1.023650826s] END
I1101 20:22:11.015778    2586 trace.go:205] Trace[1667298666]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:09.980) (total time: 1035ms):
Trace[1667298666]: [1.035464826s] [1.035464826s] END
E1101 20:22:11.015802    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:11.015711    2586 trace.go:205] Trace[1102810140]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:243e6aa8-f5d0-4185-872b-31a5ef015240 (01-Nov-2021 20:22:09.994) (total time: 1021ms):
Trace[1102810140]: [1.021327965s] [1.021327965s] END
I1101 20:22:11.015862    2586 trace.go:205] Trace[1856169025]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:09.980) (total time: 1035ms):
Trace[1856169025]: [1.035523727s] [1.035523727s] END
E1101 20:22:11.015896    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:11.380445    2586 trace.go:205] Trace[484709342]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:1e7caf72-7b6a-43fd-bd7e-5a129cec35ec (01-Nov-2021 20:22:10.335) (total time: 1045ms):
Trace[484709342]: [1.045222607s] [1.045222607s] END
I1101 20:22:11.380524    2586 trace.go:205] Trace[589920793]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:10.324) (total time: 1055ms):
Trace[589920793]: [1.05591621s] [1.05591621s] END
E1101 20:22:11.380568    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:11.399701    2586 trace.go:205] Trace[344968257]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:1da2f5cd-c4b9-4732-8c0b-6ce4dea6f17e (01-Nov-2021 20:22:10.397) (total time: 1002ms):
Trace[344968257]: [1.002276501s] [1.002276501s] END
I1101 20:22:11.399771    2586 trace.go:205] Trace[2029335240]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:10.383) (total time: 1015ms):
Trace[2029335240]: [1.015840855s] [1.015840855s] END
E1101 20:22:11.399800    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:11.687775    2586 trace.go:205] Trace[28968968]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1alpha2,object-count:1,UID:c2a2df3d-36b9-4924-b1ef-991040615564 (01-Nov-2021 20:22:10.658) (total time: 1028ms):
Trace[28968968]: [1.02878809s] [1.02878809s] END
I1101 20:22:11.687837    2586 trace.go:205] Trace[2013973756]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:10.643) (total time: 1044ms):
Trace[2013973756]: [1.044311542s] [1.044311542s] END
E1101 20:22:11.687870    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1alpha2, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:11.911709    2586 trace.go:205] Trace[1539647217]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:8113c42b-67b0-4c24-bb53-307dd6704f2a (01-Nov-2021 20:22:10.890) (total time: 1020ms):
Trace[1539647217]: [1.020818902s] [1.020818902s] END
I1101 20:22:11.911788    2586 trace.go:205] Trace[1328316829]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:10.880) (total time: 1031ms):
Trace[1328316829]: [1.031139387s] [1.031139387s] END
E1101 20:22:11.911824    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:12.007736    2586 trace.go:205] Trace[246403331]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:134b2c31-0515-4401-aaab-a91d3180b268 (01-Nov-2021 20:22:11.003) (total time: 1003ms):
Trace[246403331]: [1.00390429s] [1.00390429s] END
I1101 20:22:12.007786    2586 trace.go:205] Trace[1778260582]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:10.991) (total time: 1015ms):
Trace[1778260582]: [1.015801157s] [1.015801157s] END
E1101 20:22:12.007821    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:12.167759    2586 trace.go:205] Trace[371960321]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:18bcc5f9-5a9f-4718-a4a1-374a6724993b (01-Nov-2021 20:22:11.135) (total time: 1032ms):
Trace[371960321]: [1.032557553s] [1.032557553s] END
I1101 20:22:12.167820    2586 trace.go:205] Trace[809525899]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:11.120) (total time: 1047ms):
Trace[809525899]: [1.047428316s] [1.047428316s] END
E1101 20:22:12.167861    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:12.263763    2586 trace.go:205] Trace[1130714541]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1beta1,object-count:1,UID:afd21203-c447-4947-a26d-f5f63739415f (01-Nov-2021 20:22:11.233) (total time: 1030ms):
Trace[1130714541]: [1.030704776s] [1.030704776s] END
I1101 20:22:12.263820    2586 trace.go:205] Trace[1761687656]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:11.216) (total time: 1047ms):
Trace[1761687656]: [1.047644547s] [1.047644547s] END
E1101 20:22:12.263857    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1beta1, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:12.295717    2586 trace.go:205] Trace[1483443847]: "Call conversion webhook" custom-resource-definition:certificates.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:6394dbb6-8720-4b5f-ab88-3c8dc1cbe3cb (01-Nov-2021 20:22:11.280) (total time: 1015ms):
Trace[1483443847]: [1.015077728s] [1.015077728s] END
I1101 20:22:12.295790    2586 trace.go:205] Trace[2072856063]: "List etcd3" key:/cert-manager.io/certificates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:11.216) (total time: 1079ms):
Trace[2072856063]: [1.079034585s] [1.079034585s] END
E1101 20:22:12.295821    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=Certificate: conversion webhook for cert-manager.io/v1, Kind=Certificate failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:12.359768    2586 trace.go:205] Trace[124827904]: "Call conversion webhook" custom-resource-definition:clusterissuers.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:c93d9b83-406e-4c02-b432-7f47f1e6315b (01-Nov-2021 20:22:11.328) (total time: 1031ms):
Trace[124827904]: [1.031165923s] [1.031165923s] END
I1101 20:22:12.359833    2586 trace.go:205] Trace[1767082550]: "Call conversion webhook" custom-resource-definition:certificaterequests.cert-manager.io,desired-api-version:cert-manager.io/v1beta1,object-count:1,UID:25e930a7-b089-4d5b-b4ec-d17b88bdfc06 (01-Nov-2021 20:22:11.329) (total time: 1030ms):
Trace[1767082550]: [1.030607813s] [1.030607813s] END
I1101 20:22:12.359840    2586 trace.go:205] Trace[330647314]: "List etcd3" key:/cert-manager.io/clusterissuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:11.312) (total time: 1047ms):
Trace[330647314]: [1.047389885s] [1.047389885s] END
I1101 20:22:12.359868    2586 trace.go:205] Trace[1705676394]: "List etcd3" key:/cert-manager.io/certificaterequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:11.312) (total time: 1047ms):
Trace[1705676394]: [1.047537947s] [1.047537947s] END
E1101 20:22:12.359886    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=ClusterIssuer: conversion webhook for cert-manager.io/v1, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
E1101 20:22:12.359892    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1beta1, Kind=CertificateRequest: conversion webhook for cert-manager.io/v1, Kind=CertificateRequest failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:12.455750    2586 trace.go:205] Trace[21699309]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha2,object-count:1,UID:7b2060f7-666a-47a1-a518-5318d2749106 (01-Nov-2021 20:22:11.426) (total time: 1029ms):
Trace[21699309]: [1.029154004s] [1.029154004s] END
I1101 20:22:12.455832    2586 trace.go:205] Trace[789014930]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:11.412) (total time: 1043ms):
Trace[789014930]: [1.043196565s] [1.043196565s] END
E1101 20:22:12.455879    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha2, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:12.935737    2586 trace.go:205] Trace[261816471]: "Call conversion webhook" custom-resource-definition:issuers.cert-manager.io,desired-api-version:cert-manager.io/v1alpha3,object-count:1,UID:394bca60-14b6-4dad-a9b7-6ab39255d656 (01-Nov-2021 20:22:11.934) (total time: 1001ms):
Trace[261816471]: [1.001032752s] [1.001032752s] END
I1101 20:22:12.935811    2586 trace.go:205] Trace[1025295560]: "List etcd3" key:/cert-manager.io/issuers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:11.919) (total time: 1015ms):
Trace[1025295560]: [1.015886273s] [1.015886273s] END
E1101 20:22:12.935843    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list cert-manager.io/v1alpha3, Kind=Issuer: conversion webhook for cert-manager.io/v1, Kind=Issuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...
I1101 20:22:12.939734    2586 trace.go:205] Trace[425567092]: "Call conversion webhook" custom-resource-definition:orders.acme.cert-manager.io,desired-api-version:acme.cert-manager.io/v1alpha3,object-count:1,UID:5cc04432-6dd8-475f-b181-1ce1bf6a36b3 (01-Nov-2021 20:22:11.904) (total time: 1035ms):
Trace[425567092]: [1.035350026s] [1.035350026s] END
I1101 20:22:12.939790    2586 trace.go:205] Trace[1841862924]: "List etcd3" key:/acme.cert-manager.io/orders,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Nov-2021 20:22:11.891) (total time: 1047ms):
Trace[1841862924]: [1.047933016s] [1.047933016s] END
E1101 20:22:12.939823    2586 cacher.go:419] cacher (*unstructured.Unstructured): unexpected ListAndWatch error: failed to list acme.cert-manager.io/v1alpha3, Kind=Order: conversion webhook for acme.cert-manager.io/v1, Kind=Order failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": dial tcp 10.43.207.70:443: connect: connection refused; reinitializing...

core dns should be running

kubectl get pods --namespace=kube-system -l k8s-app=kube-dns

NAME READY STATUS RESTARTS AGE
coredns-7448499f4d-plr2q 1/1 Running 0 85d

kubectl describe pod  coredns-7448499f4d-plr2q --namespace=kube-system
Name:                 coredns-7448499f4d-plr2q
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 upspot-cluster/***.***.***.***
Start Time:           Sun, 08 Aug 2021 12:36:18 +0000
Labels:               k8s-app=kube-dns
                      pod-template-hash=7448499f4d
Annotations:          <none>
Status:               Running
IP:                   10.42.0.76
IPs:
  IP:           10.42.0.76
Controlled By:  ReplicaSet/coredns-7448499f4d
Containers:
  coredns:
    Container ID:  containerd://0620e642d29c8014e130153dfb461d4f2ed6c66d18d3591b69d94289288aa53e
    Image:         rancher/coredns-coredns:1.8.3
    Image ID:      docker.io/rancher/coredns-coredns@sha256:cd0f6ef34a9ee6145c0bcbc0a4511a04f81dc2c91582140f0f0250be834b5f27
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Running
      Started:      Mon, 01 Nov 2021 18:38:38 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=2s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rwbk8 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  kube-api-access-rwbk8:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              beta.kubernetes.io/os=linux
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                             node-role.kubernetes.io/master:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                   From             Message
  ----     ------                  ----                  ----             -------
  Warning  FailedCreatePodSandBox  31h                   kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "51c9ae6a65d704a3a4040261f56158523d29dfd1292e3b0cbade6d9e9bd579e4": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31h                   kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b72d7dbff055840ff68f3d364795d23ff6235e5db889131e20b8241f6138e8ab": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31h                   kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ca3f773654508caddcc648cda24d9fe10e34547b9bb515860ffe266915b4b4c4": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31h                   kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d01b13574e05d65dcb8656b1c7041356f93510711068d21572071525c94d6000": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31h                   kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f570ffd13e461cdc0853a990281617c5091cc4e004a28c9ae67c8b3d26f098ae": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31h                   kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fe6a0c549e77626942ad9e35ac4562ed95cb1508cb56c2d625ff5ca2386e982e": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31h                   kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ab0b2c71df867d1e95351b8841ad89ff75266fc4362e370b7aa0aef28bb10d33": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31h                   kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "41da419b6c354b515f27b35548c4a95c4fc9ea9e064317126ebce7690d243332": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31h                   kubelet          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ed13b48e3715625a62f60f25d487790075deeec66d85998a227f54fef954be3e": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31h (x3 over 31h)     kubelet          (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0e7febedb6943de8b7c295cc10afdafb0c1f848aeb85f4cc8b0ece0159eb14ef": open /run/flannel/subnet.env: no such file or directory
  Normal   SandboxChanged          27h (x1149 over 31h)  kubelet          Pod sandbox changed, it will be killed and re-created.
  Warning  FailedMount             27h                   kubelet          MountVolume.SetUp failed for volume "config-volume" : failed to sync configmap cache: timed out waiting for the condition
  Normal   Created                 27h                   kubelet          Created container coredns
  Normal   Pulled                  27h                   kubelet          Container image "rancher/coredns-coredns:1.8.3" already present on machine
  Normal   Started                 27h                   kubelet          Started container coredns
  Warning  NodeNotReady            27h                   node-controller  Node is not ready
  Normal   Pulled                  27h                   kubelet          Container image "rancher/coredns-coredns:1.8.3" already present on machine
  Normal   Created                 27h                   kubelet          Created container coredns
  Normal   Started                 27h                   kubelet          Started container coredns
  Warning  Unhealthy               27h                   kubelet          Readiness probe failed: Get "http://10.42.0.29:8181/ready": dial tcp 10.42.0.29:8181: connect: connection refused
  Warning  NodeNotReady            131m                  node-controller  Node is not ready
  Normal   Pulled                  125m                  kubelet          Container image "rancher/coredns-coredns:1.8.3" already present on machine
  Normal   Created                 125m                  kubelet          Created container coredns
  Normal   Started                 125m                  kubelet          Started container coredns

kubectl describe nodes
Name:               upspot-cluster
Roles:              control-plane,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=k3s
                    beta.kubernetes.io/os=linux
                    k3os.io/mode=local
                    k3os.io/version=v0.21.1-k3s1r0
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=upspot-cluster
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=true
                    node-role.kubernetes.io/master=true
                    node.kubernetes.io/instance-type=k3s
Annotations:        flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"***"}
                    flannel.alpha.coreos.com/backend-type: vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: *.*.*.*
                    k3s.io/hostname: upspot-cluster
                    k3s.io/internal-ip: *.*.*.*
                    k3s.io/node-args: ["server","--no-deploy","traefik","--node-label","k3os.io/mode=local","--node-label","k3os.io/version=v0.21.1-k3s1r0"]
                    k3s.io/node-config-hash: L3CO4RG3V7U4SLPXOQEKR6RYZMQJODZUCK2BVNSCXA7HTV36QU7Q====
                    k3s.io/node-env:
                      {"K3S_CLUSTER_SECRET":"********","K3S_DATA_DIR":"/var/lib/rancher/k3s/data/77457d0b09d8b94d6f5029bcbc70f94b7ae9c50a08b539b76612e713ea81825...
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sun, 31 Oct 2021 12:51:56 +0000
Taints:             node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  upspot-cluster
  AcquireTime:     <unset>
  RenewTime:       Mon, 01 Nov 2021 20:56:50 +0000
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Mon, 01 Nov 2021 20:50:45 +0000   Mon, 01 Nov 2021 20:50:45 +0000   FlannelIsUp                  Flannel is running on this node
  MemoryPressure       False   Mon, 01 Nov 2021 20:56:51 +0000   Mon, 01 Nov 2021 20:49:42 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Mon, 01 Nov 2021 20:56:51 +0000   Mon, 01 Nov 2021 20:49:42 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Mon, 01 Nov 2021 20:56:51 +0000   Mon, 01 Nov 2021 20:49:42 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                False   Mon, 01 Nov 2021 20:56:51 +0000   Mon, 01 Nov 2021 20:50:45 +0000   KubeletNotReady              PLEG is not healthy: pleg has yet to be successful
Addresses:
  InternalIP:  *.*.*.*
  Hostname:    upspot-cluster
Capacity:
  cpu:                8
  ephemeral-storage:  827392920Ki
  hugepages-2Mi:      0
  memory:             32886628Ki
  pods:               110
Allocatable:
  cpu:                8
  ephemeral-storage:  804887831945
  hugepages-2Mi:      0
  memory:             32886628Ki
  pods:               110
System Info:
  Machine ID:                 4a9d40c633f866838c2c04d460b91f75
  System UUID:                0a0f4d22-8e23-45b4-aa30-79f8b85f2989
  Boot ID:                    9773ef9f-9974-4931-b20e-b3e90fe51216
  Kernel Version:             5.4.0-73-generic
  OS Image:                   k3OS v0.21.1-k3s1r0
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.4.4-k3s2
  Kubelet Version:            v1.21.1+k3s1
  Kube-Proxy Version:         v1.21.1+k3s1
PodCIDR:                      10.42.0.0/24
PodCIDRs:                     10.42.0.0/24
ProviderID:                   k3s://upspot-cluster
Non-terminated Pods:          (55 in total)
  Namespace                   Name                                                      CPU Requests  CPU Limits   Memory Requests  Memory Limits  Age
  ---------                   ----                                                      ------------  ----------   ---------------  -------------  ---
  default                     svclb-ingress-nginx-controller-n9vvp                      0 (0%)        0 (0%)       0 (0%)           0 (0%)         86d
  default                     ingress-nginx-controller-68649d49b8-g25vh                 100m (1%)     0 (0%)       90Mi (0%)        0 (0%)         85d
  test                        order-management-api-postgresql-postgresql-0              250m (3%)     0 (0%)       256Mi (0%)       0 (0%)         3d18h
  test                        document-management-api-mongodb-5f46494477-nqzbl          0 (0%)        0 (0%)       0 (0%)           0 (0%)         16d
  test                        notification-management-api-mongodb-8458655556-m86xx      0 (0%)        0 (0%)       0 (0%)           0 (0%)         16d
  test                        document-management-api-7b6664966f-2grtb                  0 (0%)        0 (0%)       0 (0%)           0 (0%)         16d
  test                        order-management-api-axon-5c4f84d668-njwzh                0 (0%)        0 (0%)       0 (0%)           0 (0%)         13d
  keycloak                    keycloak-postgresql-0                                     250m (3%)     0 (0%)       256Mi (0%)       0 (0%)         3d18h
  test                        order-management-api-mongodb-774f8549cd-cnvd6             0 (0%)        0 (0%)       0 (0%)           0 (0%)         13d
  test                        user-management-api-mongodb-7cfcc95977-4jr6q              0 (0%)        0 (0%)       0 (0%)           0 (0%)         3d18h
  cattle-resources-system     rancher-backup-d9d984f74-h6cn7                            0 (0%)        0 (0%)       0 (0%)           0 (0%)         16d
  longhorn-system             csi-snapshotter-69f8bc8dcf-dh85b                          0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  longhorn-system             csi-resizer-5c88bfd4cf-tfqf2                              0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  kube-system                 coredns-7448499f4d-plr2q                                  100m (1%)     0 (0%)       70Mi (0%)        170Mi (0%)     85d
  rancher-operator-system     rancher-operator-595ddc6db9-w4jsj                         0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  longhorn-system             csi-attacher-75588bff58-pdmt7                             0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  longhorn-system             longhorn-driver-deployer-5fc857649c-mgp8m                 0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  cattle-monitoring-system    rancher-monitoring-operator-9779869d-6r4mh                100m (1%)     200m (2%)    100Mi (0%)       500Mi (1%)     33d
  longhorn-system             longhorn-manager-ll268                                    0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  longhorn-system             engine-image-ei-0f7c4304-d5l2p                            0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  cert-manager                cert-manager-webhook-7dc86765b4-kdsvv                     0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  test                        planung-ui-959b9d47c-x8djr                                0 (0%)        0 (0%)       0 (0%)           0 (0%)         3d8h
  cattle-monitoring-system    alertmanager-rancher-monitoring-alertmanager-0            200m (2%)     1100m (13%)  150Mi (0%)       550Mi (1%)     33d
  cattle-monitoring-system    pushprox-k3s-server-client-qwhvq                          0 (0%)        0 (0%)       0 (0%)           0 (0%)         33d
  longhorn-system             csi-resizer-5c88bfd4cf-ghxfz                              0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  kube-system                 metrics-server-86cbb8457f-tkq6f                           0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  fleet-system                fleet-controller-79554fcbf5-9zw2d                         0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  cattle-monitoring-system    rancher-monitoring-grafana-6c9976ccc6-njtt6               100m (1%)     200m (2%)    100Mi (0%)       200Mi (0%)     33d
  longhorn-system             engine-image-ei-a5a44787-7q78r                            0 (0%)        0 (0%)       0 (0%)           0 (0%)         86d
  cert-manager                cert-manager-cainjector-595fc4d55-ljzn2                   0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  cattle-system               rancher-webhook-6cccfd96b5-v5v24                          0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  longhorn-system             csi-snapshotter-69f8bc8dcf-cjvnk                          0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  longhorn-system             csi-attacher-75588bff58-rr57b                             0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  longhorn-system             csi-provisioner-6968cf94f9-wlrql                          0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  kube-system                 local-path-provisioner-5ff76fc89d-d4pdw                   0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  longhorn-system             csi-provisioner-6968cf94f9-fb894                          0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  longhorn-system             csi-provisioner-6968cf94f9-vgxks                          0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  cert-manager                cert-manager-58bbf5798-cs5x2                              0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  cattle-monitoring-system    pushprox-k3s-server-proxy-6946957f87-r75k8                0 (0%)        0 (0%)       0 (0%)           0 (0%)         33d
  longhorn-system             csi-attacher-75588bff58-rlk42                             0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  longhorn-system             csi-snapshotter-69f8bc8dcf-wcfbz                          0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  cattle-monitoring-system    rancher-monitoring-kube-state-metrics-5f56fc44ff-vfp4c    100m (1%)     100m (1%)    130Mi (0%)       200Mi (0%)     33d
  longhorn-system             csi-resizer-5c88bfd4cf-mqwj4                              0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  fleet-system                gitjob-568c57cfb9-c4mww                                   0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  test                        notification-management-api-bd7d67c5d-wfs4q               0 (0%)        0 (0%)       0 (0%)           0 (0%)         35d
  fleet-system                fleet-agent-d59db746-h7nqt                                0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  cattle-monitoring-system    rancher-monitoring-prometheus-node-exporter-rmz2w         100m (1%)     200m (2%)    30Mi (0%)        50Mi (0%)      33d
  cattle-monitoring-system    rancher-monitoring-prometheus-adapter-58d5b96d85-wwh45    0 (0%)        0 (0%)       0 (0%)           0 (0%)         33d
  cattle-system               rancher-6d8598477d-h8cf2                                  0 (0%)        0 (0%)       0 (0%)           0 (0%)         85d
  cattle-monitoring-system    prometheus-rancher-monitoring-prometheus-0                850m (10%)    1100m (13%)  1800Mi (5%)      2550Mi (7%)    33d
  longhorn-system             longhorn-ui-64b96684bc-8jbmk                              0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  keycloak                    keycloak-0                                                100m (1%)     0 (0%)       0 (0%)           0 (0%)         15d
  longhorn-system             longhorn-csi-plugin-gxqb2                                 0 (0%)        0 (0%)       0 (0%)           0 (0%)         36d
  test                        user-management-api-76c74dc97b-bxjtk                      0 (0%)        0 (0%)       0 (0%)           0 (0%)         25d
  test                        order-managament-api-7d654d8d84-jvnf5                     0 (0%)        0 (0%)       0 (0%)           0 (0%)         13d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests     Limits
  --------           --------     ------
  cpu                2250m (28%)  2900m (36%)
  memory             2982Mi (9%)  4220Mi (13%)
  ephemeral-storage  0 (0%)       0 (0%)
  hugepages-2Mi      0 (0%)       0 (0%)
Events:
  Type     Reason                   Age                     From        Message
  ----     ------                   ----                    ----        -------
  Warning  InvalidDiskCapacity      32h                     kubelet     invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  32h                     kubelet     Updated Node Allocatable limit across pods
  Normal   Starting                 32h                     kubelet     Starting kubelet.
  Normal   NodeHasNoDiskPressure    32h (x7 over 32h)       kubelet     Node upspot-cluster status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     32h (x7 over 32h)       kubelet     Node upspot-cluster status is now: NodeHasSufficientPID
  Normal   Starting                 32h                     kube-proxy  Starting kube-proxy.
  Normal   NodeHasSufficientMemory  27h (x105183 over 32h)  kubelet     Node upspot-cluster status is now: NodeHasSufficientMemory
  Normal   Starting                 27h                     kube-proxy  Starting kube-proxy.
  Normal   Starting                 27h                     kubelet     Starting kubelet.
  Warning  InvalidDiskCapacity      27h                     kubelet     invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  27h                     kubelet     Updated Node Allocatable limit across pods
  Normal   NodeReady                27h (x3 over 27h)       kubelet     Node upspot-cluster status is now: NodeReady
  Normal   NodeHasSufficientMemory  27h (x5 over 27h)       kubelet     Node upspot-cluster status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    27h (x5 over 27h)       kubelet     Node upspot-cluster status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     27h (x5 over 27h)       kubelet     Node upspot-cluster status is now: NodeHasSufficientPID
  Warning  Rebooted                 27h (x92 over 27h)      kubelet     Node upspot-cluster has been rebooted, boot id: 7cb8d273-5e32-47f1-9914-cc731dc83bda
  Normal   Starting                 27h                     kube-proxy  Starting kube-proxy.
  Normal   Starting                 27h                     kubelet     Starting kubelet.
  Warning  InvalidDiskCapacity      27h                     kubelet     invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  27h                     kubelet     Updated Node Allocatable limit across pods
  Normal   NodeNotReady             27h (x2 over 27h)       kubelet     Node upspot-cluster status is now: NodeNotReady
  Normal   NodeReady                27h                     kubelet     Node upspot-cluster status is now: NodeReady
  Normal   NodeHasSufficientPID     27h (x5 over 27h)       kubelet     Node upspot-cluster status is now: NodeHasSufficientPID
  Normal   NodeHasNoDiskPressure    27h (x5 over 27h)       kubelet     Node upspot-cluster status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientMemory  27h (x5 over 27h)       kubelet     Node upspot-cluster status is now: NodeHasSufficientMemory
  Warning  Rebooted                 26h (x211 over 27h)     kubelet     Node upspot-cluster has been rebooted, boot id: f9d367b9-bf7a-4998-886f-3fb4fc1a83be
  Normal   Starting                 138m                    kube-proxy  Starting kube-proxy.
  Normal   Starting                 138m                    kubelet     Starting kubelet.
  Warning  InvalidDiskCapacity      138m                    kubelet     invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  138m                    kubelet     Updated Node Allocatable limit across pods
  Normal   NodeNotReady             138m (x2 over 138m)     kubelet     Node upspot-cluster status is now: NodeNotReady
  Normal   NodeReady                138m                    kubelet     Node upspot-cluster status is now: NodeReady
  Normal   NodeHasSufficientMemory  138m (x5 over 138m)     kubelet     Node upspot-cluster status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    138m (x5 over 138m)     kubelet     Node upspot-cluster status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     138m (x5 over 138m)     kubelet     Node upspot-cluster status is now: NodeHasSufficientPID
  Warning  Rebooted                 8m22s (x777 over 138m)  kubelet     Node upspot-cluster has been rebooted, boot id: 1f0cbe8e-20a6-4c87-ab20-5dd1a4a8fc68
  Normal   Starting                 6m13s                   kube-proxy  Starting kube-proxy.
  Normal   Starting                 6m9s                    kubelet     Starting kubelet.
  Warning  InvalidDiskCapacity      6m9s                    kubelet     invalid capacity 0 on image filesystem
  Normal   NodeHasNoDiskPressure    6m9s                    kubelet     Node upspot-cluster status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     6m9s                    kubelet     Node upspot-cluster status is now: NodeHasSufficientPID
  Normal   NodeAllocatableEnforced  6m9s                    kubelet     Updated Node Allocatable limit across pods
  Normal   NodeNotReady             6m9s (x2 over 6m9s)     kubelet     Node upspot-cluster status is now: NodeNotReady
  Normal   NodeHasSufficientMemory  6m9s                    kubelet     Node upspot-cluster status is now: NodeHasSufficientMemory
  Warning  Rebooted                 64s (x32 over 6m9s)     kubelet     Node upspot-cluster has been rebooted, boot id: 9773ef9f-9974-4931-b20e-b3e90fe51216

patrik · November 2, 2021, 7:07am

For your interest: I have a snapshot from the VPS from beginning september. If i use this one all works fine, but then i missing my longhorn volumes. They are on the same VPS without correct backupsystem. But if i have to switch back to september to check something i can do it. but i have no idea what i have to do :-/

Topic		Replies	Views
Liveness probe failed connection refused Rancher	16	28814	November 23, 2022
Cluster api access is stuck on a missing node Rancher	4	1237	April 15, 2022
Cattle-pods failing Rancher	2	1772	October 25, 2019
Rancher 2.5 Migration issues Rancher	2	766	June 17, 2021
Rancher 2.7 not working on K3S with metallb and nginx ingress Rancher	2	623	January 25, 2023

A lot of Issues after Server-Downtime

Related topics