Some of the relevant output:
# kubectl -n cattle-system get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
rancher ClusterIP 10.43.202.96 <none> 80/TCP,443/TCP 29h app=rancher
rancher-webhook ClusterIP 10.43.55.247 <none> 443/TCP 29h app=rancher-webhook
Rancher container 1 log errors:
2023/12/25 18:24:25 [ERROR] Failed to handle tunnel request from remote address 10.42.1.9:36228: response 400: cluster not found
2023/12/25 18:24:25 [ERROR] Failed to handle tunnel request from remote address 10.42.0.9:45522: response 400: cluster not found
2023/12/25 18:24:26 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=kube-public, err=Operation cannot be fulfilled on namespaces "kube-public": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-system, err=Operation cannot be fulfilled on namespaces "cattle-system": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-fleet-system, err=Operation cannot be fulfilled on namespaces "cattle-fleet-system": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cert-manager, err=Operation cannot be fulfilled on namespaces "cert-manager": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-bktxc, err=Operation cannot be fulfilled on namespaces "p-bktxc": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-j92fh, err=Operation cannot be fulfilled on namespaces "p-j92fh": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-impersonation-system, err=Operation cannot be fulfilled on namespaces "cattle-impersonation-system": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-global-data, err=Operation cannot be fulfilled on namespaces "cattle-global-data": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=kube-node-lease, err=Operation cannot be fulfilled on namespaces "kube-node-lease": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-global-nt, err=Operation cannot be fulfilled on namespaces "cattle-global-nt": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:27 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-j92fh, err=Operation cannot be fulfilled on namespaces "p-j92fh": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:27 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=fleet-local, err=Operation cannot be fulfilled on namespaces "fleet-local": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:27 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-global-data, err=Operation cannot be fulfilled on namespaces "cattle-global-data": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:27 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-impersonation-system, err=Operation cannot be fulfilled on namespaces "cattle-impersonation-system": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:27 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-impersonation-system, err=Operation cannot be fulfilled on namespaces "cattle-impersonation-system": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:30 [ERROR] Failed to serve peer connection 10.42.0.9: websocket: close 1006 (abnormal closure): unexpected EOF
2023/12/25 18:25:39 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cluster-fleet-local-local-1a3d67d0a899, err=Operation cannot be fulfilled on namespaces "cluster-fleet-local-local-1a3d67d0a899": the object has been modified; please apply your changes to the latest version and try again
Rancher container 2 log errors:
2023/12/25 18:24:09 [ERROR] Failed to find system chart fleet will try again in 5 seconds: configmaps "" not found
2023/12/25 18:24:09 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2023/12/25 18:24:09 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2023/12/25 18:24:10 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2023/12/25 18:24:10 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2023/12/25 18:24:10 [ERROR] error syncing 'fleet-local/local': handler workspace-backport-cluster: fleetworkspaces.management.cattle.io "fleet-local" not found, requeuing
2023/12/25 18:24:10 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2023/12/25 18:24:11 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2023/12/25 18:24:11 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2023/12/25 18:24:13 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2023/12/25 18:24:14 [ERROR] Failed to find system chart fleet will try again in 5 seconds: configmaps "" not found
2023/12/25 18:24:16 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2023/12/25 18:24:19 [ERROR] Failed to find system chart fleet will try again in 5 seconds: configmaps "" not found
2023/12/25 18:24:21 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2023/12/25 18:24:25 [ERROR] Failed to connect to peer wss://10.42.2.9/v3/connect [local ID=10.42.1.9]: websocket: bad handshake
2023/12/25 18:24:25 [ERROR] Failed to connect to peer wss://10.42.0.9/v3/connect [local ID=10.42.1.9]: websocket: bad handshake
2023/12/25 18:24:26 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "kube-public": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-fleet-system": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cert-manager": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "fleet-default": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "fleet-local": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cert-manager": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:27 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "kube-system": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:27 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-system": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:53 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-fleet-clusters-system": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:53 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-fleet-clusters-system": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:25:39 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cluster-fleet-local-local-1a3d67d0a899": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:25:39 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cluster-fleet-local-local-1a3d67d0a899": the object has been modified; please apply your changes to the latest version and try again
Traefik logs:
time="2023-12-25T18:01:55Z" level=info msg="Configuration loaded from flags."
time="2023-12-25T18:22:25Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:22:25Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes ingress=rancher namespace=cattle-system serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}"
time="2023-12-25T18:22:25Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:22:25Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes ingress=rancher namespace=cattle-system serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}"
time="2023-12-25T18:22:25Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:22:25Z" level=error msg="Skipping service: no endpoints found" ingress=rancher namespace=cattle-system serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes
time="2023-12-25T18:23:38Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:23:38Z" level=error msg="Skipping service: no endpoints found" serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:23:39Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:23:39Z" level=error msg="Skipping service: no endpoints found" serviceName=rancher providerName=kubernetes ingress=rancher namespace=cattle-system servicePort="&ServiceBackendPort{Name:,Number:80,}"
time="2023-12-25T18:23:39Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" ingress=rancher namespace=cattle-system providerName=kubernetes
time="2023-12-25T18:23:39Z" level=error msg="Skipping service: no endpoints found" ingress=rancher namespace=cattle-system providerName=kubernetes serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}"
time="2023-12-25T18:24:08Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes namespace=cattle-system ingress=rancher
time="2023-12-25T18:24:08Z" level=error msg="Skipping service: no endpoints found" serviceName=rancher providerName=kubernetes servicePort="&ServiceBackendPort{Name:,Number:80,}" namespace=cattle-system ingress=rancher
time="2023-12-25T18:24:08Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:08Z" level=error msg="Skipping service: no endpoints found" ingress=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}" namespace=cattle-system serviceName=rancher providerName=kubernetes
time="2023-12-25T18:24:08Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:08Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes ingress=rancher namespace=cattle-system serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}"
time="2023-12-25T18:24:08Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:24:08Z" level=error msg="Skipping service: no endpoints found" servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=rancher namespace=cattle-system serviceName=rancher
time="2023-12-25T18:24:09Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:09Z" level=error msg="Skipping service: no endpoints found" ingress=rancher namespace=cattle-system serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes
time="2023-12-25T18:24:09Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:24:09Z" level=error msg="Skipping service: no endpoints found" servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=rancher namespace=cattle-system serviceName=rancher
time="2023-12-25T18:24:09Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:24:09Z" level=error msg="Skipping service: no endpoints found" ingress=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}" namespace=cattle-system serviceName=rancher providerName=kubernetes
time="2023-12-25T18:24:09Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:09Z" level=error msg="Skipping service: no endpoints found" servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=rancher namespace=cattle-system serviceName=rancher
time="2023-12-25T18:24:09Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes namespace=cattle-system ingress=rancher
time="2023-12-25T18:24:09Z" level=error msg="Skipping service: no endpoints found" servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes namespace=cattle-system ingress=rancher serviceName=rancher
time="2023-12-25T18:24:09Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" ingress=rancher namespace=cattle-system providerName=kubernetes
time="2023-12-25T18:24:09Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes ingress=rancher namespace=cattle-system serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}"
time="2023-12-25T18:24:10Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:10Z" level=error msg="Skipping service: no endpoints found" servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=rancher namespace=cattle-system serviceName=rancher
time="2023-12-25T18:24:24Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes namespace=cattle-system ingress=rancher
time="2023-12-25T18:24:24Z" level=error msg="Skipping service: no endpoints found" ingress=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}" serviceName=rancher providerName=kubernetes namespace=cattle-system
time="2023-12-25T18:24:24Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:24Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes ingress=rancher namespace=cattle-system servicePort="&ServiceBackendPort{Name:,Number:80,}" serviceName=rancher
time="2023-12-25T18:24:25Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:25Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" ingress=rancher namespace=cattle-system providerName=kubernetes
time="2023-12-25T18:24:25Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes namespace=cattle-system ingress=rancher
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" ingress=rancher namespace=cattle-system providerName=kubernetes
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" ingress=rancher namespace=cattle-system providerName=kubernetes
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" ingress=rancher namespace=cattle-system providerName=kubernetes
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" ingress=rancher namespace=cattle-system providerName=kubernetes
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" ingress=rancher namespace=cattle-system providerName=kubernetes
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" ingress=rancher namespace=cattle-system providerName=kubernetes
time="2023-12-25T18:24:27Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:27Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:27Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:27Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:31Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" ingress=rancher namespace=cattle-system providerName=kubernetes
time="2023-12-25T18:24:43Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:43Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
Cert-Manager logs:
I1225 18:15:52.856071 1 start.go:75] "cert-manager: starting controller" version="v1.12.7" git-commit="6d7629ba42b946978e3baaa75348c851f7ef9134"
I1225 18:15:52.856433 1 controller.go:262] "cert-manager/controller/build-context: configured acme dns01 nameservers" nameservers=["10.43.0.10:53"]
W1225 18:15:52.856680 1 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I1225 18:15:52.859186 1 controller.go:82] "cert-manager/controller: enabled controllers: [certificaterequests-approver certificaterequests-issuer-acme certificaterequests-issuer-ca certificaterequests-issuer-selfsigned certificaterequests-issuer-vault certificaterequests-issuer-venafi certificates-issuing certificates-key-manager certificates-metrics certificates-readiness certificates-request-manager certificates-revision-manager certificates-trigger challenges clusterissuers ingress-shim issuers orders]"
I1225 18:15:52.860053 1 controller.go:103] "cert-manager/controller: starting metrics server" address="[::]:9402"
I1225 18:15:52.860081 1 controller.go:156] "cert-manager/controller: starting leader election"
I1225 18:15:52.860570 1 controller.go:149] "cert-manager/controller: starting healthz server" address="[::]:9403"
I1225 18:15:52.861201 1 leaderelection.go:245] attempting to acquire leader lease kube-system/cert-manager-controller...
I1225 18:17:05.785099 1 leaderelection.go:255] successfully acquired lease kube-system/cert-manager-controller
I1225 18:17:05.786668 1 controller.go:226] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-selfsigned"
I1225 18:17:05.787805 1 controller.go:226] "cert-manager/controller: starting controller" controller="certificates-key-manager"
I1225 18:17:05.790161 1 controller.go:226] "cert-manager/controller: starting controller" controller="certificates-readiness"
I1225 18:17:05.792116 1 controller.go:203] "cert-manager/controller: not starting controller as it's disabled" controller="certificatesigningrequests-issuer-ca"
I1225 18:17:05.795483 1 controller.go:226] "cert-manager/controller: starting controller" controller="certificates-trigger"
I1225 18:17:05.795716 1 controller.go:226] "cert-manager/controller: starting controller" controller="orders"
I1225 18:17:05.796786 1 controller.go:226] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-acme"
I1225 18:17:05.798333 1 controller.go:203] "cert-manager/controller: not starting controller as it's disabled" controller="certificatesigningrequests-issuer-selfsigned"
I1225 18:17:05.798408 1 controller.go:226] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-ca"
I1225 18:17:05.799565 1 controller.go:226] "cert-manager/controller: starting controller" controller="ingress-shim"
I1225 18:17:05.800619 1 controller.go:226] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-vault"
I1225 18:17:05.801595 1 controller.go:226] "cert-manager/controller: starting controller" controller="certificates-metrics"
I1225 18:17:05.802176 1 controller.go:203] "cert-manager/controller: not starting controller as it's disabled" controller="gateway-shim"
I1225 18:17:05.802244 1 controller.go:226] "cert-manager/controller: starting controller" controller="issuers"
I1225 18:17:05.802999 1 controller.go:226] "cert-manager/controller: starting controller" controller="certificates-issuing"
I1225 18:17:05.804079 1 controller.go:226] "cert-manager/controller: starting controller" controller="clusterissuers"
I1225 18:17:05.804771 1 controller.go:226] "cert-manager/controller: starting controller" controller="certificates-request-manager"
I1225 18:17:05.805435 1 controller.go:203] "cert-manager/controller: not starting controller as it's disabled" controller="certificatesigningrequests-issuer-acme"
I1225 18:17:05.805480 1 controller.go:203] "cert-manager/controller: not starting controller as it's disabled" controller="certificatesigningrequests-issuer-vault"
I1225 18:17:05.805500 1 controller.go:203] "cert-manager/controller: not starting controller as it's disabled" controller="certificatesigningrequests-issuer-venafi"
I1225 18:17:05.805611 1 controller.go:226] "cert-manager/controller: starting controller" controller="certificates-revision-manager"
I1225 18:17:05.806494 1 controller.go:226] "cert-manager/controller: starting controller" controller="challenges"
I1225 18:17:05.806997 1 controller.go:226] "cert-manager/controller: starting controller" controller="certificaterequests-approver"
I1225 18:17:05.809476 1 controller.go:226] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-venafi"
E1225 18:22:25.341481 1 setup.go:48] "cert-manager/issuers/setup: error getting signing CA TLS certificate" err="secret \"tls-rancher\" not found" resource_name="rancher" resource_namespace="cattle-system" resource_kind="Issuer" resource_version="v1"
I1225 18:22:25.341591 1 conditions.go:96] Setting lastTransitionTime for Issuer "rancher" condition "Ready" to 2023-12-25 18:22:25.34155498 +0000 UTC m=+392.557170883
I1225 18:22:25.341691 1 sync.go:62] "cert-manager/issuers: Error initializing issuer: secret \"tls-rancher\" not found" resource_name="rancher" resource_namespace="cattle-system" resource_kind="Issuer" resource_version="v1"
I1225 18:22:25.345121 1 conditions.go:203] Setting lastTransitionTime for Certificate "tls-rancher-ingress" condition "Ready" to 2023-12-25 18:22:25.34510582 +0000 UTC m=+392.560721757
I1225 18:22:25.345180 1 trigger_controller.go:194] "cert-manager/certificates-trigger: Certificate must be re-issued" key="cattle-system/tls-rancher-ingress" reason="DoesNotExist" message="Issuing certificate as Secret does not exist"
I1225 18:22:25.345378 1 conditions.go:203] Setting lastTransitionTime for Certificate "tls-rancher-ingress" condition "Issuing" to 2023-12-25 18:22:25.345365667 +0000 UTC m=+392.560981597
E1225 18:22:25.479134 1 controller.go:167] "cert-manager/issuers: re-queuing item due to error processing" err="secret \"tls-rancher\" not found" key="cattle-system/rancher"
E1225 18:22:25.479269 1 setup.go:48] "cert-manager/issuers/setup: error getting signing CA TLS certificate" err="secret \"tls-rancher\" not found" resource_name="rancher" resource_namespace="cattle-system" resource_kind="Issuer" resource_version="v1"
I1225 18:22:25.479309 1 sync.go:62] "cert-manager/issuers: Error initializing issuer: secret \"tls-rancher\" not found" resource_name="rancher" resource_namespace="cattle-system" resource_kind="Issuer" resource_version="v1"
E1225 18:22:25.479389 1 controller.go:167] "cert-manager/issuers: re-queuing item due to error processing" err="secret \"tls-rancher\" not found" key="cattle-system/rancher"
E1225 18:22:25.479959 1 controller.go:167] "cert-manager/ingress-shim: re-queuing item due to error processing" err="certificates.cert-manager.io \"tls-rancher-ingress\" already exists" key="cattle-system/rancher"
I1225 18:22:25.501217 1 controller.go:162] "cert-manager/certificates-trigger: re-queuing item due to optimistic locking on resource" key="cattle-system/tls-rancher-ingress" error="Operation cannot be fulfilled on certificates.cert-manager.io \"tls-rancher-ingress\": the object has been modified; please apply your changes to the latest version and try again"
I1225 18:22:25.501324 1 trigger_controller.go:194] "cert-manager/certificates-trigger: Certificate must be re-issued" key="cattle-system/tls-rancher-ingress" reason="DoesNotExist" message="Issuing certificate as Secret does not exist"
I1225 18:22:25.501355 1 conditions.go:203] Setting lastTransitionTime for Certificate "tls-rancher-ingress" condition "Issuing" to 2023-12-25 18:22:25.50134574 +0000 UTC m=+392.716961670
I1225 18:22:25.934955 1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "tls-rancher-ingress-55btb" condition "Approved" to 2023-12-25 18:22:25.934940321 +0000 UTC m=+393.150556248
I1225 18:22:25.978798 1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "tls-rancher-ingress-55btb" condition "Ready" to 2023-12-25 18:22:25.978722424 +0000 UTC m=+393.194338301
E1225 18:22:30.479831 1 setup.go:48] "cert-manager/issuers/setup: error getting signing CA TLS certificate" err="secret \"tls-rancher\" not found" resource_name="rancher" resource_namespace="cattle-system" resource_kind="Issuer" resource_version="v1"
I1225 18:22:30.479954 1 sync.go:62] "cert-manager/issuers: Error initializing issuer: secret \"tls-rancher\" not found" resource_name="rancher" resource_namespace="cattle-system" resource_kind="Issuer" resource_version="v1"
E1225 18:22:30.480281 1 controller.go:167] "cert-manager/issuers: re-queuing item due to error processing" err="secret \"tls-rancher\" not found" key="cattle-system/rancher"
E1225 18:22:50.480715 1 setup.go:48] "cert-manager/issuers/setup: error getting signing CA TLS certificate" err="secret \"tls-rancher\" not found" resource_name="rancher" resource_namespace="cattle-system" resource_kind="Issuer" resource_version="v1"
I1225 18:22:50.480787 1 sync.go:62] "cert-manager/issuers: Error initializing issuer: secret \"tls-rancher\" not found" resource_name="rancher" resource_namespace="cattle-system" resource_kind="Issuer" resource_version="v1"
E1225 18:22:50.480958 1 controller.go:167] "cert-manager/issuers: re-queuing item due to error processing" err="secret \"tls-rancher\" not found" key="cattle-system/rancher"
E1225 18:23:30.481497 1 setup.go:48] "cert-manager/issuers/setup: error getting signing CA TLS certificate" err="secret \"tls-rancher\" not found" resource_name="rancher" resource_namespace="cattle-system" resource_kind="Issuer" resource_version="v1"
I1225 18:23:30.481568 1 sync.go:62] "cert-manager/issuers: Error initializing issuer: secret \"tls-rancher\" not found" resource_name="rancher" resource_namespace="cattle-system" resource_kind="Issuer" resource_version="v1"
E1225 18:23:30.481661 1 controller.go:167] "cert-manager/issuers: re-queuing item due to error processing" err="secret \"tls-rancher\" not found" key="cattle-system/rancher"
I1225 18:24:50.483029 1 conditions.go:85] Found status change for Issuer "rancher" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2023-12-25 18:24:50.483013988 +0000 UTC m=+537.698629891
I1225 18:24:50.548773 1 conditions.go:252] Found status change for CertificateRequest "tls-rancher-ingress-55btb" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2023-12-25 18:24:50.54875726 +0000 UTC m=+537.764373220
I1225 18:24:50.601572 1 conditions.go:192] Found status change for Certificate "tls-rancher-ingress" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2023-12-25 18:24:50.601556272 +0000 UTC m=+537.817172158
I1225 18:24:50.637607 1 controller.go:162] "cert-manager/certificates-readiness: re-queuing item due to optimistic locking on resource" key="cattle-system/tls-rancher-ingress" error="Operation cannot be fulfilled on certificates.cert-manager.io \"tls-rancher-ingress\": the object has been modified; please apply your changes to the latest version and try again"
I1225 18:24:50.639429 1 conditions.go:192] Found status change for Certificate "tls-rancher-ingress" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2023-12-25 18:24:50.63941676 +0000 UTC m=+537.855032633
I1225 18:24:50.655151 1 controller.go:162] "cert-manager/certificates-issuing: re-queuing item due to optimistic locking on resource" key="cattle-system/tls-rancher-ingress" error="Operation cannot be fulfilled on certificates.cert-manager.io \"tls-rancher-ingress\": the object has been modified; please apply your changes to the latest version and try again"
I1225 18:24:50.682257 1 controller.go:162] "cert-manager/certificates-key-manager: re-queuing item due to optimistic locking on resource" key="cattle-system/tls-rancher-ingress" error="Operation cannot be fulfilled on certificates.cert-manager.io \"tls-rancher-ingress\": the object has been modified; please apply your changes to the latest version and try again"
I1225 18:24:50.683594 1 controller.go:162] "cert-manager/certificates-readiness: re-queuing item due to optimistic locking on resource" key="cattle-system/tls-rancher-ingress" error="Operation cannot be fulfilled on certificates.cert-manager.io \"tls-rancher-ingress\": the object has been modified; please apply your changes to the latest version and try again"
I1225 18:24:50.684982 1 conditions.go:192] Found status change for Certificate "tls-rancher-ingress" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2023-12-25 18:24:50.684968024 +0000 UTC m=+537.900583927