Rancher on K3s with HAProxy LB - Backend down, 404

radiojosh · December 27, 2023, 12:30am

I’ve been trying to deploy Rancher on an HA K3s / etcd cluster running on VMware. HAProxy load balancer, and self-signed certificates were chosen. When I’ve completed the steps as documented, the load balancer backend is still down. Connecting directly to one of the K3s hosts gives nothing but a 404 error. If I attach to a shell on one of the rancher pods, I can get connect to 80 and 443 on the other rancher pods via curl. It appears that it’s functioning. So I think the ingress just isn’t getting set up through Traefik. There is no mention of additional steps to configure Traefik or Cert-manager, but Cert Manager and Traefik are both complaning about a missing TLS secret. Am I wrong to think that the ingress should automatically be created when installing Rancher? Not sure what to do.

I’ve tried different versions and loads of troubleshooting steps.

Versions currently installed:

Os - Rocky Linux 9.3
K3s - v1.26.11+k3s2
Rancher - 2.7.9
Cert-Manager - 1.12.7

Extra troubleshooting steps still applied:

Firewall disabled (definitely required, fixed some problems)
SELinux in permissive mode (unknown if it fixed anything)
Set Flannel to Local GW (unknown if it fixed anything)

I will try to include logs and information in a reply since the forum won’t let me put more than two links in my initial post.

radiojosh · December 27, 2023, 12:32am

Some of the relevant output:

# kubectl -n cattle-system get svc -o wide
NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE   SELECTOR
rancher           ClusterIP   10.43.202.96   <none>        80/TCP,443/TCP   29h   app=rancher
rancher-webhook   ClusterIP   10.43.55.247   <none>        443/TCP          29h   app=rancher-webhook

Rancher container 1 log errors:

2023/12/25 18:24:25 [ERROR] Failed to handle tunnel request from remote address 10.42.1.9:36228: response 400: cluster not found
2023/12/25 18:24:25 [ERROR] Failed to handle tunnel request from remote address 10.42.0.9:45522: response 400: cluster not found
2023/12/25 18:24:26 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=kube-public, err=Operation cannot be fulfilled on namespaces "kube-public": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-system, err=Operation cannot be fulfilled on namespaces "cattle-system": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-fleet-system, err=Operation cannot be fulfilled on namespaces "cattle-fleet-system": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cert-manager, err=Operation cannot be fulfilled on namespaces "cert-manager": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-bktxc, err=Operation cannot be fulfilled on namespaces "p-bktxc": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-j92fh, err=Operation cannot be fulfilled on namespaces "p-j92fh": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-impersonation-system, err=Operation cannot be fulfilled on namespaces "cattle-impersonation-system": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-global-data, err=Operation cannot be fulfilled on namespaces "cattle-global-data": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=kube-node-lease, err=Operation cannot be fulfilled on namespaces "kube-node-lease": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-global-nt, err=Operation cannot be fulfilled on namespaces "cattle-global-nt": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:27 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-j92fh, err=Operation cannot be fulfilled on namespaces "p-j92fh": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:27 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=fleet-local, err=Operation cannot be fulfilled on namespaces "fleet-local": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:27 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-global-data, err=Operation cannot be fulfilled on namespaces "cattle-global-data": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:27 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-impersonation-system, err=Operation cannot be fulfilled on namespaces "cattle-impersonation-system": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:27 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-impersonation-system, err=Operation cannot be fulfilled on namespaces "cattle-impersonation-system": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:30 [ERROR] Failed to serve peer connection 10.42.0.9: websocket: close 1006 (abnormal closure): unexpected EOF
2023/12/25 18:25:39 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cluster-fleet-local-local-1a3d67d0a899, err=Operation cannot be fulfilled on namespaces "cluster-fleet-local-local-1a3d67d0a899": the object has been modified; please apply your changes to the latest version and try again

Rancher container 2 log errors:

2023/12/25 18:24:09 [ERROR] Failed to find system chart fleet will try again in 5 seconds: configmaps "" not found
2023/12/25 18:24:09 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2023/12/25 18:24:09 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2023/12/25 18:24:10 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2023/12/25 18:24:10 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2023/12/25 18:24:10 [ERROR] error syncing 'fleet-local/local': handler workspace-backport-cluster: fleetworkspaces.management.cattle.io "fleet-local" not found, requeuing
2023/12/25 18:24:10 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2023/12/25 18:24:11 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2023/12/25 18:24:11 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2023/12/25 18:24:13 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2023/12/25 18:24:14 [ERROR] Failed to find system chart fleet will try again in 5 seconds: configmaps "" not found
2023/12/25 18:24:16 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2023/12/25 18:24:19 [ERROR] Failed to find system chart fleet will try again in 5 seconds: configmaps "" not found
2023/12/25 18:24:21 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2023/12/25 18:24:25 [ERROR] Failed to connect to peer wss://10.42.2.9/v3/connect [local ID=10.42.1.9]: websocket: bad handshake
2023/12/25 18:24:25 [ERROR] Failed to connect to peer wss://10.42.0.9/v3/connect [local ID=10.42.1.9]: websocket: bad handshake
2023/12/25 18:24:26 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "kube-public": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-fleet-system": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cert-manager": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "fleet-default": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "fleet-local": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:26 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cert-manager": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:27 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "kube-system": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:27 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-system": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:53 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-fleet-clusters-system": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:24:53 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-fleet-clusters-system": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:25:39 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cluster-fleet-local-local-1a3d67d0a899": the object has been modified; please apply your changes to the latest version and try again
2023/12/25 18:25:39 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cluster-fleet-local-local-1a3d67d0a899": the object has been modified; please apply your changes to the latest version and try again

Traefik logs:

time="2023-12-25T18:01:55Z" level=info msg="Configuration loaded from flags."
time="2023-12-25T18:22:25Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:22:25Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes ingress=rancher namespace=cattle-system serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}"
time="2023-12-25T18:22:25Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:22:25Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes ingress=rancher namespace=cattle-system serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}"
time="2023-12-25T18:22:25Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:22:25Z" level=error msg="Skipping service: no endpoints found" ingress=rancher namespace=cattle-system serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes
time="2023-12-25T18:23:38Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:23:38Z" level=error msg="Skipping service: no endpoints found" serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:23:39Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:23:39Z" level=error msg="Skipping service: no endpoints found" serviceName=rancher providerName=kubernetes ingress=rancher namespace=cattle-system servicePort="&ServiceBackendPort{Name:,Number:80,}"
time="2023-12-25T18:23:39Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" ingress=rancher namespace=cattle-system providerName=kubernetes
time="2023-12-25T18:23:39Z" level=error msg="Skipping service: no endpoints found" ingress=rancher namespace=cattle-system providerName=kubernetes serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}"
time="2023-12-25T18:24:08Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes namespace=cattle-system ingress=rancher
time="2023-12-25T18:24:08Z" level=error msg="Skipping service: no endpoints found" serviceName=rancher providerName=kubernetes servicePort="&ServiceBackendPort{Name:,Number:80,}" namespace=cattle-system ingress=rancher
time="2023-12-25T18:24:08Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:08Z" level=error msg="Skipping service: no endpoints found" ingress=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}" namespace=cattle-system serviceName=rancher providerName=kubernetes
time="2023-12-25T18:24:08Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:08Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes ingress=rancher namespace=cattle-system serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}"
time="2023-12-25T18:24:08Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:24:08Z" level=error msg="Skipping service: no endpoints found" servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=rancher namespace=cattle-system serviceName=rancher
time="2023-12-25T18:24:09Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:09Z" level=error msg="Skipping service: no endpoints found" ingress=rancher namespace=cattle-system serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes
time="2023-12-25T18:24:09Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:24:09Z" level=error msg="Skipping service: no endpoints found" servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=rancher namespace=cattle-system serviceName=rancher
time="2023-12-25T18:24:09Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:24:09Z" level=error msg="Skipping service: no endpoints found" ingress=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}" namespace=cattle-system serviceName=rancher providerName=kubernetes
time="2023-12-25T18:24:09Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:09Z" level=error msg="Skipping service: no endpoints found" servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=rancher namespace=cattle-system serviceName=rancher
time="2023-12-25T18:24:09Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes namespace=cattle-system ingress=rancher
time="2023-12-25T18:24:09Z" level=error msg="Skipping service: no endpoints found" servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes namespace=cattle-system ingress=rancher serviceName=rancher
time="2023-12-25T18:24:09Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" ingress=rancher namespace=cattle-system providerName=kubernetes
time="2023-12-25T18:24:09Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes ingress=rancher namespace=cattle-system serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}"
time="2023-12-25T18:24:10Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:10Z" level=error msg="Skipping service: no endpoints found" servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=rancher namespace=cattle-system serviceName=rancher
time="2023-12-25T18:24:24Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes namespace=cattle-system ingress=rancher
time="2023-12-25T18:24:24Z" level=error msg="Skipping service: no endpoints found" ingress=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}" serviceName=rancher providerName=kubernetes namespace=cattle-system
time="2023-12-25T18:24:24Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:24Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes ingress=rancher namespace=cattle-system servicePort="&ServiceBackendPort{Name:,Number:80,}" serviceName=rancher
time="2023-12-25T18:24:25Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:25Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" ingress=rancher namespace=cattle-system providerName=kubernetes
time="2023-12-25T18:24:25Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes namespace=cattle-system ingress=rancher
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" ingress=rancher namespace=cattle-system providerName=kubernetes
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" ingress=rancher namespace=cattle-system providerName=kubernetes
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" ingress=rancher namespace=cattle-system providerName=kubernetes
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" ingress=rancher namespace=cattle-system providerName=kubernetes
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" ingress=rancher namespace=cattle-system providerName=kubernetes
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher
time="2023-12-25T18:24:26Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" ingress=rancher namespace=cattle-system providerName=kubernetes
time="2023-12-25T18:24:27Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:27Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:27Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:27Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:31Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" ingress=rancher namespace=cattle-system providerName=kubernetes
time="2023-12-25T18:24:43Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" providerName=kubernetes ingress=rancher namespace=cattle-system
time="2023-12-25T18:24:43Z" level=error msg="Error configuring TLS: secret cattle-system/tls-rancher-ingress does not exist" namespace=cattle-system providerName=kubernetes ingress=rancher

Cert-Manager logs:

I1225 18:15:52.856071       1 start.go:75] "cert-manager: starting controller" version="v1.12.7" git-commit="6d7629ba42b946978e3baaa75348c851f7ef9134"
I1225 18:15:52.856433       1 controller.go:262] "cert-manager/controller/build-context: configured acme dns01 nameservers" nameservers=["10.43.0.10:53"]
W1225 18:15:52.856680       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1225 18:15:52.859186       1 controller.go:82] "cert-manager/controller: enabled controllers: [certificaterequests-approver certificaterequests-issuer-acme certificaterequests-issuer-ca certificaterequests-issuer-selfsigned certificaterequests-issuer-vault certificaterequests-issuer-venafi certificates-issuing certificates-key-manager certificates-metrics certificates-readiness certificates-request-manager certificates-revision-manager certificates-trigger challenges clusterissuers ingress-shim issuers orders]"
I1225 18:15:52.860053       1 controller.go:103] "cert-manager/controller: starting metrics server" address="[::]:9402"
I1225 18:15:52.860081       1 controller.go:156] "cert-manager/controller: starting leader election"
I1225 18:15:52.860570       1 controller.go:149] "cert-manager/controller: starting healthz server" address="[::]:9403"
I1225 18:15:52.861201       1 leaderelection.go:245] attempting to acquire leader lease kube-system/cert-manager-controller...
I1225 18:17:05.785099       1 leaderelection.go:255] successfully acquired lease kube-system/cert-manager-controller
I1225 18:17:05.786668       1 controller.go:226] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-selfsigned"
I1225 18:17:05.787805       1 controller.go:226] "cert-manager/controller: starting controller" controller="certificates-key-manager"
I1225 18:17:05.790161       1 controller.go:226] "cert-manager/controller: starting controller" controller="certificates-readiness"
I1225 18:17:05.792116       1 controller.go:203] "cert-manager/controller: not starting controller as it's disabled" controller="certificatesigningrequests-issuer-ca"
I1225 18:17:05.795483       1 controller.go:226] "cert-manager/controller: starting controller" controller="certificates-trigger"
I1225 18:17:05.795716       1 controller.go:226] "cert-manager/controller: starting controller" controller="orders"
I1225 18:17:05.796786       1 controller.go:226] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-acme"
I1225 18:17:05.798333       1 controller.go:203] "cert-manager/controller: not starting controller as it's disabled" controller="certificatesigningrequests-issuer-selfsigned"
I1225 18:17:05.798408       1 controller.go:226] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-ca"
I1225 18:17:05.799565       1 controller.go:226] "cert-manager/controller: starting controller" controller="ingress-shim"
I1225 18:17:05.800619       1 controller.go:226] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-vault"
I1225 18:17:05.801595       1 controller.go:226] "cert-manager/controller: starting controller" controller="certificates-metrics"
I1225 18:17:05.802176       1 controller.go:203] "cert-manager/controller: not starting controller as it's disabled" controller="gateway-shim"
I1225 18:17:05.802244       1 controller.go:226] "cert-manager/controller: starting controller" controller="issuers"
I1225 18:17:05.802999       1 controller.go:226] "cert-manager/controller: starting controller" controller="certificates-issuing"
I1225 18:17:05.804079       1 controller.go:226] "cert-manager/controller: starting controller" controller="clusterissuers"
I1225 18:17:05.804771       1 controller.go:226] "cert-manager/controller: starting controller" controller="certificates-request-manager"
I1225 18:17:05.805435       1 controller.go:203] "cert-manager/controller: not starting controller as it's disabled" controller="certificatesigningrequests-issuer-acme"
I1225 18:17:05.805480       1 controller.go:203] "cert-manager/controller: not starting controller as it's disabled" controller="certificatesigningrequests-issuer-vault"
I1225 18:17:05.805500       1 controller.go:203] "cert-manager/controller: not starting controller as it's disabled" controller="certificatesigningrequests-issuer-venafi"
I1225 18:17:05.805611       1 controller.go:226] "cert-manager/controller: starting controller" controller="certificates-revision-manager"
I1225 18:17:05.806494       1 controller.go:226] "cert-manager/controller: starting controller" controller="challenges"
I1225 18:17:05.806997       1 controller.go:226] "cert-manager/controller: starting controller" controller="certificaterequests-approver"
I1225 18:17:05.809476       1 controller.go:226] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-venafi"
E1225 18:22:25.341481       1 setup.go:48] "cert-manager/issuers/setup: error getting signing CA TLS certificate" err="secret \"tls-rancher\" not found" resource_name="rancher" resource_namespace="cattle-system" resource_kind="Issuer" resource_version="v1"
I1225 18:22:25.341591       1 conditions.go:96] Setting lastTransitionTime for Issuer "rancher" condition "Ready" to 2023-12-25 18:22:25.34155498 +0000 UTC m=+392.557170883
I1225 18:22:25.341691       1 sync.go:62] "cert-manager/issuers: Error initializing issuer: secret \"tls-rancher\" not found" resource_name="rancher" resource_namespace="cattle-system" resource_kind="Issuer" resource_version="v1"
I1225 18:22:25.345121       1 conditions.go:203] Setting lastTransitionTime for Certificate "tls-rancher-ingress" condition "Ready" to 2023-12-25 18:22:25.34510582 +0000 UTC m=+392.560721757
I1225 18:22:25.345180       1 trigger_controller.go:194] "cert-manager/certificates-trigger: Certificate must be re-issued" key="cattle-system/tls-rancher-ingress" reason="DoesNotExist" message="Issuing certificate as Secret does not exist"
I1225 18:22:25.345378       1 conditions.go:203] Setting lastTransitionTime for Certificate "tls-rancher-ingress" condition "Issuing" to 2023-12-25 18:22:25.345365667 +0000 UTC m=+392.560981597
E1225 18:22:25.479134       1 controller.go:167] "cert-manager/issuers: re-queuing item due to error processing" err="secret \"tls-rancher\" not found" key="cattle-system/rancher"
E1225 18:22:25.479269       1 setup.go:48] "cert-manager/issuers/setup: error getting signing CA TLS certificate" err="secret \"tls-rancher\" not found" resource_name="rancher" resource_namespace="cattle-system" resource_kind="Issuer" resource_version="v1"
I1225 18:22:25.479309       1 sync.go:62] "cert-manager/issuers: Error initializing issuer: secret \"tls-rancher\" not found" resource_name="rancher" resource_namespace="cattle-system" resource_kind="Issuer" resource_version="v1"
E1225 18:22:25.479389       1 controller.go:167] "cert-manager/issuers: re-queuing item due to error processing" err="secret \"tls-rancher\" not found" key="cattle-system/rancher"
E1225 18:22:25.479959       1 controller.go:167] "cert-manager/ingress-shim: re-queuing item due to error processing" err="certificates.cert-manager.io \"tls-rancher-ingress\" already exists" key="cattle-system/rancher"
I1225 18:22:25.501217       1 controller.go:162] "cert-manager/certificates-trigger: re-queuing item due to optimistic locking on resource" key="cattle-system/tls-rancher-ingress" error="Operation cannot be fulfilled on certificates.cert-manager.io \"tls-rancher-ingress\": the object has been modified; please apply your changes to the latest version and try again"
I1225 18:22:25.501324       1 trigger_controller.go:194] "cert-manager/certificates-trigger: Certificate must be re-issued" key="cattle-system/tls-rancher-ingress" reason="DoesNotExist" message="Issuing certificate as Secret does not exist"
I1225 18:22:25.501355       1 conditions.go:203] Setting lastTransitionTime for Certificate "tls-rancher-ingress" condition "Issuing" to 2023-12-25 18:22:25.50134574 +0000 UTC m=+392.716961670
I1225 18:22:25.934955       1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "tls-rancher-ingress-55btb" condition "Approved" to 2023-12-25 18:22:25.934940321 +0000 UTC m=+393.150556248
I1225 18:22:25.978798       1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "tls-rancher-ingress-55btb" condition "Ready" to 2023-12-25 18:22:25.978722424 +0000 UTC m=+393.194338301
E1225 18:22:30.479831       1 setup.go:48] "cert-manager/issuers/setup: error getting signing CA TLS certificate" err="secret \"tls-rancher\" not found" resource_name="rancher" resource_namespace="cattle-system" resource_kind="Issuer" resource_version="v1"
I1225 18:22:30.479954       1 sync.go:62] "cert-manager/issuers: Error initializing issuer: secret \"tls-rancher\" not found" resource_name="rancher" resource_namespace="cattle-system" resource_kind="Issuer" resource_version="v1"
E1225 18:22:30.480281       1 controller.go:167] "cert-manager/issuers: re-queuing item due to error processing" err="secret \"tls-rancher\" not found" key="cattle-system/rancher"
E1225 18:22:50.480715       1 setup.go:48] "cert-manager/issuers/setup: error getting signing CA TLS certificate" err="secret \"tls-rancher\" not found" resource_name="rancher" resource_namespace="cattle-system" resource_kind="Issuer" resource_version="v1"
I1225 18:22:50.480787       1 sync.go:62] "cert-manager/issuers: Error initializing issuer: secret \"tls-rancher\" not found" resource_name="rancher" resource_namespace="cattle-system" resource_kind="Issuer" resource_version="v1"
E1225 18:22:50.480958       1 controller.go:167] "cert-manager/issuers: re-queuing item due to error processing" err="secret \"tls-rancher\" not found" key="cattle-system/rancher"
E1225 18:23:30.481497       1 setup.go:48] "cert-manager/issuers/setup: error getting signing CA TLS certificate" err="secret \"tls-rancher\" not found" resource_name="rancher" resource_namespace="cattle-system" resource_kind="Issuer" resource_version="v1"
I1225 18:23:30.481568       1 sync.go:62] "cert-manager/issuers: Error initializing issuer: secret \"tls-rancher\" not found" resource_name="rancher" resource_namespace="cattle-system" resource_kind="Issuer" resource_version="v1"
E1225 18:23:30.481661       1 controller.go:167] "cert-manager/issuers: re-queuing item due to error processing" err="secret \"tls-rancher\" not found" key="cattle-system/rancher"
I1225 18:24:50.483029       1 conditions.go:85] Found status change for Issuer "rancher" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2023-12-25 18:24:50.483013988 +0000 UTC m=+537.698629891
I1225 18:24:50.548773       1 conditions.go:252] Found status change for CertificateRequest "tls-rancher-ingress-55btb" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2023-12-25 18:24:50.54875726 +0000 UTC m=+537.764373220
I1225 18:24:50.601572       1 conditions.go:192] Found status change for Certificate "tls-rancher-ingress" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2023-12-25 18:24:50.601556272 +0000 UTC m=+537.817172158
I1225 18:24:50.637607       1 controller.go:162] "cert-manager/certificates-readiness: re-queuing item due to optimistic locking on resource" key="cattle-system/tls-rancher-ingress" error="Operation cannot be fulfilled on certificates.cert-manager.io \"tls-rancher-ingress\": the object has been modified; please apply your changes to the latest version and try again"
I1225 18:24:50.639429       1 conditions.go:192] Found status change for Certificate "tls-rancher-ingress" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2023-12-25 18:24:50.63941676 +0000 UTC m=+537.855032633
I1225 18:24:50.655151       1 controller.go:162] "cert-manager/certificates-issuing: re-queuing item due to optimistic locking on resource" key="cattle-system/tls-rancher-ingress" error="Operation cannot be fulfilled on certificates.cert-manager.io \"tls-rancher-ingress\": the object has been modified; please apply your changes to the latest version and try again"
I1225 18:24:50.682257       1 controller.go:162] "cert-manager/certificates-key-manager: re-queuing item due to optimistic locking on resource" key="cattle-system/tls-rancher-ingress" error="Operation cannot be fulfilled on certificates.cert-manager.io \"tls-rancher-ingress\": the object has been modified; please apply your changes to the latest version and try again"
I1225 18:24:50.683594       1 controller.go:162] "cert-manager/certificates-readiness: re-queuing item due to optimistic locking on resource" key="cattle-system/tls-rancher-ingress" error="Operation cannot be fulfilled on certificates.cert-manager.io \"tls-rancher-ingress\": the object has been modified; please apply your changes to the latest version and try again"
I1225 18:24:50.684982       1 conditions.go:192] Found status change for Certificate "tls-rancher-ingress" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2023-12-25 18:24:50.684968024 +0000 UTC m=+537.900583927

virgiliofilho · March 12, 2024, 5:52pm

i’ve the same problem… do you resolved?

redhat 8.9
rke2 1.26 or 1.27
cert-manager 1.14 or 1.15
rancher 2.8.2

i can deploy:
rke2 (no set plugin cni active, but it’s work with default)- ok
deploy cert-manager - ok
rancher - “not good”

but rancher don’t work well…

with ingress (rke2 or ingress-nginx)
with loadbalancer (i’m trying using metallb)
with nodeport

the rancher stay ‘operational’ but i CAN’T create anothers cluster too, wherever rke2 or k3s.

kubectl logs -n cattle-system -l app=rancher -f

[ERROR] error syncing ‘harvester’: handler feature-handler: Internal error occurred: failed calling webhook “rancher.cattle.io.nodedrivers.management.cattle.io”: failed to call webhook: Post “https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/nodedrivers.management.cattle.io?timeout=10s”: no endpoints available for service “rancher-webhook”, requeuing

2024/03/12 16:30:58 [ERROR] error syncing ‘all’: handler user-controllers-controller: userControllersController: failed to set peers for key all: failed to start user controllers for cluster c-m-xtkmnbsz: ClusterUnavailable 503: cluster not found, requeuing
2024/03/12 16:31:02 [ERROR] Failed syncing peers [{10.42.0.26 true false}]: failed to start user controllers for cluster c-m-xtkmnbsz: ClusterUnavailable 503: cluster not found
2024/03/12 16:31:02 [ERROR] Failed to handle tunnel request from remote address 192.168.1.56:60728: response 400: cluster not found
2024/03/12 16:31:07 [ERROR] Failed to handle tunnel request from remote address 192.168.1.56:60732: response 400: cluster not found
2024/03/12 16:31:09 [ERROR] error syncing ‘all’: handler user-controllers-controller: userControllersController: failed to set peers for key all: failed to start user controllers for cluster c-m-xtkmnbsz: ClusterUnavailable 503: cluster not found, requeuing
2024/03/12 16:31:10 [INFO] Adding peer wss://10.42.0.27/v3/connect, 10.42.0.27
2024/03/12 16:31:10 [ERROR] Failed syncing peers [{10.42.0.26 [10.42.0.27] true false}]: failed to start user controllers for cluster c-m-xtkmnbsz: ClusterUnavailable 503: cluster not found

Topic		Replies	Views
Rancher on K3S cluster with nginx ingress controller Rancher 2.x	1	488	December 14, 2021
How to fix 404? Rancher 2.x	2	2684	December 26, 2023
Rancher + MetalLb +ingress.tls.source=secret = ERR_SSL_PROTOCOL_ERROR Rancher 2.x	0	986	May 16, 2020
Install failed, Then it went downhill. Rancher 2.6.7 Rancher 2.x	2	363	September 5, 2022
Looking for a tutorial installing loadbalancer Rancher 2.x	7	279	December 13, 2023

Rancher on K3s with HAProxy LB - Backend down, 404

kubectl logs -n cattle-system -l app=rancher -f

Related Topics