I am a newbie in k3s and rancher, started using k3s and rancher in the past few weeks. I installed a 2-server,2-agent k3s cluster of version v1.23.8+k3s2 with a external postgresql and nginx load balancer. The installation command for k3s is curl -sfL https://get.k3s.io | sh -s - server --datastore-endpoint=“postgres://root:123123@192.168.1.211:5432/k3s4?sslmode=disable” --node-taint CriticalAddonsonly=true:NoExecute --tls-san 192.168.1.52. where 192.168.1.52 is the address of my nginx load balancer. Then I installed rancher of version 2.6.6 with helm chart. I also use a DNS to redirect the domain name used in rancher to the ip of nginx load balancer. Everything seems fine but I cant access to the rancher web UI with the domain name.
When I logged into one of the rancher pods, with
kubectl -n cattle-system logs pod/rancher-864b9b7d48-7rv52 | grep ERROR
2022/07/22 07:51:56 [ERROR] Failed to read API for groups map[monitoring.coreos.com/v1:the server could not find the requested resource]
2022/07/22 07:52:02 [ERROR] Failed to find system chart fleet will try again in 5 seconds: configmaps "" not found
2022/07/22 07:52:02 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/07/22 07:52:02 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/07/22 07:52:03 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/07/22 07:52:04 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/07/22 07:52:04 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/07/22 07:52:04 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/07/22 07:52:04 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/07/22 07:52:06 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/07/22 07:52:07 [ERROR] Failed to find system chart fleet will try again in 5 seconds: configmaps "" not found
2022/07/22 07:52:07 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/07/22 07:52:07 [ERROR] error syncing 'fleet-local/local': handler cluster-create: server url is missing, can't generate kubeconfig for fleet import cluster, requeuing
2022/07/22 07:52:07 [ERROR] error syncing 'fleet-local/local': handler cluster-create: server url is missing, can't generate kubeconfig for fleet import cluster, requeuing
2022/07/22 07:52:08 [ERROR] error syncing 'fleet-local/local': handler cluster-create: server url is missing, can't generate kubeconfig for fleet import cluster, requeuing
2022/07/22 07:52:09 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/07/22 07:52:09 [ERROR] error syncing 'fleet-local/local': handler cluster-create: server url is missing, can't generate kubeconfig for fleet import cluster, requeuing
2022/07/22 07:52:10 [ERROR] error syncing 'fleet-local/local': handler cluster-create: server url is missing, can't generate kubeconfig for fleet import cluster, requeuing
2022/07/22 07:52:11 [ERROR] error syncing 'fleet-local/local': handler cluster-create: server url is missing, can't generate kubeconfig for fleet import cluster, requeuing
2022/07/22 07:52:11 [ERROR] error syncing 'fleet-local/local': handler cluster-create: server url is missing, can't generate kubeconfig for fleet import cluster, requeuing
2022/07/22 07:52:12 [ERROR] Failed to find system chart fleet will try again in 5 seconds: configmaps "" not found
2022/07/22 07:52:12 [ERROR] error parsing azure-group-cache-size, skipping update strconv.Atoi: parsing "": invalid syntax
2022/07/22 07:52:13 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/07/22 07:52:13 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=Cluster
2022/07/22 07:52:13 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=Machine
2022/07/22 07:52:13 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=MachineHealthCheck
2022/07/22 07:52:13 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=MachineDeployment
2022/07/22 07:52:13 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=MachineSet
2022/07/22 07:52:13 [ERROR] failed to start controller for cluster.x-k8s.io/v1alpha3, Kind=Cluster: failed to wait for caches to sync
2022/07/22 07:52:13 [ERROR] failed to start controller for cluster.x-k8s.io/v1alpha3, Kind=Machine: failed to wait for caches to sync
2022/07/22 07:52:13 [ERROR] failed to start controller for cluster.x-k8s.io/v1alpha3, Kind=MachineHealthCheck: failed to wait for caches to sync
2022/07/22 07:52:13 [ERROR] failed to start controller for cluster.x-k8s.io/v1alpha3, Kind=MachineDeployment: failed to wait for caches to sync
2022/07/22 07:52:13 [ERROR] failed to start controller for cluster.x-k8s.io/v1alpha3, Kind=MachineSet: failed to wait for caches to sync
2022/07/22 07:52:14 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/07/22 07:52:15 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/07/22 07:52:16 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/07/22 07:52:16 [ERROR] error syncing 'fleet-local/local': handler workspace-backport-cluster: fleetworkspaces.management.cattle.io "fleet-local" not found, requeuing
2022/07/22 07:52:16 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=MachineSet
2022/07/22 07:52:16 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/07/22 07:52:16 [ERROR] failed to cleanup orphan catalog bindings
2022/07/22 07:52:17 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/07/22 07:52:17 [ERROR] Failed to find system chart fleet will try again in 5 seconds: configmaps "" not found
2022/07/22 07:52:17 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/07/22 07:52:18 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/07/22 07:52:19 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/07/22 07:52:24 [ERROR] failed to sync cache for rke-machine-config.cattle.io/v1, Kind=DigitaloceanConfig
2022/07/22 07:52:24 [ERROR] failed to sync cache for rke-machine.cattle.io/v1, Kind=LinodeMachineTemplate
2022/07/22 07:52:24 [ERROR] failed to start controller for rke-machine-config.cattle.io/v1, Kind=DigitaloceanConfig: failed to wait for caches to sync
2022/07/22 07:52:24 [ERROR] failed to start controller for rke-machine.cattle.io/v1, Kind=LinodeMachineTemplate: failed to wait for caches to sync
2022/07/22 07:52:25 [ERROR] failed to sync cache for rke-machine-config.cattle.io/v1, Kind=Amazonec2Config
2022/07/22 07:52:25 [ERROR] failed to sync cache for rke-machine.cattle.io/v1, Kind=Amazonec2MachineTemplate
2022/07/22 07:52:25 [ERROR] failed to sync cache for rke-machine.cattle.io/v1, Kind=LinodeMachineTemplate
2022/07/22 07:52:25 [ERROR] failed to sync cache for rke-machine.cattle.io/v1, Kind=Amazonec2Machine
2022/07/22 07:52:25 [ERROR] failed to sync cache for rke-machine.cattle.io/v1, Kind=DigitaloceanMachineTemplate
2022/07/22 07:52:25 [ERROR] failed to start controller for rke-machine-config.cattle.io/v1, Kind=Amazonec2Config: failed to wait for caches to sync
2022/07/22 07:52:25 [ERROR] failed to start controller for rke-machine.cattle.io/v1, Kind=Amazonec2MachineTemplate: failed to wait for caches to sync
2022/07/22 07:52:25 [ERROR] failed to start controller for rke-machine.cattle.io/v1, Kind=LinodeMachineTemplate: failed to wait for caches to sync
2022/07/22 07:52:25 [ERROR] failed to start controller for rke-machine.cattle.io/v1, Kind=Amazonec2Machine: failed to wait for caches to sync
2022/07/22 07:52:25 [ERROR] failed to start controller for rke-machine.cattle.io/v1, Kind=DigitaloceanMachineTemplate: failed to wait for caches to sync
2022/07/22 07:52:30 [ERROR] Failed to handling tunnel request from remote address 10.42.3.9:52792: response 400: cluster not found
2022/07/22 07:52:31 [ERROR] error parsing azure-group-cache-size, skipping update strconv.Atoi: parsing "": invalid syntax
2022/07/22 07:52:33 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=kube-public, err=Operation cannot be fulfilled on namespaces "kube-public": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:33 [ERROR] Failed to handling tunnel request from remote address 10.42.3.10:45098: response 400: cluster not found
2022/07/22 07:52:34 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=kube-node-lease, err=Operation cannot be fulfilled on namespaces "kube-node-lease": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:34 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-system, err=Operation cannot be fulfilled on namespaces "cattle-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:35 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "default": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:35 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cert-manager, err=Operation cannot be fulfilled on namespaces "cert-manager": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:35 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "kube-node-lease": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:35 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "kube-public": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:35 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-global-data": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:35 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-global-nt, err=Operation cannot be fulfilled on namespaces "cattle-global-nt": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:35 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "fleet-default": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:36 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-global-data": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:36 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:36 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-global-data, err=Operation cannot be fulfilled on namespaces "cattle-global-data": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:36 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "kube-public": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:37 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-impersonation-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:37 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-g5cnd, err=Operation cannot be fulfilled on namespaces "p-g5cnd": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:37 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-nbq6f, err=Operation cannot be fulfilled on namespaces "p-nbq6f": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:37 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-fleet-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:38 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-fleet-system, err=Operation cannot be fulfilled on namespaces "cattle-fleet-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:38 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-impersonation-system, err=Operation cannot be fulfilled on namespaces "cattle-impersonation-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:39 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cert-manager": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:39 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-impersonation-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:39 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=fleet-local, err=Operation cannot be fulfilled on namespaces "fleet-local": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:39 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-fleet-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:40 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-global-nt, err=Operation cannot be fulfilled on namespaces "cattle-global-nt": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:40 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:40 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cert-manager": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:40 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-impersonation-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:40 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-fleet-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:40 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-impersonation-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:41 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cert-manager": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:42 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:43 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:43 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:44 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "kube-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:44 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:46 [ERROR] Failed to serve peer connection 10.42.3.9: websocket: close 1006 (abnormal closure): unexpected EOF
2022/07/22 07:52:46 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:46 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:46 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:55 [ERROR] Failed to handling tunnel request from remote address 10.42.3.10:45108: response 400: cluster not found
2022/07/22 07:53:01 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-fleet-clusters-system, err=Operation cannot be fulfilled on namespaces "cattle-fleet-clusters-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:53:01 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-fleet-clusters-system, err=Operation cannot be fulfilled on namespaces "cattle-fleet-clusters-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:53:01 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-fleet-clusters-system, err=Operation cannot be fulfilled on namespaces "cattle-fleet-clusters-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:53:01 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-fleet-clusters-system, err=Operation cannot be fulfilled on namespaces "cattle-fleet-clusters-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:53:01 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-fleet-clusters-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:53:41 [ERROR] error syncing 'cattle-fleet-system/helm-operation-lwxdm': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-lwxdm), requeuing
2022/07/22 07:53:47 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cluster-fleet-local-local-1a3d67d0a899, err=Operation cannot be fulfilled on namespaces "cluster-fleet-local-local-1a3d67d0a899": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:53:47 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cluster-fleet-local-local-1a3d67d0a899": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:53:49 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-fleet-local-system, err=Operation cannot be fulfilled on namespaces "cattle-fleet-local-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:53:50 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-fleet-local-system, err=Operation cannot be fulfilled on namespaces "cattle-fleet-local-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:53:52 [ERROR] error syncing 'cattle-fleet-system/helm-operation-czbd5': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-czbd5), requeuing
2022/07/22 07:53:52 [ERROR] error syncing 'cattle-fleet-system/helm-operation-czbd5': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-czbd5), requeuing
2022/07/22 07:53:52 [ERROR] error syncing 'cattle-fleet-system/helm-operation-czbd5': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-czbd5), requeuing
2022/07/22 07:53:52 [ERROR] error syncing 'cattle-fleet-system/helm-operation-czbd5': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-czbd5), requeuing
2022/07/22 07:53:53 [ERROR] error syncing 'cattle-fleet-system/helm-operation-czbd5': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-czbd5), requeuing
2022/07/22 07:53:53 [ERROR] error syncing 'cattle-fleet-system/helm-operation-czbd5': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-czbd5), requeuing
2022/07/22 07:53:53 [ERROR] error syncing 'cattle-fleet-system/helm-operation-czbd5': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-czbd5), requeuing
2022/07/22 07:53:58 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: secrets "rancher-webhook-tls" already exists, requeuing
Here is the log of another rancher pod:
root@m2:~# kubectl -n cattle-system logs pod/rancher-864b9b7d48-hfn99 | grep ERROR
2022/07/22 07:52:33 [ERROR] Failed to connect to peer wss://10.42.3.9/v3/connect [local ID=10.42.3.10]: websocket: bad handshake
2022/07/22 07:52:33 [ERROR] Failed to connect to peer wss://10.42.2.15/v3/connect [local ID=10.42.3.10]: websocket: bad handshake
2022/07/22 07:52:36 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:36 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:40 [ERROR] Failed to connect to peer wss://10.42.2.15/v3/connect [local ID=10.42.3.10]: read tcp 10.42.3.10:45102->10.42.2.15:443: read: connection reset by peer
2022/07/22 07:52:41 [ERROR] Failed to connect to peer wss://10.42.3.9/v3/connect [local ID=10.42.3.10]: websocket: bad handshake
2022/07/22 07:52:44 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:45 [ERROR] Failed to connect to peer wss://10.42.2.15/v3/connect [local ID=10.42.3.10]: EOF
2022/07/22 07:52:46 [ERROR] Failed to connect to peer wss://10.42.3.9/v3/connect [local ID=10.42.3.10]: EOF
2022/07/22 07:52:46 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:46 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:50 [ERROR] Failed to connect to peer wss://10.42.2.15/v3/connect [local ID=10.42.3.10]: read tcp 10.42.3.10:45106->10.42.2.15:443: read: connection reset by peer
2022/07/22 07:52:51 [ERROR] Failed to connect to peer wss://10.42.3.9/v3/connect [local ID=10.42.3.10]: EOF
2022/07/22 07:52:55 [ERROR] Failed to connect to peer wss://10.42.2.15/v3/connect [local ID=10.42.3.10]: websocket: bad handshake
2022/07/22 07:52:56 [ERROR] Failed to connect to peer wss://10.42.3.9/v3/connect [local ID=10.42.3.10]: websocket: bad handshake
Here is the status of the pods:
root@m2:~# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system helm-install-traefik-crd-zxznm 0/1 Completed 0 70m
kube-system helm-install-traefik-wkbzg 0/1 Completed 1 (69m ago) 70m
kube-system svclb-traefik-0759131d-mwp9m 2/2 Running 0 69m
kube-system traefik-df4ff85d6-qp757 1/1 Running 0 69m
kube-system helm-install-traefik-crd-7bcsj 0/1 Completed 0 69m
kube-system helm-install-traefik-7npqx 0/1 Completed 0 69m
kube-system svclb-traefik-0759131d-f9qkg 2/2 Running 2 (67m ago) 69m
kube-system coredns-d76bd69b-nfc4p 1/1 Running 1 (67m ago) 70m
kube-system metrics-server-7cd5fcb6b7-l6kxk 1/1 Running 1 (67m ago) 109m
kube-system local-path-provisioner-6c79684f77-zh4vg 1/1 Running 1 (66m ago) 109m
cert-manager cert-manager-76d44b459c-96fsw 1/1 Running 0 51m
cert-manager cert-manager-cainjector-9b679cc6-rs9gz 1/1 Running 0 51m
cert-manager cert-manager-webhook-57c994b6b9-6t5kl 1/1 Running 0 51m
cattle-system rancher-864b9b7d48-7rv52 1/1 Running 0 49m
cattle-system rancher-864b9b7d48-hr9t4 1/1 Running 0 49m
cattle-system rancher-864b9b7d48-hfn99 1/1 Running 1 (47m ago) 49m
cattle-fleet-system gitjob-6b977748fc-dn622 1/1 Running 0 46m
cattle-fleet-system fleet-controller-784d6fbcd8-dwgnt 1/1 Running 0 46m
cattle-system helm-operation-8p92x 0/2 Completed 0 47m
cattle-system helm-operation-lwxdm 0/2 Completed 0 46m
cattle-system helm-operation-czbd5 0/2 Completed 0 46m
cattle-fleet-local-system fleet-agent-699b5fb945-6467t 1/1 Running 0 45m
cattle-system rancher-webhook-5b65595df9-m5c8g 1/1 Running 0 45m
cattle-system helm-operation-kvggh 0/2 Completed 0 45m
kubecnfig:
root@m2:~# kubectl config view
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://192.168.1.52:6443
name: default
contexts:
- context:
cluster: default
user: default
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
anyone knows how to fix the bug?