Unable to access the rancher web UI with k3s and rancher2.6.6

I am a newbie in k3s and rancher, started using k3s and rancher in the past few weeks. I installed a 2-server,2-agent k3s cluster of version v1.23.8+k3s2 with a external postgresql and nginx load balancer. The installation command for k3s is curl -sfL https://get.k3s.io | sh -s - server --datastore-endpoint=“postgres://root:123123@192.168.1.211:5432/k3s4?sslmode=disable” --node-taint CriticalAddonsonly=true:NoExecute --tls-san 192.168.1.52. where 192.168.1.52 is the address of my nginx load balancer. Then I installed rancher of version 2.6.6 with helm chart. I also use a DNS to redirect the domain name used in rancher to the ip of nginx load balancer. Everything seems fine but I cant access to the rancher web UI with the domain name.
When I logged into one of the rancher pods, with
kubectl -n cattle-system logs pod/rancher-864b9b7d48-7rv52 | grep ERROR

2022/07/22 07:51:56 [ERROR] Failed to read API for groups map[monitoring.coreos.com/v1:the server could not find the requested resource]
2022/07/22 07:52:02 [ERROR] Failed to find system chart fleet will try again in 5 seconds: configmaps "" not found
2022/07/22 07:52:02 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/07/22 07:52:02 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/07/22 07:52:03 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/07/22 07:52:04 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/07/22 07:52:04 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/07/22 07:52:04 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/07/22 07:52:04 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/07/22 07:52:06 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/07/22 07:52:07 [ERROR] Failed to find system chart fleet will try again in 5 seconds: configmaps "" not found
2022/07/22 07:52:07 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/07/22 07:52:07 [ERROR] error syncing 'fleet-local/local': handler cluster-create: server url is missing, can't generate kubeconfig for fleet import cluster, requeuing
2022/07/22 07:52:07 [ERROR] error syncing 'fleet-local/local': handler cluster-create: server url is missing, can't generate kubeconfig for fleet import cluster, requeuing
2022/07/22 07:52:08 [ERROR] error syncing 'fleet-local/local': handler cluster-create: server url is missing, can't generate kubeconfig for fleet import cluster, requeuing
2022/07/22 07:52:09 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/07/22 07:52:09 [ERROR] error syncing 'fleet-local/local': handler cluster-create: server url is missing, can't generate kubeconfig for fleet import cluster, requeuing
2022/07/22 07:52:10 [ERROR] error syncing 'fleet-local/local': handler cluster-create: server url is missing, can't generate kubeconfig for fleet import cluster, requeuing
2022/07/22 07:52:11 [ERROR] error syncing 'fleet-local/local': handler cluster-create: server url is missing, can't generate kubeconfig for fleet import cluster, requeuing
2022/07/22 07:52:11 [ERROR] error syncing 'fleet-local/local': handler cluster-create: server url is missing, can't generate kubeconfig for fleet import cluster, requeuing
2022/07/22 07:52:12 [ERROR] Failed to find system chart fleet will try again in 5 seconds: configmaps "" not found
2022/07/22 07:52:12 [ERROR] error parsing azure-group-cache-size, skipping update strconv.Atoi: parsing "": invalid syntax
2022/07/22 07:52:13 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/07/22 07:52:13 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=Cluster
2022/07/22 07:52:13 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=Machine
2022/07/22 07:52:13 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=MachineHealthCheck
2022/07/22 07:52:13 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=MachineDeployment
2022/07/22 07:52:13 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=MachineSet
2022/07/22 07:52:13 [ERROR] failed to start controller for cluster.x-k8s.io/v1alpha3, Kind=Cluster: failed to wait for caches to sync
2022/07/22 07:52:13 [ERROR] failed to start controller for cluster.x-k8s.io/v1alpha3, Kind=Machine: failed to wait for caches to sync
2022/07/22 07:52:13 [ERROR] failed to start controller for cluster.x-k8s.io/v1alpha3, Kind=MachineHealthCheck: failed to wait for caches to sync
2022/07/22 07:52:13 [ERROR] failed to start controller for cluster.x-k8s.io/v1alpha3, Kind=MachineDeployment: failed to wait for caches to sync
2022/07/22 07:52:13 [ERROR] failed to start controller for cluster.x-k8s.io/v1alpha3, Kind=MachineSet: failed to wait for caches to sync
2022/07/22 07:52:14 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/07/22 07:52:15 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/07/22 07:52:16 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/07/22 07:52:16 [ERROR] error syncing 'fleet-local/local': handler workspace-backport-cluster: fleetworkspaces.management.cattle.io "fleet-local" not found, requeuing
2022/07/22 07:52:16 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=MachineSet
2022/07/22 07:52:16 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/07/22 07:52:16 [ERROR] failed to cleanup orphan catalog bindings
2022/07/22 07:52:17 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/07/22 07:52:17 [ERROR] Failed to find system chart fleet will try again in 5 seconds: configmaps "" not found
2022/07/22 07:52:17 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/07/22 07:52:18 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/07/22 07:52:19 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/07/22 07:52:24 [ERROR] failed to sync cache for rke-machine-config.cattle.io/v1, Kind=DigitaloceanConfig
2022/07/22 07:52:24 [ERROR] failed to sync cache for rke-machine.cattle.io/v1, Kind=LinodeMachineTemplate
2022/07/22 07:52:24 [ERROR] failed to start controller for rke-machine-config.cattle.io/v1, Kind=DigitaloceanConfig: failed to wait for caches to sync
2022/07/22 07:52:24 [ERROR] failed to start controller for rke-machine.cattle.io/v1, Kind=LinodeMachineTemplate: failed to wait for caches to sync
2022/07/22 07:52:25 [ERROR] failed to sync cache for rke-machine-config.cattle.io/v1, Kind=Amazonec2Config
2022/07/22 07:52:25 [ERROR] failed to sync cache for rke-machine.cattle.io/v1, Kind=Amazonec2MachineTemplate
2022/07/22 07:52:25 [ERROR] failed to sync cache for rke-machine.cattle.io/v1, Kind=LinodeMachineTemplate
2022/07/22 07:52:25 [ERROR] failed to sync cache for rke-machine.cattle.io/v1, Kind=Amazonec2Machine
2022/07/22 07:52:25 [ERROR] failed to sync cache for rke-machine.cattle.io/v1, Kind=DigitaloceanMachineTemplate
2022/07/22 07:52:25 [ERROR] failed to start controller for rke-machine-config.cattle.io/v1, Kind=Amazonec2Config: failed to wait for caches to sync
2022/07/22 07:52:25 [ERROR] failed to start controller for rke-machine.cattle.io/v1, Kind=Amazonec2MachineTemplate: failed to wait for caches to sync
2022/07/22 07:52:25 [ERROR] failed to start controller for rke-machine.cattle.io/v1, Kind=LinodeMachineTemplate: failed to wait for caches to sync
2022/07/22 07:52:25 [ERROR] failed to start controller for rke-machine.cattle.io/v1, Kind=Amazonec2Machine: failed to wait for caches to sync
2022/07/22 07:52:25 [ERROR] failed to start controller for rke-machine.cattle.io/v1, Kind=DigitaloceanMachineTemplate: failed to wait for caches to sync
2022/07/22 07:52:30 [ERROR] Failed to handling tunnel request from remote address 10.42.3.9:52792: response 400: cluster not found
2022/07/22 07:52:31 [ERROR] error parsing azure-group-cache-size, skipping update strconv.Atoi: parsing "": invalid syntax
2022/07/22 07:52:33 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=kube-public, err=Operation cannot be fulfilled on namespaces "kube-public": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:33 [ERROR] Failed to handling tunnel request from remote address 10.42.3.10:45098: response 400: cluster not found
2022/07/22 07:52:34 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=kube-node-lease, err=Operation cannot be fulfilled on namespaces "kube-node-lease": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:34 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-system, err=Operation cannot be fulfilled on namespaces "cattle-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:35 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "default": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:35 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cert-manager, err=Operation cannot be fulfilled on namespaces "cert-manager": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:35 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "kube-node-lease": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:35 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "kube-public": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:35 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-global-data": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:35 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-global-nt, err=Operation cannot be fulfilled on namespaces "cattle-global-nt": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:35 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "fleet-default": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:36 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-global-data": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:36 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:36 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-global-data, err=Operation cannot be fulfilled on namespaces "cattle-global-data": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:36 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "kube-public": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:37 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-impersonation-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:37 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-g5cnd, err=Operation cannot be fulfilled on namespaces "p-g5cnd": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:37 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-nbq6f, err=Operation cannot be fulfilled on namespaces "p-nbq6f": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:37 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-fleet-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:38 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-fleet-system, err=Operation cannot be fulfilled on namespaces "cattle-fleet-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:38 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-impersonation-system, err=Operation cannot be fulfilled on namespaces "cattle-impersonation-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:39 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cert-manager": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:39 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-impersonation-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:39 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=fleet-local, err=Operation cannot be fulfilled on namespaces "fleet-local": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:39 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-fleet-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:40 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-global-nt, err=Operation cannot be fulfilled on namespaces "cattle-global-nt": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:40 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:40 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cert-manager": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:40 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-impersonation-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:40 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-fleet-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:40 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-impersonation-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:41 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cert-manager": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:42 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:43 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:43 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:44 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "kube-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:52:44 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:46 [ERROR] Failed to serve peer connection 10.42.3.9: websocket: close 1006 (abnormal closure): unexpected EOF
2022/07/22 07:52:46 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:46 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:46 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:55 [ERROR] Failed to handling tunnel request from remote address 10.42.3.10:45108: response 400: cluster not found
2022/07/22 07:53:01 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-fleet-clusters-system, err=Operation cannot be fulfilled on namespaces "cattle-fleet-clusters-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:53:01 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-fleet-clusters-system, err=Operation cannot be fulfilled on namespaces "cattle-fleet-clusters-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:53:01 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-fleet-clusters-system, err=Operation cannot be fulfilled on namespaces "cattle-fleet-clusters-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:53:01 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-fleet-clusters-system, err=Operation cannot be fulfilled on namespaces "cattle-fleet-clusters-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:53:01 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-fleet-clusters-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:53:41 [ERROR] error syncing 'cattle-fleet-system/helm-operation-lwxdm': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-lwxdm), requeuing
2022/07/22 07:53:47 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cluster-fleet-local-local-1a3d67d0a899, err=Operation cannot be fulfilled on namespaces "cluster-fleet-local-local-1a3d67d0a899": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:53:47 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cluster-fleet-local-local-1a3d67d0a899": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:53:49 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-fleet-local-system, err=Operation cannot be fulfilled on namespaces "cattle-fleet-local-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:53:50 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-fleet-local-system, err=Operation cannot be fulfilled on namespaces "cattle-fleet-local-system": the object has been modified; please apply your changes to the latest version and try again
2022/07/22 07:53:52 [ERROR] error syncing 'cattle-fleet-system/helm-operation-czbd5': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-czbd5), requeuing
2022/07/22 07:53:52 [ERROR] error syncing 'cattle-fleet-system/helm-operation-czbd5': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-czbd5), requeuing
2022/07/22 07:53:52 [ERROR] error syncing 'cattle-fleet-system/helm-operation-czbd5': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-czbd5), requeuing
2022/07/22 07:53:52 [ERROR] error syncing 'cattle-fleet-system/helm-operation-czbd5': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-czbd5), requeuing
2022/07/22 07:53:53 [ERROR] error syncing 'cattle-fleet-system/helm-operation-czbd5': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-czbd5), requeuing
2022/07/22 07:53:53 [ERROR] error syncing 'cattle-fleet-system/helm-operation-czbd5': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-czbd5), requeuing
2022/07/22 07:53:53 [ERROR] error syncing 'cattle-fleet-system/helm-operation-czbd5': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-czbd5), requeuing
2022/07/22 07:53:58 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: secrets "rancher-webhook-tls" already exists, requeuing

Here is the log of another rancher pod:

root@m2:~# kubectl -n cattle-system logs pod/rancher-864b9b7d48-hfn99 | grep ERROR
2022/07/22 07:52:33 [ERROR] Failed to connect to peer wss://10.42.3.9/v3/connect [local ID=10.42.3.10]: websocket: bad handshake
2022/07/22 07:52:33 [ERROR] Failed to connect to peer wss://10.42.2.15/v3/connect [local ID=10.42.3.10]: websocket: bad handshake
2022/07/22 07:52:36 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:36 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:40 [ERROR] Failed to connect to peer wss://10.42.2.15/v3/connect [local ID=10.42.3.10]: read tcp 10.42.3.10:45102->10.42.2.15:443: read: connection reset by peer
2022/07/22 07:52:41 [ERROR] Failed to connect to peer wss://10.42.3.9/v3/connect [local ID=10.42.3.10]: websocket: bad handshake
2022/07/22 07:52:44 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:45 [ERROR] Failed to connect to peer wss://10.42.2.15/v3/connect [local ID=10.42.3.10]: EOF
2022/07/22 07:52:46 [ERROR] Failed to connect to peer wss://10.42.3.9/v3/connect [local ID=10.42.3.10]: EOF
2022/07/22 07:52:46 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:46 [ERROR] error syncing 'cattle-system/serving-cert': handler tls-storage: Secret "serving-cert" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value], requeuing
2022/07/22 07:52:50 [ERROR] Failed to connect to peer wss://10.42.2.15/v3/connect [local ID=10.42.3.10]: read tcp 10.42.3.10:45106->10.42.2.15:443: read: connection reset by peer
2022/07/22 07:52:51 [ERROR] Failed to connect to peer wss://10.42.3.9/v3/connect [local ID=10.42.3.10]: EOF
2022/07/22 07:52:55 [ERROR] Failed to connect to peer wss://10.42.2.15/v3/connect [local ID=10.42.3.10]: websocket: bad handshake
2022/07/22 07:52:56 [ERROR] Failed to connect to peer wss://10.42.3.9/v3/connect [local ID=10.42.3.10]: websocket: bad handshake

Here is the status of the pods:

root@m2:~# kubectl get pods -A
NAMESPACE                   NAME                                      READY   STATUS      RESTARTS      AGE
kube-system                 helm-install-traefik-crd-zxznm            0/1     Completed   0             70m
kube-system                 helm-install-traefik-wkbzg                0/1     Completed   1 (69m ago)   70m
kube-system                 svclb-traefik-0759131d-mwp9m              2/2     Running     0             69m
kube-system                 traefik-df4ff85d6-qp757                   1/1     Running     0             69m
kube-system                 helm-install-traefik-crd-7bcsj            0/1     Completed   0             69m
kube-system                 helm-install-traefik-7npqx                0/1     Completed   0             69m
kube-system                 svclb-traefik-0759131d-f9qkg              2/2     Running     2 (67m ago)   69m
kube-system                 coredns-d76bd69b-nfc4p                    1/1     Running     1 (67m ago)   70m
kube-system                 metrics-server-7cd5fcb6b7-l6kxk           1/1     Running     1 (67m ago)   109m
kube-system                 local-path-provisioner-6c79684f77-zh4vg   1/1     Running     1 (66m ago)   109m
cert-manager                cert-manager-76d44b459c-96fsw             1/1     Running     0             51m
cert-manager                cert-manager-cainjector-9b679cc6-rs9gz    1/1     Running     0             51m
cert-manager                cert-manager-webhook-57c994b6b9-6t5kl     1/1     Running     0             51m
cattle-system               rancher-864b9b7d48-7rv52                  1/1     Running     0             49m
cattle-system               rancher-864b9b7d48-hr9t4                  1/1     Running     0             49m
cattle-system               rancher-864b9b7d48-hfn99                  1/1     Running     1 (47m ago)   49m
cattle-fleet-system         gitjob-6b977748fc-dn622                   1/1     Running     0             46m
cattle-fleet-system         fleet-controller-784d6fbcd8-dwgnt         1/1     Running     0             46m
cattle-system               helm-operation-8p92x                      0/2     Completed   0             47m
cattle-system               helm-operation-lwxdm                      0/2     Completed   0             46m
cattle-system               helm-operation-czbd5                      0/2     Completed   0             46m
cattle-fleet-local-system   fleet-agent-699b5fb945-6467t              1/1     Running     0             45m
cattle-system               rancher-webhook-5b65595df9-m5c8g          1/1     Running     0             45m
cattle-system               helm-operation-kvggh                      0/2     Completed   0             45m

kubecnfig:

root@m2:~# kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://192.168.1.52:6443
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

anyone knows how to fix the bug?