Hi,
We upgraded Rancher from 2.5.15 to 2.6.0 (deployed in a Docker container). The downstream cluster is working fine and Rancher is able to manage it. However, on the local cluster only coredns is up and running. I see a lot of errored helm-operation pods. The logs of the helm-operation pods complain about inavailability of the Kubernetes API. The Docker logs show a bunch of “error syncing xxx … services webhook-service not found” every minute
2023-02-27 11:56:34.637075 I | mvcc: store.index: compact 179969231
2023-02-27 11:56:34.693136 I | mvcc: finished scheduled compaction at 179969231 (took 54.332494ms)
2023/02/27 11:56:46 [ERROR] error syncing 'machinesets.cluster.x-k8s.io': handler need-a-cert: services "webhook-service" not found, requeuing
2023/02/27 11:56:46 [ERROR] error syncing 'machines.cluster.x-k8s.io': handler need-a-cert: services "webhook-service" not found, requeuing
2023/02/27 11:56:46 [ERROR] error syncing 'machinedeployments.cluster.x-k8s.io': handler need-a-cert: services "webhook-service" not found, requeuing
2023/02/27 11:56:46 [ERROR] error syncing 'clusters.cluster.x-k8s.io': handler need-a-cert: services "webhook-service" not found, requeuing
2023/02/27 11:56:46 [ERROR] error syncing 'machinehealthchecks.cluster.x-k8s.io': handler need-a-cert: services "webhook-service" not found, requeuing
2023/02/27 11:56:49 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2023/02/27 11:56:49 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2023/02/27 11:56:57 [ERROR] error syncing 'cattle-fleet-system/helm-operation-vdkcg': handler helm-operation: Operation cannot be fulfilled on operations.catalog.cattle.io "helm-operation-vdkcg": StorageError: invalid object, Code: 4, Key: /registry/catalog.cattle.io/operations/cattle-fleet-system/helm-operation-vdkcg, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 173a7345-6b78-433e-b52d-b0d85f00de0b, UID in object meta: , requeuing
2023/02/27 11:57:08 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-dp6z9 failed, watch closed
2023/02/27 11:58:08 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-c6ljh failed, watch closed
Any ideas to fix this problem are appreciated.
Many thanks in advance