Nodes hang "Waiting for cluster agent to connect"

Hello,

i am a little bit lost.
In my homelab i have installed a k3s cluster ha. on this k3s I have installed rancher with following command.

i using static dns entries on my linux hosts

helm install rancher rancher-stable/rancher
–version 2.7.5
–namespace cattle-system
–set hostname=rancher.fritz.box

–set bootstrapPassword=admin

after the installation i want to deploy a rke2 cluster for my 4 test host.
but the node will be hang here

[INFO ] waiting for infrastructure ready
[INFO ] waiting for at least one control plane, etcd, and worker node to be registered
[INFO ] waiting for viable init node
[INFO ] configuring bootstrap node(s) custom-457540da32a5: waiting for agent to check in and apply initial plan
[INFO ] configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico, etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kubelet
[INFO ] configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico, etcd, kube-apiserver, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico, kube-apiserver, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
[INFO ] configuring bootstrap node(s) custom-457540da32a5: waiting for cluster agent to connect

Waiting for cluster agent to connect

I looked in to the agent but I am not shure what is the meaning of this error.

kubectl logs -n cattle-system rancher-669557c8fd-jpvz8

2023/10/18 11:12:10 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:12:11 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:12:12 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:12:31 [ERROR] [rkebootstrap] fleet-default/custom-075520fe920c: error getting machine by owner reference no matching controller owner ref
2023/10/18 11:12:31 [ERROR] error syncing 'fleet-default/custom-075520fe920c': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/10/18 11:12:32 [ERROR] [rkebootstrap] fleet-default/custom-075520fe920c: error getting machine by owner reference no matching controller owner ref
2023/10/18 11:12:32 [ERROR] error syncing 'fleet-default/custom-075520fe920c': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/10/18 11:12:32 [ERROR] [rkebootstrap] fleet-default/custom-075520fe920c: error getting machine by owner reference no matching controller owner ref
2023/10/18 11:12:32 [ERROR] error syncing 'fleet-default/custom-075520fe920c': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/10/18 11:12:32 [ERROR] [rkebootstrap] fleet-default/custom-075520fe920c: error getting machine by owner reference no matching controller owner ref
2023/10/18 11:12:32 [ERROR] error syncing 'fleet-default/custom-075520fe920c': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/10/18 11:12:32 [ERROR] [rkebootstrap] fleet-default/custom-075520fe920c: error getting machine by owner reference no matching controller owner ref
2023/10/18 11:12:32 [ERROR] error syncing 'fleet-default/custom-075520fe920c': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/10/18 11:12:32 [ERROR] [rkebootstrap] fleet-default/custom-075520fe920c: error getting machine by owner reference no matching controller owner ref
2023/10/18 11:12:32 [ERROR] error syncing 'fleet-default/custom-075520fe920c': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/10/18 11:12:32 [ERROR] [rkebootstrap] fleet-default/custom-075520fe920c: error getting machine by owner reference no matching controller owner ref
2023/10/18 11:12:32 [ERROR] error syncing 'fleet-default/custom-075520fe920c': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/10/18 11:12:33 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:12:33 [ERROR] [rkebootstrap] fleet-default/custom-075520fe920c: error getting machine by owner reference no matching controller owner ref
2023/10/18 11:12:33 [ERROR] error syncing 'fleet-default/custom-075520fe920c': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/10/18 11:12:34 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:12:38 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:12:39 [INFO] EnsureSecretForServiceAccount: waiting for secret [custom-075520fe920c-machine-bootstrap-token-qlr6k] to be populated with token
2023/10/18 11:12:40 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:12:41 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:12:42 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:12:44 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:12:45 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:13:06 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for cluster agent to connect
2023/10/18 11:13:08 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for cluster agent to connect
2023/10/18 11:13:10 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for cluster agent to connect
2023/10/18 11:14:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:16:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:17:12 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for cluster agent to connect
2023/10/18 11:18:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:20:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:22:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:24:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:26:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:27:12 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for cluster agent to connect
2023/10/18 11:28:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:30:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:32:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:34:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:36:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:37:13 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for cluster agent to connect
2023/10/18 11:38:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:40:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:42:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:44:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:46:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:47:13 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for cluster agent to connect
2023/10/18 11:48:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:50:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:52:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:54:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:56:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:57:13 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for cluster agent to connect
2023/10/18 11:58:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 12:00:01 [INFO] [snapshotbackpopulate] rkecluster fleet-local/local: processing configmap kube-system/k3s-etcd-snapshots
2023/10/18 12:00:02 [INFO] [snapshotbackpopulate] rkecluster fleet-local/local: processing configmap kube-system/k3s-etcd-snapshots
2023/10/18 12:00:03 [INFO] [snapshotbackpopulate] rkecluster fleet-local/local: processing configmap kube-system/k3s-etcd-snapshots
2023/10/18 12:00:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 12:02:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing

here is my yaml config from my test cluster

apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
  annotations:
    field.cattle.io/creatorId: user-hntfd
  creationTimestamp: '2023-10-18T11:06:08Z'
  finalizers:
    - wrangler.cattle.io/provisioning-cluster-remove
    - wrangler.cattle.io/rke-cluster-remove
    - wrangler.cattle.io/cloud-config-secret-remover
  generation: 2
  managedFields:
    - apiVersion: provisioning.cattle.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:finalizers:
            .: {}
            v:"wrangler.cattle.io/provisioning-cluster-remove": {}
            v:"wrangler.cattle.io/rke-cluster-remove": {}
        f:spec:
          .: {}
          f:kubernetesVersion: {}
          f:localClusterAuthEndpoint: {}
          f:rkeConfig:
            .: {}
            f:chartValues:
              .: {}
              f:rke2-calico: {}
            f:etcd:
              .: {}
              f:snapshotRetention: {}
              f:snapshotScheduleCron: {}
            f:machineGlobalConfig:
              .: {}
              f:cluster-cidr: {}
              f:cni: {}
              f:disable-kube-proxy: {}
              f:etcd-expose-metrics: {}
              f:service-cidr: {}
            f:machinePoolDefaults: {}
            f:machineSelectorConfig: {}
            f:registries: {}
            f:upgradeStrategy:
              .: {}
              f:controlPlaneConcurrency: {}
              f:controlPlaneDrainOptions:
                .: {}
                f:deleteEmptyDirData: {}
                f:disableEviction: {}
                f:enabled: {}
                f:force: {}
                f:gracePeriod: {}
                f:ignoreDaemonSets: {}
                f:ignoreErrors: {}
                f:postDrainHooks: {}
                f:preDrainHooks: {}
                f:skipWaitForDeleteTimeoutSeconds: {}
                f:timeout: {}
              f:workerConcurrency: {}
              f:workerDrainOptions:
                .: {}
                f:deleteEmptyDirData: {}
                f:disableEviction: {}
                f:enabled: {}
                f:force: {}
                f:gracePeriod: {}
                f:ignoreDaemonSets: {}
                f:ignoreErrors: {}
                f:postDrainHooks: {}
                f:preDrainHooks: {}
                f:skipWaitForDeleteTimeoutSeconds: {}
                f:timeout: {}
      manager: rancher
      operation: Update
      time: '2023-10-18T11:06:10Z'
    - apiVersion: provisioning.cattle.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:finalizers:
            v:"wrangler.cattle.io/cloud-config-secret-remover": {}
      manager: rancher-v2.7.5-secret-migrator
      operation: Update
      time: '2023-10-18T11:06:11Z'
    - apiVersion: provisioning.cattle.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:clusterName: {}
          f:conditions: {}
          f:observedGeneration: {}
      manager: rancher
      operation: Update
      subresource: status
      time: '2023-10-18T11:13:07Z'
  name: mykube1
  namespace: fleet-default
  resourceVersion: '32049'
  uid: 66a5f42e-6c1e-4507-8811-dd9c44bb9b4d
spec:
  kubernetesVersion: v1.25.13+rke2r1
  localClusterAuthEndpoint: {}
  rkeConfig:
    chartValues:
      rke2-calico: {}
    etcd:
      snapshotRetention: 5
      snapshotScheduleCron: 0 */5 * * *
    machineGlobalConfig:
      cluster-cidr: 10.244.0.0/16
      cni: calico
      disable-kube-proxy: false
      etcd-expose-metrics: false
      service-cidr: 10.96.0.0/12
    machinePoolDefaults: {}
    machineSelectorConfig:
      - config:
          protect-kernel-defaults: false
    registries: {}
    upgradeStrategy:
      controlPlaneConcurrency: '1'
      controlPlaneDrainOptions:
        deleteEmptyDirData: true
        disableEviction: false
        enabled: false
        force: false
        gracePeriod: -1
        ignoreDaemonSets: true
        ignoreErrors: false
        postDrainHooks: null
        preDrainHooks: null
        skipWaitForDeleteTimeoutSeconds: 0
        timeout: 120
      workerConcurrency: '1'
      workerDrainOptions:
        deleteEmptyDirData: true
        disableEviction: false
        enabled: false
        force: false
        gracePeriod: -1
        ignoreDaemonSets: true
        ignoreErrors: false
        postDrainHooks: null
        preDrainHooks: null
        skipWaitForDeleteTimeoutSeconds: 0
        timeout: 120
status:
  clusterName: c-m-qgbx2dtz
  conditions:
    - lastUpdateTime: '2023-10-18T11:06:08Z'
      reason: Reconciling
      status: 'True'
      type: Reconciling
    - lastUpdateTime: '2023-10-18T11:06:08Z'
      status: 'False'
      type: Stalled
    - lastUpdateTime: '2023-10-18T11:06:23Z'
      status: 'True'
      type: Created
    - lastUpdateTime: '2023-10-18T11:13:07Z'
      status: 'True'
      type: RKECluster
    - lastUpdateTime: '2023-10-18T11:06:09Z'
      status: 'True'
      type: BackingNamespaceCreated
    - lastUpdateTime: '2023-10-18T11:06:09Z'
      status: 'True'
      type: DefaultProjectCreated
    - lastUpdateTime: '2023-10-18T11:06:10Z'
      status: 'True'
      type: SystemProjectCreated
    - lastUpdateTime: '2023-10-18T11:06:11Z'
      status: 'True'
      type: InitialRolesPopulated
    - lastUpdateTime: '2023-10-18T11:13:07Z'
      message: >-
        configuring bootstrap node(s) custom-457540da32a5: waiting for cluster
        agent to connect
      reason: Waiting
      status: Unknown
      type: Updated
    - lastUpdateTime: '2023-10-18T11:13:07Z'
      message: >-
        configuring bootstrap node(s) custom-457540da32a5: waiting for cluster
        agent to connect
      reason: Waiting
      status: Unknown
      type: Provisioned
    - lastUpdateTime: '2023-10-18T11:13:07Z'
      message: >-
        configuring bootstrap node(s) custom-457540da32a5: waiting for cluster
        agent to connect
      reason: Waiting
      status: Unknown
      type: Ready
    - lastUpdateTime: '2023-10-18T11:06:15Z'
      status: 'True'
      type: CreatorMadeOwner
    - lastUpdateTime: '2023-10-18T11:06:20Z'
      status: 'True'
      type: NoDiskPressure
    - lastUpdateTime: '2023-10-18T11:06:20Z'
      status: 'True'
      type: NoMemoryPressure
    - lastUpdateTime: '2023-10-18T11:06:21Z'
      status: 'True'
      type: SecretsMigrated
    - lastUpdateTime: '2023-10-18T11:06:21Z'
      status: 'False'
      type: Connected
    - lastUpdateTime: '2023-10-18T11:06:22Z'
      status: 'True'
      type: ServiceAccountSecretsMigrated
    - lastUpdateTime: '2023-10-18T11:06:22Z'
      status: 'True'
      type: RKESecretsMigrated
    - lastUpdateTime: '2023-10-18T11:06:22Z'
      status: 'True'
      type: ACISecretsMigrated
  observedGeneration: 2

i’ve the same problem… do you resolved?