Hello,
i am a little bit lost.
In my homelab i have installed a k3s cluster ha. on this k3s I have installed rancher with following command.
i using static dns entries on my linux hosts
helm install rancher rancher-stable/rancher
–version 2.7.5
–namespace cattle-system
–set hostname=rancher.fritz.box
–set bootstrapPassword=admin
after the installation i want to deploy a rke2 cluster for my 4 test host.
but the node will be hang here
[INFO ] waiting for infrastructure ready
[INFO ] waiting for at least one control plane, etcd, and worker node to be registered
[INFO ] waiting for viable init node
[INFO ] configuring bootstrap node(s) custom-457540da32a5: waiting for agent to check in and apply initial plan
[INFO ] configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico, etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kubelet
[INFO ] configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico, etcd, kube-apiserver, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico, kube-apiserver, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
[INFO ] configuring bootstrap node(s) custom-457540da32a5: waiting for cluster agent to connect
Waiting for cluster agent to connect
I looked in to the agent but I am not shure what is the meaning of this error.
kubectl logs -n cattle-system rancher-669557c8fd-jpvz8
2023/10/18 11:12:10 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:12:11 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:12:12 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:12:31 [ERROR] [rkebootstrap] fleet-default/custom-075520fe920c: error getting machine by owner reference no matching controller owner ref
2023/10/18 11:12:31 [ERROR] error syncing 'fleet-default/custom-075520fe920c': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/10/18 11:12:32 [ERROR] [rkebootstrap] fleet-default/custom-075520fe920c: error getting machine by owner reference no matching controller owner ref
2023/10/18 11:12:32 [ERROR] error syncing 'fleet-default/custom-075520fe920c': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/10/18 11:12:32 [ERROR] [rkebootstrap] fleet-default/custom-075520fe920c: error getting machine by owner reference no matching controller owner ref
2023/10/18 11:12:32 [ERROR] error syncing 'fleet-default/custom-075520fe920c': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/10/18 11:12:32 [ERROR] [rkebootstrap] fleet-default/custom-075520fe920c: error getting machine by owner reference no matching controller owner ref
2023/10/18 11:12:32 [ERROR] error syncing 'fleet-default/custom-075520fe920c': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/10/18 11:12:32 [ERROR] [rkebootstrap] fleet-default/custom-075520fe920c: error getting machine by owner reference no matching controller owner ref
2023/10/18 11:12:32 [ERROR] error syncing 'fleet-default/custom-075520fe920c': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/10/18 11:12:32 [ERROR] [rkebootstrap] fleet-default/custom-075520fe920c: error getting machine by owner reference no matching controller owner ref
2023/10/18 11:12:32 [ERROR] error syncing 'fleet-default/custom-075520fe920c': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/10/18 11:12:32 [ERROR] [rkebootstrap] fleet-default/custom-075520fe920c: error getting machine by owner reference no matching controller owner ref
2023/10/18 11:12:32 [ERROR] error syncing 'fleet-default/custom-075520fe920c': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/10/18 11:12:33 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:12:33 [ERROR] [rkebootstrap] fleet-default/custom-075520fe920c: error getting machine by owner reference no matching controller owner ref
2023/10/18 11:12:33 [ERROR] error syncing 'fleet-default/custom-075520fe920c': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/10/18 11:12:34 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:12:38 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:12:39 [INFO] EnsureSecretForServiceAccount: waiting for secret [custom-075520fe920c-machine-bootstrap-token-qlr6k] to be populated with token
2023/10/18 11:12:40 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:12:41 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:12:42 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:12:44 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:12:45 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for probes: calico
2023/10/18 11:13:06 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for cluster agent to connect
2023/10/18 11:13:08 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for cluster agent to connect
2023/10/18 11:13:10 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for cluster agent to connect
2023/10/18 11:14:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:16:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:17:12 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for cluster agent to connect
2023/10/18 11:18:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:20:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:22:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:24:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:26:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:27:12 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for cluster agent to connect
2023/10/18 11:28:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:30:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:32:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:34:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:36:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:37:13 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for cluster agent to connect
2023/10/18 11:38:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:40:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:42:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:44:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:46:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:47:13 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for cluster agent to connect
2023/10/18 11:48:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:50:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:52:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:54:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:56:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 11:57:13 [INFO] [planner] rkecluster fleet-default/mykube1: waiting: configuring bootstrap node(s) custom-457540da32a5: waiting for cluster agent to connect
2023/10/18 11:58:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 12:00:01 [INFO] [snapshotbackpopulate] rkecluster fleet-local/local: processing configmap kube-system/k3s-etcd-snapshots
2023/10/18 12:00:02 [INFO] [snapshotbackpopulate] rkecluster fleet-local/local: processing configmap kube-system/k3s-etcd-snapshots
2023/10/18 12:00:03 [INFO] [snapshotbackpopulate] rkecluster fleet-local/local: processing configmap kube-system/k3s-etcd-snapshots
2023/10/18 12:00:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
2023/10/18 12:02:07 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers for cluster c-m-qgbx2dtz: ClusterUnavailable 503: cluster not found, requeuing
here is my yaml config from my test cluster
apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
annotations:
field.cattle.io/creatorId: user-hntfd
creationTimestamp: '2023-10-18T11:06:08Z'
finalizers:
- wrangler.cattle.io/provisioning-cluster-remove
- wrangler.cattle.io/rke-cluster-remove
- wrangler.cattle.io/cloud-config-secret-remover
generation: 2
managedFields:
- apiVersion: provisioning.cattle.io/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.: {}
v:"wrangler.cattle.io/provisioning-cluster-remove": {}
v:"wrangler.cattle.io/rke-cluster-remove": {}
f:spec:
.: {}
f:kubernetesVersion: {}
f:localClusterAuthEndpoint: {}
f:rkeConfig:
.: {}
f:chartValues:
.: {}
f:rke2-calico: {}
f:etcd:
.: {}
f:snapshotRetention: {}
f:snapshotScheduleCron: {}
f:machineGlobalConfig:
.: {}
f:cluster-cidr: {}
f:cni: {}
f:disable-kube-proxy: {}
f:etcd-expose-metrics: {}
f:service-cidr: {}
f:machinePoolDefaults: {}
f:machineSelectorConfig: {}
f:registries: {}
f:upgradeStrategy:
.: {}
f:controlPlaneConcurrency: {}
f:controlPlaneDrainOptions:
.: {}
f:deleteEmptyDirData: {}
f:disableEviction: {}
f:enabled: {}
f:force: {}
f:gracePeriod: {}
f:ignoreDaemonSets: {}
f:ignoreErrors: {}
f:postDrainHooks: {}
f:preDrainHooks: {}
f:skipWaitForDeleteTimeoutSeconds: {}
f:timeout: {}
f:workerConcurrency: {}
f:workerDrainOptions:
.: {}
f:deleteEmptyDirData: {}
f:disableEviction: {}
f:enabled: {}
f:force: {}
f:gracePeriod: {}
f:ignoreDaemonSets: {}
f:ignoreErrors: {}
f:postDrainHooks: {}
f:preDrainHooks: {}
f:skipWaitForDeleteTimeoutSeconds: {}
f:timeout: {}
manager: rancher
operation: Update
time: '2023-10-18T11:06:10Z'
- apiVersion: provisioning.cattle.io/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
v:"wrangler.cattle.io/cloud-config-secret-remover": {}
manager: rancher-v2.7.5-secret-migrator
operation: Update
time: '2023-10-18T11:06:11Z'
- apiVersion: provisioning.cattle.io/v1
fieldsType: FieldsV1
fieldsV1:
f:status:
.: {}
f:clusterName: {}
f:conditions: {}
f:observedGeneration: {}
manager: rancher
operation: Update
subresource: status
time: '2023-10-18T11:13:07Z'
name: mykube1
namespace: fleet-default
resourceVersion: '32049'
uid: 66a5f42e-6c1e-4507-8811-dd9c44bb9b4d
spec:
kubernetesVersion: v1.25.13+rke2r1
localClusterAuthEndpoint: {}
rkeConfig:
chartValues:
rke2-calico: {}
etcd:
snapshotRetention: 5
snapshotScheduleCron: 0 */5 * * *
machineGlobalConfig:
cluster-cidr: 10.244.0.0/16
cni: calico
disable-kube-proxy: false
etcd-expose-metrics: false
service-cidr: 10.96.0.0/12
machinePoolDefaults: {}
machineSelectorConfig:
- config:
protect-kernel-defaults: false
registries: {}
upgradeStrategy:
controlPlaneConcurrency: '1'
controlPlaneDrainOptions:
deleteEmptyDirData: true
disableEviction: false
enabled: false
force: false
gracePeriod: -1
ignoreDaemonSets: true
ignoreErrors: false
postDrainHooks: null
preDrainHooks: null
skipWaitForDeleteTimeoutSeconds: 0
timeout: 120
workerConcurrency: '1'
workerDrainOptions:
deleteEmptyDirData: true
disableEviction: false
enabled: false
force: false
gracePeriod: -1
ignoreDaemonSets: true
ignoreErrors: false
postDrainHooks: null
preDrainHooks: null
skipWaitForDeleteTimeoutSeconds: 0
timeout: 120
status:
clusterName: c-m-qgbx2dtz
conditions:
- lastUpdateTime: '2023-10-18T11:06:08Z'
reason: Reconciling
status: 'True'
type: Reconciling
- lastUpdateTime: '2023-10-18T11:06:08Z'
status: 'False'
type: Stalled
- lastUpdateTime: '2023-10-18T11:06:23Z'
status: 'True'
type: Created
- lastUpdateTime: '2023-10-18T11:13:07Z'
status: 'True'
type: RKECluster
- lastUpdateTime: '2023-10-18T11:06:09Z'
status: 'True'
type: BackingNamespaceCreated
- lastUpdateTime: '2023-10-18T11:06:09Z'
status: 'True'
type: DefaultProjectCreated
- lastUpdateTime: '2023-10-18T11:06:10Z'
status: 'True'
type: SystemProjectCreated
- lastUpdateTime: '2023-10-18T11:06:11Z'
status: 'True'
type: InitialRolesPopulated
- lastUpdateTime: '2023-10-18T11:13:07Z'
message: >-
configuring bootstrap node(s) custom-457540da32a5: waiting for cluster
agent to connect
reason: Waiting
status: Unknown
type: Updated
- lastUpdateTime: '2023-10-18T11:13:07Z'
message: >-
configuring bootstrap node(s) custom-457540da32a5: waiting for cluster
agent to connect
reason: Waiting
status: Unknown
type: Provisioned
- lastUpdateTime: '2023-10-18T11:13:07Z'
message: >-
configuring bootstrap node(s) custom-457540da32a5: waiting for cluster
agent to connect
reason: Waiting
status: Unknown
type: Ready
- lastUpdateTime: '2023-10-18T11:06:15Z'
status: 'True'
type: CreatorMadeOwner
- lastUpdateTime: '2023-10-18T11:06:20Z'
status: 'True'
type: NoDiskPressure
- lastUpdateTime: '2023-10-18T11:06:20Z'
status: 'True'
type: NoMemoryPressure
- lastUpdateTime: '2023-10-18T11:06:21Z'
status: 'True'
type: SecretsMigrated
- lastUpdateTime: '2023-10-18T11:06:21Z'
status: 'False'
type: Connected
- lastUpdateTime: '2023-10-18T11:06:22Z'
status: 'True'
type: ServiceAccountSecretsMigrated
- lastUpdateTime: '2023-10-18T11:06:22Z'
status: 'True'
type: RKESecretsMigrated
- lastUpdateTime: '2023-10-18T11:06:22Z'
status: 'True'
type: ACISecretsMigrated
observedGeneration: 2