Hi Rancher! how are you?
using rancher2_cluster_v2 I get some errors while provisioning and the cluster fails to finish provisioning / registration
rancher pod:
2023/08/24 13:18:12 [INFO] Loaded configuration from https://releases.rancher.com/kontainer-driver-metadata/release-v2.7/data.json in [0x6f27ec8]
time="2023-08-24 13:18:12" level=error msg="Failed to get Cluster Loggings err=Unknown schema type [clusterLogging]"
time="2023-08-24 13:18:13" level=error msg="Failed to get Pipelines for project local:p-hvp8q err=Unknown schema type [pipeline]"
time="2023-08-24 13:18:13" level=error msg="Failed to get SourceCodeProvider for project local:p-hvp8q err=Unknown schema type [sourceCodeProvider]"
time="2023-08-24 13:18:13" level=error msg="Failed to get Pipelines for project local:p-x4qd7 err=Unknown schema type [pipeline]"
time="2023-08-24 13:18:13" level=error msg="Failed to get SourceCodeProvider for project local:p-x4qd7 err=Unknown schema type [sourceCodeProvider]"
E0824 13:18:27.560067 33 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: gitjob.cattle.io/v1: the server could not find the requested resource
E0824 13:18:27.797662 33 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: gitjob.cattle.io/v1: the server could not find the requested resource
2023/08/24 13:18:27 [ERROR] Failed to read API for groups map[gitjob.cattle.io/v1:the server could not find the requested resource]
2023/08/24 13:18:29 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistrationToken
2023/08/24 13:18:29 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistration
2023/08/24 13:18:29 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=BundleNamespaceMapping
2023/08/24 13:18:29 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=BundleDeployment
2023/08/24 13:18:29 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ImageScan
2023/08/24 13:18:29 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=GitRepoRestriction
2023/08/24 13:18:29 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=GitRepo
2023/08/24 13:18:29 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Content
2023/08/24 13:18:29 [ERROR] error syncing 'cattle-fleet-system/helm-operation-nj2sg': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-nj2sg), requeuing
2023/08/24 13:18:29 [ERROR] error syncing 'cattle-fleet-system/helm-operation-nj2sg': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-nj2sg), requeuing
2023/08/24 13:18:30 [ERROR] error syncing 'cattle-fleet-system/helm-operation-nj2sg': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-nj2sg), requeuing
2023/08/24 13:18:30 [ERROR] error syncing 'cattle-fleet-system/helm-operation-nj2sg': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-nj2sg), requeuing
2023/08/24 13:18:30 [ERROR] error syncing 'cattle-fleet-system/helm-operation-nj2sg': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-nj2sg), requeuing
2023/08/24 13:18:30 [ERROR] error syncing 'cattle-fleet-system/helm-operation-nj2sg': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-nj2sg), requeuing
2023/08/24 13:18:30 [ERROR] error syncing 'cattle-fleet-system/helm-operation-nj2sg': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-nj2sg), requeuing
2023/08/24 13:18:30 [INFO] Watching metadata for gitjob.cattle.io/v1, Kind=GitJob
2023/08/24 13:19:02 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca@1692613019,O=dynamiclistener-org: notBefore=2023-08-21 10:16:59 +0000 UTC notAfter=2024-08-23 13:19:02 +0000 UTC
2023/08/24 13:19:02 [INFO] Updating TLS secret for cattle-system/tls-rancher-internal (count: 10): map[field.cattle.io/projectId:local:p-hvp8q listener.cattle.io/cn-10.42.0.100:10.42.0.100 listener.cattle.io/cn-10.42.0.109:10.42.0.109 listener.cattle.io/cn-10.42.0.122:10.42.0.122 listener.cattle.io/cn-10.42.0.18:10.42.0.18 listener.cattle.io/cn-10.42.0.93:10.42.0.93 listener.cattle.io/cn-10.42.0.95:10.42.0.95 listener.cattle.io/cn-10.43.120.188:10.43.120.188 listener.cattle.io/cn-10.43.15.120:10.43.15.120 listener.cattle.io/cn-10.43.45.22:10.43.45.22 listener.cattle.io/fingerprint:SHA1=28626BC923540DA985A2D7842E8AAACA888F9C63]
2023/08/24 13:19:02 [INFO] Active TLS secret cattle-system/tls-rancher-internal (ver=910728) (count 10): map[field.cattle.io/projectId:local:p-hvp8q listener.cattle.io/cn-10.42.0.100:10.42.0.100 listener.cattle.io/cn-10.42.0.109:10.42.0.109 listener.cattle.io/cn-10.42.0.122:10.42.0.122 listener.cattle.io/cn-10.42.0.18:10.42.0.18 listener.cattle.io/cn-10.42.0.93:10.42.0.93 listener.cattle.io/cn-10.42.0.95:10.42.0.95 listener.cattle.io/cn-10.43.120.188:10.43.120.188 listener.cattle.io/cn-10.43.15.120:10.43.15.120 listener.cattle.io/cn-10.43.45.22:10.43.45.22 listener.cattle.io/fingerprint:SHA1=28626BC923540DA985A2D7842E8AAACA888F9C63]
2023/08/24 13:19:02 [INFO] Updating TLS secret for cattle-system/tls-rancher-internal (count: 10): map[field.cattle.io/projectId:local:p-hvp8q listener.cattle.io/cn-10.42.0.100:10.42.0.100 listener.cattle.io/cn-10.42.0.109:10.42.0.109 listener.cattle.io/cn-10.42.0.122:10.42.0.122 listener.cattle.io/cn-10.42.0.18:10.42.0.18 listener.cattle.io/cn-10.42.0.93:10.42.0.93 listener.cattle.io/cn-10.42.0.95:10.42.0.95 listener.cattle.io/cn-10.43.120.188:10.43.120.188 listener.cattle.io/cn-10.43.15.120:10.43.15.120 listener.cattle.io/cn-10.43.45.22:10.43.45.22 listener.cattle.io/fingerprint:SHA1=28626BC923540DA985A2D7842E8AAACA888F9C63]
2023/08/24 13:23:54 [ERROR] Error during subscribe websocket: close sent
2023/08/24 13:23:57 [INFO] [mgmt-cluster-rbac-delete] Creating namespace c-m-ppkh4pgf
2023/08/24 13:23:57 [INFO] [mgmt-cluster-rbac-delete] Creating Default project for cluster c-m-ppkh4pgf
2023/08/24 13:23:57 [INFO] [mgmt-project-rbac-create] Creating namespace p-qnxh9
2023/08/24 13:23:57 [INFO] [mgmt-cluster-rbac-delete] Creating System project for cluster c-m-ppkh4pgf
2023/08/24 13:23:57 [INFO] [mgmt-project-rbac-create] Creating namespace p-b78ww
2023/08/24 13:23:57 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=c-m-ppkh4pgf, err=Operation cannot be fulfilled on namespaces "c-m-ppkh4pgf": the object has been modified; please apply your changes to the latest version and try again
2023/08/24 13:23:57 [INFO] [mgmt-cluster-rbac-delete] Updating cluster c-m-ppkh4pgf
2023/08/24 13:23:57 [INFO] [mgmt-project-rbac-create] Creating creator projectRoleTemplateBinding for user user-xcdnk for project p-qnxh9
2023/08/24 13:23:57 [INFO] [mgmt-project-rbac-create] Creating creator clusterRoleTemplateBinding for user user-xcdnk for cluster c-m-ppkh4pgf
2023/08/24 13:23:57 [INFO] [mgmt-project-rbac-create] Creating creator projectRoleTemplateBinding for user user-xcdnk for project p-b78ww
2023/08/24 13:23:57 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=c-m-ppkh4pgf, err=Operation cannot be fulfilled on namespaces "c-m-ppkh4pgf": the object has been modified; please apply your changes to the latest version and try again
2023/08/24 13:23:57 [INFO] [mgmt-auth-crtb-controller] Setting InitialRolesPopulated condition on project p-qnxh9
2023/08/24 13:23:57 [INFO] [mgmt-auth-prtb-controller] Creating role/clusterRole p-qnxh9-projectowner
2023/08/24 13:23:57 [INFO] [mgmt-project-rbac-create] Updating project p-qnxh9
2023/08/24 13:23:57 [INFO] [mgmt-auth-crtb-controller] Setting InitialRolesPopulated condition on project p-b78ww
2023/08/24 13:23:57 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for membership in project p-qnxh9 for subject user-xcdnk
2023/08/24 13:23:57 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=c-m-ppkh4pgf, err=Operation cannot be fulfilled on namespaces "c-m-ppkh4pgf": the object has been modified; please apply your changes to the latest version and try again
2023/08/24 13:23:57 [INFO] [mgmt-project-rbac-create] Updating project p-b78ww
2023/08/24 13:23:57 [INFO] [mgmt-auth-prtb-controller] Creating role/clusterRole p-b78ww-projectowner
2023/08/24 13:23:57 [INFO] [mgmt-auth-crtb-controller] Creating role/clusterRole c-m-ppkh4pgf-clusterowner
2023/08/24 13:23:57 [INFO] [mgmt-auth-prtb-controller] Creating role/clusterRole c-m-ppkh4pgf-clustermember
2023/08/24 13:23:57 [INFO] [mgmt-project-rbac-create] Updating project p-qnxh9
2023/08/24 13:23:57 [INFO] [mgmt-auth-prtb-controller] Creating clusterRoleBinding for membership in cluster c-m-ppkh4pgf for subject user-xcdnk
2023/08/24 13:23:57 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-qnxh9, err=Operation cannot be fulfilled on namespaces "p-qnxh9": the object has been modified; please apply your changes to the latest version and try again
2023/08/24 13:23:57 [INFO] [mgmt-project-rbac-create] Updating project p-b78ww
2023/08/24 13:23:57 [INFO] [mgmt-auth-prtb-controller] Creating role project-owner in namespace c-m-ppkh4pgf
2023/08/24 13:23:57 [INFO] [mgmt-auth-crtb-controller] Creating clusterRoleBinding for membership in cluster c-m-ppkh4pgf for subject user-xcdnk
2023/08/24 13:23:57 [INFO] [mgmt-auth-crtb-controller] Setting InitialRolesPopulated condition on cluster c-m-ppkh4pgf
2023/08/24 13:23:57 [INFO] [mgmt-cluster-rbac-delete] Updating cluster c-m-ppkh4pgf
2023/08/24 13:23:57 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for membership in project p-b78ww for subject user-xcdnk
2023/08/24 13:23:57 [ERROR] [rkecluster] fleet-default/vsphere-testnotable-mustang: error getting CAPI cluster no matching controller owner ref
2023/08/24 13:23:57 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang': handler rke-cluster: no matching controller owner ref, requeuing
2023/08/24 13:23:57 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-xcdnk with role project-owner in namespace c-m-ppkh4pgf
2023/08/24 13:23:57 [ERROR] [rkecluster] fleet-default/vsphere-testnotable-mustang: error getting CAPI cluster no matching controller owner ref
2023/08/24 13:23:57 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang': handler rke-cluster: no matching controller owner ref, requeuing
2023/08/24 13:23:57 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-b78ww, err=Operation cannot be fulfilled on namespaces "p-b78ww": the object has been modified; please apply your changes to the latest version and try again
2023/08/24 13:23:57 [INFO] [mgmt-auth-prtb-controller] Updating clusterRoleBinding crb-gmpid66inc for cluster membership in cluster c-m-ppkh4pgf for subject user-xcdnk
2023/08/24 13:23:57 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-qnxh9, err=Operation cannot be fulfilled on namespaces "p-qnxh9": the object has been modified; please apply your changes to the latest version and try again
2023/08/24 13:23:57 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace c-m-ppkh4pgf
2023/08/24 13:23:57 [ERROR] [rkecluster] fleet-default/vsphere-testnotable-mustang: error getting CAPI cluster no matching controller owner ref
2023/08/24 13:23:57 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang': handler rke-cluster: no matching controller owner ref, requeuing
2023/08/24 13:23:57 [ERROR] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: error encountered during plan processing was no matching controller owner ref
2023/08/24 13:23:57 [ERROR] [rkecluster] fleet-default/vsphere-testnotable-mustang: error getting CAPI cluster no matching controller owner ref
2023/08/24 13:23:57 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang': handler rke-cluster: no matching controller owner ref, requeuing
2023/08/24 13:23:57 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject user-xcdnk with role cluster-owner in namespace c-m-ppkh4pgf
2023/08/24 13:23:57 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-xcdnk with role project-owner in namespace c-m-ppkh4pgf
2023/08/24 13:23:57 [INFO] [mgmt-auth-prtb-controller] Creating role admin in namespace p-qnxh9
2023/08/24 13:23:57 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace p-qnxh9
2023/08/24 13:23:57 [ERROR] [rkecluster] fleet-default/vsphere-testnotable-mustang: error getting CAPI cluster no matching controller owner ref
2023/08/24 13:23:57 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang': handler rke-cluster: no matching controller owner ref, requeuing
2023/08/24 13:23:58 [INFO] [mgmt-auth-prtb-controller] Creating role project-owner in namespace p-b78ww
2023/08/24 13:23:58 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject user-xcdnk with role cluster-owner in namespace p-qnxh9
2023/08/24 13:23:58 [INFO] [mgmt-auth-prtb-controller] Creating role project-owner in namespace p-qnxh9
2023/08/24 13:23:58 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-b78ww, err=Operation cannot be fulfilled on namespaces "p-b78ww": the object has been modified; please apply your changes to the latest version and try again
2023/08/24 13:23:58 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-xcdnk with role project-owner in namespace p-qnxh9
2023/08/24 13:23:58 [ERROR] [rkecluster] fleet-default/vsphere-testnotable-mustang: error getting CAPI cluster no matching controller owner ref
2023/08/24 13:23:58 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang': handler rke-cluster: no matching controller owner ref, requeuing
2023/08/24 13:23:58 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-qnxh9, err=Operation cannot be fulfilled on namespaces "p-qnxh9": the object has been modified; please apply your changes to the latest version and try again
2023/08/24 13:23:58 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-xcdnk with role admin in namespace p-qnxh9
2023/08/24 13:23:58 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace p-b78ww
2023/08/24 13:23:58 [INFO] [mgmt-auth-prtb-controller] Creating role admin in namespace p-b78ww
2023/08/24 13:23:58 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject user-xcdnk with role cluster-owner in namespace p-b78ww
2023/08/24 13:23:58 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-xcdnk with role project-owner in namespace p-b78ww
2023/08/24 13:23:58 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang': handler planner: no matching controller owner ref, requeuing
2023/08/24 13:23:58 [ERROR] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: error encountered during plan processing was no matching controller owner ref
2023/08/24 13:23:58 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-xcdnk with role admin in namespace p-b78ww
2023/08/24 13:23:58 [ERROR] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: error encountered during plan processing was no matching controller owner ref
2023/08/24 13:23:58 [ERROR] [rkecluster] fleet-default/vsphere-testnotable-mustang: error getting CAPI cluster no matching controller owner ref
2023/08/24 13:23:58 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang': handler rke-cluster: no matching controller owner ref, requeuing
2023/08/24 13:23:58 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang': handler planner: no matching controller owner ref, requeuing
2023/08/24 13:23:58 [ERROR] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: error encountered during plan processing was no matching controller owner ref
2023/08/24 13:23:58 [INFO] [mgmt-cluster-rbac-delete] Updating cluster c-m-ppkh4pgf
2023/08/24 13:23:58 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang': handler planner: no matching controller owner ref, requeuing
2023/08/24 13:23:58 [ERROR] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: error encountered during plan processing was no matching controller owner ref
2023/08/24 13:23:58 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang': handler planner: no matching controller owner ref, requeuing
2023/08/24 13:23:58 [ERROR] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: error encountered during plan processing was no matching controller owner ref
2023/08/24 13:23:58 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang': handler planner: no matching controller owner ref, requeuing
2023/08/24 13:23:58 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for infrastructure ready
2023/08/24 13:23:58 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for infrastructure ready
2023/08/24 13:23:58 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for infrastructure ready
2023/08/24 13:23:58 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for infrastructure ready
2023/08/24 13:23:59 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/24 13:23:59 [ERROR] [rkebootstrap] fleet-default/vsphere-testnotable-mustang-bootstrap-template-r8kvb: error getting machine by owner reference no matching controller owner ref
2023/08/24 13:23:59 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang-bootstrap-template-r8kvb': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/08/24 13:23:59 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/24 13:23:59 [ERROR] [rkebootstrap] fleet-default/vsphere-testnotable-mustang-bootstrap-template-r8kvb: error getting machine by owner reference no matching controller owner ref
2023/08/24 13:23:59 [ERROR] [machineprovision] fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv: error getting machine by owner reference: no matching controller owner ref
2023/08/24 13:23:59 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv': handler machine-provision: no matching controller owner ref, requeuing
2023/08/24 13:23:59 [ERROR] [machineprovision] fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv: error getting machine by owner reference: no matching controller owner ref
2023/08/24 13:23:59 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv': handler machine-provision: no matching controller owner ref, requeuing
2023/08/24 13:23:59 [ERROR] [machineprovision] fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv: error getting machine by owner reference: no matching controller owner ref
2023/08/24 13:23:59 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv': handler machine-provision: no matching controller owner ref, requeuing
2023/08/24 13:23:59 [ERROR] [rkebootstrap] fleet-default/vsphere-testnotable-mustang-bootstrap-template-r8kvb: error getting machine by owner reference no matching controller owner ref
2023/08/24 13:23:59 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang-bootstrap-template-r8kvb': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/08/24 13:23:59 [ERROR] [machineprovision] fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv: error getting machine by owner reference: no matching controller owner ref
2023/08/24 13:23:59 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv': handler machine-provision: no matching controller owner ref, requeuing
2023/08/24 13:23:59 [ERROR] [rkebootstrap] fleet-default/vsphere-testnotable-mustang-bootstrap-template-r8kvb: error getting machine by owner reference no matching controller owner ref
2023/08/24 13:23:59 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang-bootstrap-template-r8kvb': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/08/24 13:23:59 [ERROR] [rkebootstrap] fleet-default/vsphere-testnotable-mustang-bootstrap-template-r8kvb: error getting machine by owner reference no matching controller owner ref
2023/08/24 13:23:59 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang-bootstrap-template-r8kvb': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/08/24 13:23:59 [ERROR] [machineprovision] fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv: error getting machine by owner reference: no matching controller owner ref
2023/08/24 13:23:59 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv': handler machine-provision: no matching controller owner ref, requeuing
2023/08/24 13:23:59 [ERROR] [rkebootstrap] fleet-default/vsphere-testnotable-mustang-bootstrap-template-r8kvb: error getting machine by owner reference no matching controller owner ref
2023/08/24 13:23:59 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang-bootstrap-template-r8kvb': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/08/24 13:23:59 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/24 13:23:59 [ERROR] [machineprovision] fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv: error getting machine by owner reference: no matching controller owner ref
2023/08/24 13:23:59 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv': handler machine-provision: no matching controller owner ref, requeuing
2023/08/24 13:23:59 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/24 13:23:59 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/24 13:23:59 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/24 13:24:00 [ERROR] [machineprovision] fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv: error getting machine by owner reference: no matching controller owner ref
2023/08/24 13:24:00 [ERROR] error syncing 'fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv': handler machine-provision: no matching controller owner ref, requeuing
2023/08/24 13:24:00 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/24 13:24:00 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/24 13:24:00 [INFO] EnsureSecretForServiceAccount: waiting for secret [vsphere-testnotable-mustang-bootstrap-template-r8kvb-mach-vfpm7] to be populated with token
2023/08/24 13:24:01 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/24 13:24:01 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/24 13:24:01 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/24 13:24:01 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/24 13:24:02 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/24 13:24:02 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/24 13:24:02 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/24 13:24:03 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/24 13:24:03 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/24 13:24:05 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/24 13:24:10 [INFO] [machineprovision] fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv: reconciling machine job
2023/08/24 13:24:10 [INFO] [machineprovision] fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv: reconciling machine job
2023/08/24 13:24:10 [INFO] [machineprovision] fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv: reconciling machine job
2023/08/24 13:24:10 [INFO] [machineprovision] fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv: reconciling machine job
2023/08/24 13:24:10 [INFO] [machineprovision] fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv: reconciling machine job
2023/08/24 13:24:10 [INFO] [machineprovision] fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv: reconciling machine job
2023/08/24 13:24:10 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/24 13:24:14 [INFO] [machineprovision] fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv: reconciling machine job
2023/08/24 13:27:28 [INFO] [machineprovision] fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv: reconciling machine job
2023/08/24 13:27:30 [INFO] [machineprovision] fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv: reconciling machine job
2023/08/24 13:27:30 [INFO] [machineprovision] fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv: reconciling machine job
2023/08/24 13:27:30 [INFO] [machineprovision] fleet-default/vsphere-testnotable-mustang-control-fc49384c-8r2nv: reconciling machine job
2023/08/24 13:27:31 [INFO] [planner] rkecluster fleet-default/vsphere-testnotable-mustang: waiting: waiting for at least one control plane, etcd, and worker node to be registered
fleet-agent:
time="2023-08-24T13:25:06Z" level=error msg="Failed to register agent: registration failed: cannot create clusterregistration on management cluster for cluster id 'kbxw6ffjd4g9qdvmpnklq6wmchf4xvmlr68qs6f48g8m4jpqcqv79v': Post \"https://172.14.12.3.sslip.io/apis/fleet.cattle.io/v1alpha1/namespaces/fleet-local/clusterregistrations\": tls: failed to verify certificate: x509: certificate is valid for 3c819d17171583e2e5d774dcf481398e.4668070069aedc2d0e0bcd99bc7425f0.traefik.default, not 172.14.12.3.sslip.io"
time="2023-08-24T13:26:06Z" level=warning msg="Cannot find fleet-agent secret, running registration"
time="2023-08-24T13:26:08Z" level=error msg="Failed to register agent: registration failed: cannot create clusterregistration on management cluster for cluster id 'kbxw6ffjd4g9qdvmpnklq6wmchf4xvmlr68qs6f48g8m4jpqcqv79v': Post \"https://172.14.12.3.sslip.io/apis/fleet.cattle.io/v1alpha1/namespaces/fleet-local/clusterregistrations\": tls: failed to verify certificate: x509: certificate is valid for 3c819d17171583e2e5d774dcf481398e.4668070069aedc2d0e0bcd99bc7425f0.traefik.default, not 172.14.12.3.sslip.io"
time="2023-08-24T13:27:08Z" level=warning msg="Cannot find fleet-agent secret, running registration"
time="2023-08-24T13:27:08Z" level=error msg="Failed to register agent: registration failed: cannot create clusterregistration on management cluster for cluster id 'kbxw6ffjd4g9qdvmpnklq6wmchf4xvmlr68qs6f48g8m4jpqcqv79v': Post \"https://172.14.12.3.sslip.io/apis/fleet.cattle.io/v1alpha1/namespaces/fleet-local/clusterregistrations\": tls: failed to verify certificate: x509: certificate is valid for 3c819d17171583e2e5d774dcf481398e.4668070069aedc2d0e0bcd99bc7425f0.traefik.default, not 172.14.12.3.sslip.io"
time="2023-08-24T13:28:08Z" level=warning msg="Cannot find fleet-agent secret, running registration"
time="2023-08-24T13:28:08Z" level=error msg="Failed to register agent: registration failed: cannot create clusterregistration on management cluster for cluster id 'kbxw6ffjd4g9qdvmpnklq6wmchf4xvmlr68qs6f48g8m4jpqcqv79v': Post \"https://172.14.12.3.sslip.io/apis/fleet.cattle.io/v1alpha1/namespaces/fleet-local/clusterregistrations\": tls: failed to verify certificate: x509: certificate is valid for 3c819d17171583e2e5d774dcf481398e.4668070069aedc2d0e0bcd99bc7425f0.traefik.default, not 172.14.12.3.sslip.io"
time="2023-08-24T13:29:08Z" level=warning msg="Cannot find fleet-agent secret, running registration"
time="2023-08-24T13:29:08Z" level=error msg="Failed to register agent: registration failed: cannot create clusterregistration on management cluster for cluster id 'kbxw6ffjd4g9qdvmpnklq6wmchf4xvmlr68qs6f48g8m4jpqcqv79v': Post \"https://172.14.12.3.sslip.io/apis/fleet.cattle.io/v1alpha1/namespaces/fleet-local/clusterregistrations\": tls: failed to verify certificate: x509: certificate is valid for 3c819d17171583e2e5d774dcf481398e.4668070069aedc2d0e0bcd99bc7425f0.traefik.default, not 172.14.12.3.sslip.io"
time="2023-08-24T13:30:08Z" level=warning msg="Cannot find fleet-agent secret, running registration"
time="2023-08-24T13:30:08Z" level=error msg="Failed
are these red herrings? helps would be appreciated!