Rancher 2.6.9 - phantom resource after cluster deletion

Rancher 2.x
1

Quick background:

Rancher 2.6.9 in a 3-node RKE cluster. Provisioned an EKS cluster in Amazon AWS, but due to network configuration issues it never succeeded. I added the cluster to Rancher (using the cloud credentials & selected the cluster from a list in the UI) and it never fully finished adding the cluster (cattle-cluster-agent was never able to talk back to Rancher).

No way to delete the cluster inside the UI because it wasn’t fully provisioned yet. I found the clusters.v3.management.cattle.io object which had a reference to the new cluster, and I deleted it. That finally removed the cluster from the UI.

However, since then, I occasionally find a couple entries in the list-

(blue)Removing - Resource scheduled for deletion - Kubernetes Version Unknown
(blue)Reconciling - Kubernetes Version Unknown

When I hit CTRL+R to reload the rancher UI, these entries disappear. But they always come back.

Is there a stuck reference to this old cluster somewhere in the rancher-hosting RKE environment I should look for & delete?

3

rancher269-stale-reference-resource-scheduled-for-deletion
rancher269-stale-reference-resource-scheduled-for-deletion2804×616 47.1 KB

4

Update- On a different rancher 2.6.9 instance, I deleted a cluster this morning (which had been running for 2 years and no longer needed) and just found hours later another one of these phantom “Resource scheduled for deletion” items show up in the list. Refreshing the dashboard makes it disappear.

5

Using the Cluster Management view, I caught this one in the act, with the old cluster context string in place of the name of the cluster.

I clicked on it-

https://[rancher-hostname-redacted]/dashboard/c/_/manager/provisioning.cattle.io.cluster/fleet-default/c-ppj9h

Presumably then, this “phantom” cluster is somehow in the Fleet configuration? Anywhere in particular I should look for orphaned fleet entries for old deleted clusters?