Problems removing (legacy) cattle-logging

Hi there,

We have a Rancher v2.6.6 Installation, it’s actually very old and went through a lot of updates.
On this Rancher we still had the old/legacy rancher-logging:0.2.4 installed in namespace cattle-logging.
We now wanted to remove that old installation, but it turns out to be more of a problem then expected:

First we tried to uninstall it in the new “Apps” section, where it actually appered.
But hitting delete, it went into an error:

helm uninstall --namespace=cattle-logging rancher-logging
Error: uninstall: Release not loaded: rancher-logging: release: not found

So we deleted its components manually:

kubectl -n cattle-logging delete service rancher-logging-fluentd
kubectl -n cattle-logging delete daemonset rancher-logging-fluentd-linux
kubectl -n cattle-logging delete daemonset rancher-logging-log-aggregator-linux

After that removing the namespace from the Rancher GUI.
This worked, besides the namespace which is still in state “terminating”.

Also, there are many errors in the rancher logs like these:

2022/08/31 11:35:40 [ERROR] error syncing 'fleet-system': handler namespace-logging-configsysncer: get secret cattle-logging:rancher-logging-fluentd-ssl failed: secrets "rancher-logging-fluentd-ssl" not found, requeuing
2022/08/31 11:35:40 [ERROR] error syncing 'cluster-fleet-default-c-tkrjk-71df7f2f91c2': handler namespace-logging-configsysncer: get secret cattle-logging:rancher-logging-fluentd-ssl failed: secrets "rancher-logging-fluentd-ssl" not found, requeuing
2022/08/31 11:35:40 [ERROR] error syncing 'cluster-fleet-default-c-hxdh5-2015335cb204': handler namespace-logging-configsysncer: get secret cattle-logging:rancher-logging-fluentd-ssl failed: secrets "rancher-logging-fluentd-ssl" not found, requeuing
2022/08/31 11:35:40 [ERROR] error syncing 'cluster-fleet-default-c-tmcj9-98eb85c13449': handler namespace-logging-configsysncer: get secret cattle-logging:rancher-logging-fluentd-ssl failed: secrets "rancher-logging-fluentd-ssl" not found, requeuing
2022/08/31 11:35:40 [ERROR] error syncing 'fleet-default': handler namespace-logging-configsysncer: get secret cattle-logging:rancher-logging-fluentd-ssl failed: secrets "rancher-logging-fluentd-ssl" not found, requeuing
2022/08/31 11:35:42 [ERROR] error syncing 'rancher-master3': handler node-syncer: timeout checking app status, app deployed namespace: cattle-logging, labels: map[app:fluentd], requeuing
2022/08/31 11:35:58 [ERROR] error syncing 'rancher-master1': handler node-syncer: timeout checking app status, app deployed namespace: cattle-logging, labels: map[app:fluentd], requeuing
2022/08/31 11:35:58 [ERROR] error syncing 'rancher-master2': handler node-syncer: timeout checking app status, app deployed namespace: cattle-logging, labels: map[app:fluentd], requeuing
...

How can we get rid of these errors?
Maybe: How can we get rid of “namespace-logging-configsysncer” or any other leftovers that may be there from legacy logging?

for i in $(kubectl get crd --no-headers | awk '{print $1}'); do echo " >>> $i"; kubectl get $i -o yaml | grep 'rancher-logging'; done

Using this command in the namespace of the cattle-logging project (“p-xyz123” namespace), I actually found the entry “rancher-logging” in CRD “apps.project.cattle.io”.
Deleting it will succeed, but something then recreates it.

Update:
We got the namespace to be removed by an API call to https://…/k8s/clusters/local/api/v1/namespaces/cattle-logging/finalize, removing the finalizer.
But actually the ns was recreated magically, afterwards …

Have you figured this one out? We’re having the same errors in all our clusters now.