Hi there,
We have a Rancher v2.6.6 Installation, it’s actually very old and went through a lot of updates.
On this Rancher we still had the old/legacy rancher-logging:0.2.4 installed in namespace cattle-logging.
We now wanted to remove that old installation, but it turns out to be more of a problem then expected:
First we tried to uninstall it in the new “Apps” section, where it actually appered.
But hitting delete, it went into an error:
helm uninstall --namespace=cattle-logging rancher-logging
Error: uninstall: Release not loaded: rancher-logging: release: not found
So we deleted its components manually:
kubectl -n cattle-logging delete service rancher-logging-fluentd
kubectl -n cattle-logging delete daemonset rancher-logging-fluentd-linux
kubectl -n cattle-logging delete daemonset rancher-logging-log-aggregator-linux
After that removing the namespace from the Rancher GUI.
This worked, besides the namespace which is still in state “terminating”.
Also, there are many errors in the rancher logs like these:
2022/08/31 11:35:40 [ERROR] error syncing 'fleet-system': handler namespace-logging-configsysncer: get secret cattle-logging:rancher-logging-fluentd-ssl failed: secrets "rancher-logging-fluentd-ssl" not found, requeuing
2022/08/31 11:35:40 [ERROR] error syncing 'cluster-fleet-default-c-tkrjk-71df7f2f91c2': handler namespace-logging-configsysncer: get secret cattle-logging:rancher-logging-fluentd-ssl failed: secrets "rancher-logging-fluentd-ssl" not found, requeuing
2022/08/31 11:35:40 [ERROR] error syncing 'cluster-fleet-default-c-hxdh5-2015335cb204': handler namespace-logging-configsysncer: get secret cattle-logging:rancher-logging-fluentd-ssl failed: secrets "rancher-logging-fluentd-ssl" not found, requeuing
2022/08/31 11:35:40 [ERROR] error syncing 'cluster-fleet-default-c-tmcj9-98eb85c13449': handler namespace-logging-configsysncer: get secret cattle-logging:rancher-logging-fluentd-ssl failed: secrets "rancher-logging-fluentd-ssl" not found, requeuing
2022/08/31 11:35:40 [ERROR] error syncing 'fleet-default': handler namespace-logging-configsysncer: get secret cattle-logging:rancher-logging-fluentd-ssl failed: secrets "rancher-logging-fluentd-ssl" not found, requeuing
2022/08/31 11:35:42 [ERROR] error syncing 'rancher-master3': handler node-syncer: timeout checking app status, app deployed namespace: cattle-logging, labels: map[app:fluentd], requeuing
2022/08/31 11:35:58 [ERROR] error syncing 'rancher-master1': handler node-syncer: timeout checking app status, app deployed namespace: cattle-logging, labels: map[app:fluentd], requeuing
2022/08/31 11:35:58 [ERROR] error syncing 'rancher-master2': handler node-syncer: timeout checking app status, app deployed namespace: cattle-logging, labels: map[app:fluentd], requeuing
...
How can we get rid of these errors?
Maybe: How can we get rid of “namespace-logging-configsysncer” or any other leftovers that may be there from legacy logging?
for i in $(kubectl get crd --no-headers | awk '{print $1}'); do echo " >>> $i"; kubectl get $i -o yaml | grep 'rancher-logging'; done
Using this command in the namespace of the cattle-logging project (“p-xyz123” namespace), I actually found the entry “rancher-logging” in CRD “apps.project.cattle.io”.
Deleting it will succeed, but something then recreates it.