Namespace "Terminating" and undeletable secrets

Rancher
1

In our Rancher v2.3.2 we try to delete a namespace that is stuck in “Terminating”.
Deletion fails because there is “some content” left in the namespace: secrets. But even these can’t be deleted.
Tried all tricks available, for example editing the namespace and remove kubernetes from the finalizers, but no.
Also this nice script https://github.com/thyarles/knsk got stuck…

This is what happens when tried from the CLI with kubectl aka k:

$ k get namespaces donut -n donut
NAME   STATUS        AGE
donut   Terminating   408d

$ k delete namespace donut --force --grace-period=0
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
Error from server (Conflict): Operation cannot be fulfilled on namespaces "donut": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.

$ k get secrets -n donut
NAME        TYPE                             DATA   AGE
ndc-727vd   kubernetes.io/dockerconfigjson   1      408d
ns-rwp7x    Opaque                           1      408d

$ k delete secret ns-rwp7x -n donut
secret "ns-rwp7x" deleted
error: An error occurred while waiting for the object to be deleted: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 5; INTERNAL_ERROR") has prevented the request from succeeding^C

$ k delete secret ndc-727vd -n donut --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
secret "ndc-727vd" force deleted
error: An error occurred while waiting for the object to be deleted: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 5; INTERNAL_ERROR") has prevented the request from succeeding
error: An error occurred while waiting for the object to be deleted: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 9; INTERNAL_ERROR") has prevented the request from succeeding
error: An error occurred while waiting for the object to be deleted: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 13; INTERNAL_ERROR") has prevented the request from succeeding
error: An error occurred while waiting for the object to be deleted: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 17; INTERNAL_ERROR") has prevented the request from succeeding
error: An error occurred while waiting for the object to be deleted: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 21; INTERNAL_ERROR") has prevented the request from succeeding
error: An error occurred while waiting for the object to be deleted: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 25; INTERNAL_ERROR") has prevented the request from succeeding
error: An error occurred while waiting for the object to be deleted: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 29; INTERNAL_ERROR") has prevented the request from succeeding^C

The stream errors are a concern, but is there another way to get this done?

2

Just some ideas:
Write some feedback in https://github.com/thyarles/knsk/issues, the author seems interested. By the way, were does it stuck?
I am sure you already read https://github.com/kubernetes/kubernetes/issues/60807 (where the script is announced). Maybe you can ask there anyway.
It is also listed as k3s issue: https://github.com/rancher/k3s/issues/1432

3

Thanks, Simon.
What I do wonder now, if this is a Kubernetes specific issue or more related to Rancher v2 / k3s…