Suggestion about how to force delete failed longhorn node

Longhorn
1

Related question: [Question] Delete nodes in v1.10 · Issue #2267 · longhorn/longhorn · GitHub

Longhorn v1.1.0 node will be removed automatically when the related Kubernetes objects are deleted from the cluster. Hence there are 2 options to delete the node from Longhorn:

  1. Delete the related Kubernetes node objects if they still exist in the cluster. You can check the nodes first and see if it’s safe to delete them: kubectl get node , kubectl delete node <node name>
  2. Or directly delete the Longhorn node objects: kubectl -n longhorn-system delete nodes.longhorn.io <node name>

Hi, thanks, my issue was resolved by this.

Background:
Previously, my one node need to be “force deleted” from rancher because this node is failed when I upgrading kubernetes version from rancher UI. The kubernetes node was deleted successfully from the rancher UI.
There are still 2 volumes failed in that node (because I just set it into single replica), and for some reasons, longhorn didn’t delete the node automatically (I assume this is caused by the failed volume?)

What I was doing:
I delete the problematic pod using the failed volume, and then delete the PVC. The volumes were deleted successfully in longhorn UI, but the deleted node was still appear in the longhorn UI, and I couldn’t find any delete node option in longhorn-frontend:80/proxy/#/node page. And also, the Edit Node button was disabled. I just found disable scheduling for the failed node without any effect

I have 3 feedback related to this

  1. If we are allowed to delete nodes.longhorn.io from cli, why is there no option to do that from the UI?
  2. I didn’t see this solution on the docs site, from version 1.0.0 until 1.1.1. I expect it will be appear in here
  3. If my assumption about the node still appear on the UI because of the failed volume is true, is it possible to see this explanation in the UI? I also expect something like nodes.longhorn.io will be automatically deleted because the node was also already deleted from kubectl get node anyway

Thank you!

2

Hi! May I know the version of Longhorn you are using? In Longhorn v1.1.1, we have added the feature of Node removal. If the issue occurs in v1.1.1, can you help us create an issue in GitHub - longhorn/longhorn: Cloud-Native distributed block storage built on and for Kubernetes and provide the support bundle? Thanks!

1 Like
3

Hi
My longhorn version is 1.1.0

Ok I will try to update my longhorn first and will see if this problem appear again or not in the next time.

Thank you for your answer!