CSI Looking for Deleted PV

Longhorn
1

After a cleanup of old volumes, my logs are getting spammed from csi-attacher looking for deleted volumes. I actually think that the underlying replica directories were manually deleted and that is the root cause of the errors. The volumes are all removed now, is there a way I can tell the attacher to stop looking for them?

Everything is working fine, just getting tons of errors in the logs.

Rancher 2.2.4
Longhorn 0.6.2

Here’s a chunk of the log

I1205 14:20:15.213970       1 request.go:530] Throttling request took 1.958015711s, request: PUT:https://10.43.0.1:443/apis/storage.k8s.io/v1beta1/volumeattachments/csi-64541f53aa9e14480ed0edf2b5e50a7133712feea28c8723cd6a1e730868eb47 
I1205 14:20:15.231056       1 csi_handler.go:333] Saved attach error to "csi-64541f53aa9e14480ed0edf2b5e50a7133712feea28c8723cd6a1e730868eb47" 
I1205 14:20:15.231113       1 csi_handler.go:95] Error processing "csi-64541f53aa9e14480ed0edf2b5e50a7133712feea28c8723cd6a1e730868eb47": failed to attach: persistentvolume "pvc-b812b40a-f100-11e9-8c4e-b6eae319b517" not found 
I1205 14:20:15.231253       1 controller.go:167] Started VA processing "csi-41ea402ea13b833719185a2947408d23ae81938feaa472e6a7b21c84697dd185" 
I1205 14:20:15.231282       1 csi_handler.go:85] CSIHandler: processing VA "csi-41ea402ea13b833719185a2947408d23ae81938feaa472e6a7b21c84697dd185" 
I1205 14:20:15.231300       1 csi_handler.go:112] Attaching "csi-41ea402ea13b833719185a2947408d23ae81938feaa472e6a7b21c84697dd185" 
I1205 14:20:15.231425       1 csi_handler.go:217] Starting attach operation for "csi-41ea402ea13b833719185a2947408d23ae81938feaa472e6a7b21c84697dd185" 
I1205 14:20:15.231453       1 csi_handler.go:323] Saving attach error to "csi-41ea402ea13b833719185a2947408d23ae81938feaa472e6a7b21c84697dd185" 
I1205 14:20:15.414024       1 request.go:530] Throttling request took 1.985001353s, request: PUT:https://10.43.0.1:443/apis/storage.k8s.io/v1beta1/volumeattachments/csi-651d2c6bd8ba4109c979cefc288cf66e3093849e48ec4cbf6d44427f6ecd3104 
I1205 14:20:15.448153       1 csi_handler.go:333] Saved attach error to "csi-651d2c6bd8ba4109c979cefc288cf66e3093849e48ec4cbf6d44427f6ecd3104" 
I1205 14:20:15.448221       1 csi_handler.go:95] Error processing "csi-651d2c6bd8ba4109c979cefc288cf66e3093849e48ec4cbf6d44427f6ecd3104": failed to attach: persistentvolume "pvc-b7fcd1dc-f100-11e9-8c4e-b6eae319b517" not found 
I1205 14:20:15.448401       1 controller.go:167] Started VA processing "csi-3bea66e1fe2bbf2bdfa15ddb0e386cc4b46ebb2652dbb213d87a73dc79eeb5cc" 
I1205 14:20:15.448433       1 csi_handler.go:85] CSIHandler: processing VA "csi-3bea66e1fe2bbf2bdfa15ddb0e386cc4b46ebb2652dbb213d87a73dc79eeb5cc" 
I1205 14:20:15.448454       1 csi_handler.go:112] Attaching "csi-3bea66e1fe2bbf2bdfa15ddb0e386cc4b46ebb2652dbb213d87a73dc79eeb5cc" 
I1205 14:20:15.448477       1 csi_handler.go:217] Starting attach operation for "csi-3bea66e1fe2bbf2bdfa15ddb0e386cc4b46ebb2652dbb213d87a73dc79eeb5cc" 
I1205 14:20:15.448508       1 csi_handler.go:323] Saving attach error to "csi-3bea66e1fe2bbf2bdfa15ddb0e386cc4b46ebb2652dbb213d87a73dc79eeb5cc" 
I1205 14:20:15.614083       1 request.go:530] Throttling request took 1.898110577s, request: PUT:https://10.43.0.1:443/apis/storage.k8s.io/v1beta1/volumeattachments/csi-07ed8e27c4416049959f13850439ca98ea5ed29f02687b6f722607ab8046f390 
I1205 14:20:15.655881       1 csi_handler.go:333] Saved attach error to "csi-07ed8e27c4416049959f13850439ca98ea5ed29f02687b6f722607ab8046f390" 
I1205 14:20:15.655935       1 csi_handler.go:95] Error processing "csi-07ed8e27c4416049959f13850439ca98ea5ed29f02687b6f722607ab8046f390": failed to attach: persistentvolume "pvc-b812b40a-f100-11e9-8c4e-b6eae319b517" not found 
I1205 14:20:15.656176       1 controller.go:167] Started VA processing "csi-3ace65c7dc15138a3121d46f5ffb36549c2e34395b761d5020a0cd81acd6a42c" 
I1205 14:20:15.656229       1 csi_handler.go:85] CSIHandler: processing VA "csi-3ace65c7dc15138a3121d46f5ffb36549c2e34395b761d5020a0cd81acd6a42c" 
I1205 14:20:15.656253       1 csi_handler.go:112] Attaching "csi-3ace65c7dc15138a3121d46f5ffb36549c2e34395b761d5020a0cd81acd6a42c" 
I1205 14:20:15.656283       1 csi_handler.go:217] Starting attach operation for "csi-3ace65c7dc15138a3121d46f5ffb36549c2e34395b761d5020a0cd81acd6a42c" 
I1205 14:20:15.656317       1 csi_handler.go:323] Saving attach error to "csi-3ace65c7dc15138a3121d46f5ffb36549c2e34395b761d5020a0cd81acd6a42c" 
I1205 14:20:15.799240       1 leaderelection.go:227] successfully renewed lease longhorn-system/external-attacher-leader-io-rancher-longhorn 
I1205 14:20:15.813788       1 request.go:530] Throttling request took 1.977599162s, request: PUT:https://10.43.0.1:443/apis/storage.k8s.io/v1beta1/volumeattachments/csi-0f89d78a325031d8eef9b6a0f7f73d68ccbc976378a60e2268374586c441d1e7 
I1205 14:20:15.828260       1 csi_handler.go:333] Saved attach error to "csi-0f89d78a325031d8eef9b6a0f7f73d68ccbc976378a60e2268374586c441d1e7" 
I1205 14:20:15.828315       1 csi_handler.go:95] Error processing "csi-0f89d78a325031d8eef9b6a0f7f73d68ccbc976378a60e2268374586c441d1e7": failed to attach: persistentvolume "pvc-ae8fef26-1639-11ea-89fb-42e66f259a6c" not found 
I1205 14:20:15.828494       1 controller.go:167] Started VA processing "csi-f3de0b4e96e05bbacc712008658939163ec927f968bdb09c2b009f4a6d3fcc9c" 
I1205 14:20:15.828530       1 csi_handler.go:85] CSIHandler: processing VA "csi-f3de0b4e96e05bbacc712008658939163ec927f968bdb09c2b009f4a6d3fcc9c" 
I1205 14:20:15.828551       1 csi_handler.go:112] Attaching "csi-f3de0b4e96e05bbacc712008658939163ec927f968bdb09c2b009f4a6d3fcc9c" 
I1205 14:20:15.828567       1 csi_handler.go:217] Starting attach operation for "csi-f3de0b4e96e05bbacc712008658939163ec927f968bdb09c2b009f4a6d3fcc9c" 
I1205 14:20:15.828593       1 csi_handler.go:323] Saving attach error to "csi-f3de0b4e96e05bbacc712008658939163ec927f968bdb09c2b009f4a6d3fcc9c" 
I1205 14:20:16.014070       1 request.go:530] Throttling request took 1.985694197s, request: PUT:https://10.43.0.1:443/apis/storage.k8s.io/v1beta1/volumeattachments/csi-5e26d620e246dc3047536117e537126800388e34212cb6deebe3eab85f45b5b0 
I1205 14:20:16.028715       1 csi_handler.go:333] Saved attach error to "csi-5e26d620e246dc3047536117e537126800388e34212cb6deebe3eab85f45b5b0" 
I1205 14:20:16.028791       1 csi_handler.go:95] Error processing "csi-5e26d620e246dc3047536117e537126800388e34212cb6deebe3eab85f45b5b0": failed to attach: persistentvolume "pvc-54508908-1626-11ea-972e-b6eae319b517" not found 
I1205 14:20:16.029147       1 controller.go:167] Started VA processing "csi-ad1b07f954b54de2e0921c083485ca7841ff2e13c6e3daae45c5a170edff48db" 
I1205 14:20:16.029241       1 csi_handler.go:85] CSIHandler: processing VA "csi-ad1b07f954b54de2e0921c083485ca7841ff2e13c6e3daae45c5a170edff48db" 
I1205 14:20:16.029260       1 csi_handler.go:112] Attaching "csi-ad1b07f954b54de2e0921c083485ca7841ff2e13c6e3daae45c5a170edff48db" 
I1205 14:20:16.029277       1 csi_handler.go:217] Starting attach operation for "csi-ad1b07f954b54de2e0921c083485ca7841ff2e13c6e3daae45c5a170edff48db" 
I1205 14:20:16.029310       1 csi_handler.go:323] Saving attach error to "csi-ad1b07f954b54de2e0921c083485ca7841ff2e13c6e3daae45c5a170edff48db" 
I1205 14:20:16.213912       1 request.go:530] Throttling request took 1.98206254s, request: PUT:https://10.43.0.1:443/apis/storage.k8s.io/v1beta1/volumeattachments/csi-58b81730e769255a846bf72848949b1feb4e44ee0068b058d50bc684ef123b55 
I1205 14:20:16.229669       1 csi_handler.go:333] Saved attach error to "csi-58b81730e769255a846bf72848949b1feb4e44ee0068b058d50bc684ef123b55" 
I1205 14:20:16.229730       1 csi_handler.go:95] Error processing "csi-58b81730e769255a846bf72848949b1feb4e44ee0068b058d50bc684ef123b55": failed to attach: persistentvolume "pvc-b7fcd1dc-f100-11e9-8c4e-b6eae319b517" not found 
I1205 14:20:16.229953       1 controller.go:167] Started VA processing "csi-64541f53aa9e14480ed0edf2b5e50a7133712feea28c8723cd6a1e730868eb47" 
I1205 14:20:16.230018       1 csi_handler.go:85] CSIHandler: processing VA "csi-64541f53aa9e14480ed0edf2b5e50a7133712feea28c8723cd6a1e730868eb47" 
I1205 14:20:16.230037       1 csi_handler.go:112] Attaching "csi-64541f53aa9e14480ed0edf2b5e50a7133712feea28c8723cd6a1e730868eb47" 
I1205 14:20:16.230062       1 csi_handler.go:217] Starting attach operation for "csi-64541f53aa9e14480ed0edf2b5e50a7133712feea28c8723cd6a1e730868eb47" 
I1205 14:20:16.230094       1 csi_handler.go:323] Saving attach error to "csi-64541f53aa9e14480ed0edf2b5e50a7133712feea28c8723cd6a1e730868eb47" 
I1205 14:20:16.414021       1 request.go:530] Throttling request took 1.97928645s, request: PUT:https://10.43.0.1:443/apis/storage.k8s.io/v1beta1/volumeattachments/csi-8222a0b9c86e34d1994686d8e2223ff12b15de3e96836c3328e970bc54ed182b 
I1205 14:20:16.430213       1 csi_handler.go:333] Saved attach error to "csi-8222a0b9c86e34d1994686d8e2223ff12b15de3e96836c3328e970bc54ed182b" 
I1205 14:20:16.430263       1 csi_handler.go:95] Error processing "csi-8222a0b9c86e34d1994686d8e2223ff12b15de3e96836c3328e970bc54ed182b": failed to attach: persistentvolume "pvc-0a965f63-1632-11ea-89fb-42e66f259a6c" not found 
I1205 14:20:16.430446       1 controller.go:167] Started VA processing "csi-651d2c6bd8ba4109c979cefc288cf66e3093849e48ec4cbf6d44427f6ecd3104" 
I1205 14:20:16.430493       1 csi_handler.go:85] CSIHandler: processing VA "csi-651d2c6bd8ba4109c979cefc288cf66e3093849e48ec4cbf6d44427f6ecd3104" 
I1205 14:20:16.430517       1 csi_handler.go:112] Attaching "csi-651d2c6bd8ba4109c979cefc288cf66e3093849e48ec4cbf6d44427f6ecd3104" 
I1205 14:20:16.430538       1 csi_handler.go:217] Starting attach operation for "csi-651d2c6bd8ba4109c979cefc288cf66e3093849e48ec4cbf6d44427f6ecd3104" 
I1205 14:20:16.430568       1 csi_handler.go:323] Saving attach error to "csi-651d2c6bd8ba4109c979cefc288cf66e3093849e48ec4cbf6d44427f6ecd3104" 
I1205 14:20:16.613921       1 request.go:530] Throttling request took 1.985661667s, request: PUT:https://10.43.0.1:443/apis/storage.k8s.io/v1beta1/volumeattachments/csi-3412b398032fd9d311736957944f925066fa92c7b1c4ab628eaae77f7c33ed6a 
I1205 14:20:16.631433       1 csi_handler.go:333] Saved attach error to "csi-3412b398032fd9d311736957944f925066fa92c7b1c4ab628eaae77f7c33ed6a" 
I1205 14:20:16.631572       1 csi_handler.go:95] Error processing "csi-3412b398032fd9d311736957944f925066fa92c7b1c4ab628eaae77f7c33ed6a": failed to attach: persistentvolume "pvc-b812b40a-f100-11e9-8c4e-b6eae319b517" not found 
I1205 14:20:16.631784       1 controller.go:167] Started VA processing "csi-07ed8e27c4416049959f13850439ca98ea5ed29f02687b6f722607ab8046f390" 
I1205 14:20:16.631827       1 csi_handler.go:85] CSIHandler: processing VA "csi-07ed8e27c4416049959f13850439ca98ea5ed29f02687b6f722607ab8046f390" 
I1205 14:20:16.631845       1 csi_handler.go:112] Attaching "csi-07ed8e27c4416049959f13850439ca98ea5ed29f02687b6f722607ab8046f390" 
I1205 14:20:16.631861       1 csi_handler.go:217] Starting attach operation for "csi-07ed8e27c4416049959f13850439ca98ea5ed29f02687b6f722607ab8046f390" 
I1205 14:20:16.631887       1 csi_handler.go:323] Saving attach error to "csi-07ed8e27c4416049959f13850439ca98ea5ed29f02687b6f722607ab8046f390"
2

  1. The csi-attacher is trying to attach non-existing Longhorn volumes. These attach requests are actually from kubelet: If you want to launch/scale up workloads with Longhorn volumes, kubelet will send attach requests to csi-attacher to attach these related Longhorn volumes for the workloads. Hence it is better not to delete the volumes provisioned by kubelet (with the StorageClass) in Longhorn UI. And you can check if you somehow clean up the volumes but are still using the related PVC.

  2. We already upgraded the csi-attacher version in Longhorn v0.7.0. You can check if this issue can be triggered in the new version. Or you can describe the reproduce steps then we can test it.