Unable to provision volumes in fresh install of Longhorn (v0.7)

I am running Rancher (v.2.3.3) to manage a bare-metal (running RancherOS 1.5.5) Kubernetes cluster. This is all running in my home-lab.I used the App Launch feature of Rancher to install the Longhorn Helm Chart. That all seem to have worked fine. I am able to to go to the Longhorn dashboard and it reports that everything is Ok. It shows me that I have around 235 GB of available storage. I also verified that the longhorn storage class was created in my cluster.

I am having no luck deploying the WordPress App (just to test the volume creation) because the PVC that the WordPress creates is stuck in Pending status. If I check the Events on the PVC I can see the following:

Warning ProvisioningFailed failed to provision volume with StorageClass “longhorn”: rpc error: code = DeadlineExceeded desc = context deadline exceeded 3 minutes ago
Normal Provisioning External provisioner is provisioning volume for claim “wordpress/data-wordpress-mariadb-0” 3 minutes ago
Normal ExternalProvisioning waiting for a volume to be created, either by external provisioner “driver.longhorn.io” or manually created by system administrator 3 minutes ago

I am also not able to create and attach a Volume in the Longhorn Dashboard. I have tried both Block Device and iSCSI front end. I am able to create the Volume and I see that it creates 3 replicas for me but it ends up detached. When I try to attach it it goes back to being detached. This is what I see in the Longhorn Dashboard UI:

Engine of volume test dead unexpectedly, reattach the volume

Before I installed Longhorn I went through the instructions on how to add open-iscsi to all my nodes since I am running on RancherOS.

Appreciate any help or hints on where I can start troubleshooting this.


You mean you had followed this doc to config the nodes before launching Longhorn?

If YES, you can check if the instance-manager-e-xxxx pod and the instance-manager-r-xxxx pod for each node work fine when you attach/detach a Longhorn volume via Longhorn UI. (All longhorn components are in the namespace longhorn-system.)

If you cannot find anything, you can click Generate Support Bundle link at the bottom of the UI and send it to longhorn-support-bundle@rancher.com.