NFS Backup - Cannot create mount directory

Longhorn
1

Hello there,

I am trying to configure a backup for my cluster. I use

  • Rancher 2.5.5
  • Longhorn 1.1.0
  • Devices - Raspberry Pi 4
  • QNap NAS

I have configurated NFS4 on NAS, added all devices of the cluster and 10.42.0.0/16 for pods in allowed devices
I tryed to mount the NFS shared folder from each node of the cluster and it works fine

In Longhorn settings, I added
nfs://192.168.1.110:/backup_cluster

Now when I want to create a backup, the following error rise

fail to backup snapshot: failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.1.0/longhorn [--url 10.42.0.88:10000 backup create --dest nfs://192.168.1.110:/backup_cluster --label KubernetesStatus={"pvName":"pvc-229542b4-43ff-4b31-ba16-12e36592c273","pvStatus":"Bound","namespace":"remote-access","pvcName":"rdesktop-config","lastPVCRefAt":"","workloadsStatus":[{"podName":"rdesktop-599b9cbf69-79s9x","podStatus":"Running","workloadName":"rdesktop-599b9cbf69","workloadType":"ReplicaSet"}],"lastPodRefAt":""} dbf84ee4-2649-4fc5-aa97-b9891d1a0db4], output , stderr, time="2021-01-31T21:26:03Z" level=info msg="Backing up dbf84ee4-2649-4fc5-aa97-b9891d1a0db4 on tcp://10.42.7.120:10030, to nfs://192.168.1.110:/backup_cluster" time="2021-01-31T21:26:03Z" level=fatal msg="Error running create backup command: failed to create backup to nfs://192.168.1.110:/backup_cluster for volume pvc-229542b4-43ff-4b31-ba16-12e36592c273: rpc error: code = Unknown desc = Cannot create mount directory /var/lib/longhorn-backupstore-mounts/192_168_1_110/backup_cluster for NFS server" , error exit status 1

And when I access to Backup session, the following error rise

error listing backups: error listing backup volumes: Failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.1.0/longhorn [backup ls --volume-only nfs://192.168.1.110:/backup_cluster], output Cannot create mount directory /var/lib/longhorn-backupstore-mounts/192_168_1_110/backup_cluster for NFS server , stderr, time="2021-01-31T21:28:35Z" level=error msg="Cannot create mount directory /var/lib/longhorn-backupstore-mounts/192_168_1_110/backup_cluster for NFS server" , error exit status 1

I tryed to create the folder /var/lib/longhorn-backupstore-mounts on each node, it doesn’t help

Any idea what I missed?
Thanks

2

Nobody encounter this problem?

3

Longhorn try to mount the NFS directory inside the manager pod when listing backups, and inside the instance manager - r pod when creating backups.

Can you exec into the longhorn manager pods (longhorn-manager-xxxxx) and longhorn instance manager -r pods (instance-manager-r-xxxxxxxx) and try to manually mount the NFS directory?

If you cannot mount it manually then I think that something is wrong with NFS’s setting. Maybe because of permission?

4

Thanks for the answer

I tryed to mount the NFS directory on each instance-manager-r-xxxxxxxx, and it works
I noticed that a new folder was created by Longhorn backupstore and inside there is some data

But I have still the same error in the backup section of the UI

error listing backups: error listing backup volumes: Failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.1.0/longhorn [backup ls --volume-only nfs://192.168.1.110:/backup_cluster], output Cannot create mount directory /var/lib/longhorn-backupstore-mounts/192_168_1_110/backup_cluster for NFS server , stderr, time="2021-02-06T10:07:54Z" level=error msg="Cannot create mount directory /var/lib/longhorn-backupstore-mounts/192_168_1_110/backup_cluster for NFS server" , error exit status 1
5

I have the same issue.

@PhanLe0110 can you give an example how the command to test should look like? I guess it is not a mkdir ?
How can I figure out which of my manager pods/nodes is the one having the problem? Is it the one the volume is attached to or can it be any of the ones a replica resides on?

thanks for help.

6

@ppcm you can try to create the directory inside Longhorn manager pod longhorn-manager-xxxxx to fix the backup listing issue.

But we want to know why Longhorn cannot create the volume automatically. Can you provide us some information as described in this comment about your environment so we can debug it?

7

@rsoika
I replied in the GitHub issue [Question] unable to mount NFS backup · Issue #2242 · longhorn/longhorn · GitHub