NFS backup target (localhost)

Longhorn
1

Setting up NFS target for backup was successful using a separate NFS server.
I want to change now NFS to my localhost (well, being not best practice, but in my case useful).
NFS server is running and manual mount via shell is successful (138.201.199.xxx is my localhost).

Changing NFS target in Longhorn shows following:

2020-06-11T05:53:27.363339046+02:00 time=“2020-06-11T03:53:27Z” level=error msg=“Error in request: error listing backups: error listing backup volumes: Failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.0.0/longhorn [backup ls --volume-only nfs://138.201.199.xxx:/longhorn], output Cannot mount nfs 138.201.199.xxx:/longhorn: Failed to execute: mount [-t nfs4 138.201.199.xxx:/longhorn /var/lib/longhorn-backupstore-mounts/138_201_199_xxx/longhorn], output mount.nfs4: access denied by server while mounting 138.201.199.xxx:/longhorn\n, error exit status 32\n, stderr, time="2020-06-11T03:53:27Z" level=error msg="Cannot mount nfs 138.201.199.xxx:/longhorn: Failed to execute: mount [-t nfs4 138.201.199.xxx:/longhorn /var/lib/longhorn-backupstore-mounts/138_201_199_xxx/longhorn], output mount.nfs4: access denied by server while mounting 138.201.199.xxx:/longhorn\n, error exit status 32"\n, error exit status 1”

I checked many times and could not find the issue.

/var/lib/longhorn-backupstore-mounts/138_201_199_xxx/longhorn

exists and has all permissions. Any idea from your side?

2

It sounds like your NFS server has some kind of access control set up. Can you try to run the same mount command inside longhorn-manager pod? It should be easier to debug in that way.

Also, make sure your NFS server supports NFSv4. Longhorn doesn’t support NFSv3.

3

Yes, always on NFSv4.
As said: mount within my node works well. If target is another node, as well no problems.
From within longhorn-manager pod I have the same result: no success.

First try set /etc/exports to (as it works with another target node):

/srv/nfsv4/ 138.201.199.xxx/32(rw,sync,root_squash,no_subtree_check,fsid=0)
/srv/nfsv4/longhorn 138.201.199.xxx/32(rw,sync,root_squash,no_subtree_check)

will result (within longhorn-manager) in:

mount -v -t nfs4 138.201.199.xxx:/longhorn ./longhorn
mount.nfs4: timeout set for Fri Jun 12 03:20:56 2020
mount.nfs4: trying text-based options ‘vers=4.2,addr=138.201.199.xxx,clientaddr=10.42.0.zzz’
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting 138.201.199.xxx:/longhorn

Then I added into /etc/exports longhorn-manager pod IP:

/srv/nfsv4/ 138.201.199.xxx/32(rw,sync,root_squash,no_subtree_check,fsid=0)
/srv/nfsv4/longhorn 138.201.199.xxx/32(rw,sync,root_squash,no_subtree_check)
/srv/nfsv4/longhorn 10.42.0.zzz/32(rw,sync,root_squash,no_subtree_check)

Slightly different result:

mount -v -t nfs4 138.201.199.xxx:/longhorn ./longhorn
mount.nfs4: timeout set for Fri Jun 12 03:23:33 2020
mount.nfs4: trying text-based options ‘vers=4.2,addr=138.201.199.xxx,clientaddr=10.42.0.yyy’
mount.nfs4: mount(2): No such file or directory
/usr/sbin/start-statd: 23: /usr/sbin/start-statd: systemctl: not found
mount.nfs4: trying text-based options ‘addr=138.201.199.xxx’
mount.nfs4: prog 100003, trying vers=3, prot=6
mount.nfs4: trying 138.201.199.xxx prog 100003 vers 3 prot TCP port 2049
mount.nfs4: prog 100005, trying vers=3, prot=17
mount.nfs4: trying 138.201.199.xxx prog 100005 vers 3 prot UDP port 47256
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting 138.201.199.xxx:/longhorn

A port problem between pod and localhost, which is “different” when using another node?
Using a Bridge, when accessing another node, but not when using localhost?
Still confusing…

4

Hi,
I hit the same (or very similar) problem.
In my case the problem was firewall and NFS server access rules. The pods in fact live on the overlayed network, one need to give access to NFS ports also from that network.

My flannel nework is: 10.42.0.0/16 (you may change that)

Add firewall rules on host:

sudo ufw allow in from 10.42.0.0/16 to any port 2049
sudo ufw allow in from 10.42.0.0/16 to any port 111

Add to “/etc/exports” file

nfs-dir 10.42.0.0/16(rw,sync,no_subtree_check)

Then restart NFS:

sudo exportfs -a
sudo systemctl restart nfs-kernel-server