Rancher-nfs painfully slow

Rancher 1.x
1

I’m running Rancher 1.2 on RancherOS 0.7 iso. I’m using rancher-nfs with joebiellik/nfs4 as nfs-server. Everything starts fine, there are no healthcheck issues and I’m able to create and add volumes to containers. The problem is the speed. All diskoperations on the rancher-nfs volumes takes forever. It takes about 15 seconds just to get a simple directory listing. The nfs-driver logs are full of errors, but they doesn’t say me anything. Here’s an example:

2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=mount.request name=qgisserverprojects
2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=“qgisserverprojects already mounted on /var/lib/rancher/volumes/rancher-nfs/qgisserverprojects”
2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=mount.response mountpoint=“/var/lib/rancher/volumes/rancher-nfs/qgisserverprojects”
2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=mount.request name=qgisservervarnish
2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=“qgisservervarnish already mounted on /var/lib/rancher/volumes/rancher-nfs/qgisservervarnish”
2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=mount.response mountpoint=“/var/lib/rancher/volumes/rancher-nfs/qgisservervarnish”
2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=unmount.request name=qgisserverprojects
2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=unmount.response
2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=unmount.request name=qgisservervarnish
2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=unmount.response
2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=mount.request name=qgisservervarnish
2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=“qgisservervarnish already mounted on /var/lib/rancher/volumes/rancher-nfs/qgisservervarnish”
2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=mount.response mountpoint=“/var/lib/rancher/volumes/rancher-nfs/qgisservervarnish”
2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=unmount.request name=qgisservervarnish
2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=unmount.response
2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=mount.request name=qgisserverprojects
2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=“qgisserverprojects already mounted on /var/lib/rancher/volumes/rancher-nfs/qgisserverprojects”
2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=mount.response mountpoint=“/var/lib/rancher/volumes/rancher-nfs/qgisserverprojects”
2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=mount.request name=qgisservervarnish
2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=“qgisservervarnish already mounted on /var/lib/rancher/volumes/rancher-nfs/qgisservervarnish”
2016-12-07 10:00:00time=“2016-12-07T09:00:00Z” level=info msg=mount.response mountpoint=“/var/lib/rancher/volumes/rancher-nfs/qgisservervarnish”
2016-12-07 10:00:01time=“2016-12-07T09:00:01Z” level=info msg=unmount.request name=qgisserverprojects
2016-12-07 10:00:01time=“2016-12-07T09:00:01Z” level=info msg=unmount.response
2016-12-07 10:00:01time=“2016-12-07T09:00:01Z” level=info msg=unmount.request name=qgisservervarnish
2016-12-07 10:00:01time=“2016-12-07T09:00:01Z” level=info msg=unmount.response
2016-12-07 10:00:01time=“2016-12-07T09:00:01Z” level=info msg=mount.request name=qgisservervarnish
2016-12-07 10:00:01time=“2016-12-07T09:00:01Z” level=info msg=“qgisservervarnish already mounted on /var/lib/rancher/volumes/rancher-nfs/qgisservervarnish”
2016-12-07 10:00:01time=“2016-12-07T09:00:01Z” level=info msg=mount.response mountpoint=“/var/lib/rancher/volumes/rancher-nfs/qgisservervarnish”
2016-12-07 10:00:01time=“2016-12-07T09:00:01Z” level=info msg=unmount.request name=qgisservervarnish
2016-12-07 10:00:01time=“2016-12-07T09:00:01Z” level=info msg=unmount.response

Those kinds of errors appears over and over again in stderror.

Any suggestions how to fix this?

2

Hi,

I’ve got the same issue :

NFS export config :
/mnt *(rw,fsid=0,no_root_squash,no_subtree_check,insecure)

Rancher-NFS config :
MOUNT_OPTS: proto=tcp,port=2049,rw,nfsvers=4
and i tried also
MOUNT_OPTS: rw,nfsvers=4

My response time on services is about 15-20 seconds the first time, then it works normally for 1 minutes. Then if my services stays Idle for more than 1 minute the 15-20 seconds delay reappears.

Thanks for your reply,

K.

3

I did a fresh install of server and hosts and after that it worked fine.

NFS export config :
/nfs-share *(rw,async,no_subtree_check,no_auth_nlm,no_root_squash,crossmnt,no_ac,fsid=0)

Rancher-NFS config :
noacl,noatime,nodiratime,minorversion=1,nolock,nfsvers=4

I’ve set owner and group for the shared folder to nobody:nogroup. My settings makes the share totally insecure but I haven’t published any ports outside Rancher so it should be ok.

1 Like