I’ve just installed a fresh RancherOS (4.9.40-rancher) on some dedicated hardware here at home - but now I get some issues when I’m trying to mount an NFS share I get “no space left on device” errors inside Rancher. I’ve checked - but there is plenty of space on both the rancher box and on the NFS server, and RancherOS is installed to disk.
So after some research, it may have something to do with the number of inodes / number of open files.
Now, when I do sudo df -h I get 16447 lines as output, most of them /var/run lines.
Is this normal? I only run a handful of containers.
This is a couple of lines out of those thousands of lines:
none 7.7G 1.2M 7.7G 0% /var/run
none 7.7G 1.2M 7.7G 0% /var/run
none 7.7G 1.2M 7.7G 0% /run
none 7.7G 1.2M 7.7G 0% /run
If someone could point me in the right direction that would be great. Thanks!
I’m very surprised that you’d get that many on a fresh RancherOS - I can imagine that if there’s a bug, that it might happen if you’ve been running for quite a while, or been starting lots of containers.
Can you tell us a little more about what this machine has been doing since it was installed? did you change user docker version, are you running containers in the system-docker, or the user-docker, what console?
Hi Sven, Thanks for the quick reply!
I’ve done the following steps on this machine so far:
- put rancheros iso on a usb stick, installed it on three hosts by setting cloud config with an ssh key and doing ros install. I was assuming this command also erases the harddrive its pointed at?
- then without doing anything else, I installed rancher.
- i then added the three agents for the three hosts, specifiying each host ip to make sure it gets the right one
- then the hosts show up, I installed rancher nfs.
- i configured it, created three volumes and added a stack that uses these volumes
- at first I named the volume driver ‘rancher-nfs’, and since it was mentioning that its a legacy driver, I’ve renamed that to nfs. (the default name of the nfs stack?)
- at that moment, it seemed to work briefly - the volumes were showing in the ui as being mounted,
- then I’m trying to use ‘nfs’ as a driver name instead of ‘rancher-nfs’. that upgrade never finishes, it seems that the ‘no space left on device’ error is preventing it from completing.
After a night of running, it still at:
[rancher@rancher2 ~]$ sudo df -h | wc -l
16447
So no increase it seems. I’ve just checked the other two hosts, and they give:
[rancher@rancher1 ~]$ sudo df -h | wc -l
4153
[rancher@rancher3 ~]$ sudo df -h | wc -l
16437
since its 3 new machines (set up yesterday) - i’m fine to try anything to resolve the issue.
thanks for helping me out!
I think I’ll try to wipe the partition table before doing a ros install. See if that helps, maybe there is something leftover of older rancher installs on that disk.
Ok maybe this needs to move to the Rancher forum instead.
I’ve reinstalled rancheros, and everything is fine up to the moment I deploy a stack that uses an NFS mount.
This is the compose file:
version: '2'
services:
openhab:
image: openhab/openhab:2.1.0-amd64
environment:
OPENHAB_HTTPS_PORT: '8443'
OPENHAB_HTTP_PORT: '8090'
network_mode: host
dns_search:
- home.local
volumes:
- openhab_addons:/openhab/addons
- openhab_conf:/openhab/conf
- openhab_userdata:/openhab/userdata
dns:
- 192.168.178.2
domainname: ha
The three volumes have no mention of nfs - but it seems that based on the names they start to get mounted. Then I get the following error:
It gives this error and keeps restarting.
Then the df filesystems rise on the host the stack was allocated to, all the way to 16445
and then it stops and gives the initial ‘out of space’ error.
What do you think could be causing this ‘legacy plugin, plugin not found error’
I’m installing NFS on a fresh rancher, using the catalog.
