Hi
I’m booting the rancheros.iso, v1.1.0 on a Vmware machine with plenty of ram. After running a Rancher Server container and a Mariadb container for a few minutes, everything stops working and I get out of memory messages. Cat /proc/meminfo shows that plenty of memory is still available but df shows that the subfolders of /var/lib/docker/ is full:
[root@rancherserver rancher]# cat /proc/meminfo
MemTotal: 8177064 kB
MemFree: 3012180 kB
MemAvailable: 2977136 kB
Buffers: 0 kB
Cached: 4408668 kB
SwapCached: 0 kB
Active: 1506452 kB
Inactive: 3189440 kB
Active(anon): 1506384 kB
Inactive(anon): 3189212 kB
Active(file): 68 kB
Inactive(file): 228 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 250736 kB
Mapped: 118816 kB
Shmem: 4408372 kB
Slab: 429980 kB
SReclaimable: 186320 kB
SUnreclaim: 243660 kB
KernelStack: 5024 kB
PageTables: 3700 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 4088532 kB
Committed_AS: 6493232 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 96256 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 139200 kB
DirectMap2M: 8249344 kB
[root@rancherserver rancher]# df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 4088532 339920 3748612 8% /
tmpfs 4067200 0 4067200 0% /dev
tmpfs 4088532 0 4088532 0% /sys/fs/cgroup
tmpfs 4088532 0 4088532 0% /mnt
tmpfs 4088532 0 4088532 0% /media
none 4088532 1264 4087268 0% /run
devtmpfs 4067200 0 4067200 0% /host/dev
shm 65536 0 65536 0% /host/dev/shm
none 4088532 1264 4087268 0% /var/run
tmpfs 4088532 339920 3748612 8% /etc/hostname
shm 65536 0 65536 0% /dev/shm
devtmpfs 4067200 0 4067200 0% /dev
shm 65536 0 65536 0% /dev/shm
10.10.33.20:/MSF_Backup/rancher-configs/rancherserver
209715200 2480384 207234816 1% /home/rancher/conf
10.10.33.20:/MSF_Backup/rancher-configs/rancherserver/ssh
209715200 2480384 207234816 1% /etc/ssh
10.10.33.20:/MSF_Backup/rancher-db/conf
209715200 2480384 207234816 1% /home/rancher/rancher-db/conf
10.10.33.20:/MSF_Backup/rancher-states/rancherserver
209715200 2480384 207234816 1% /var/lib/rancher/state
overlay 4088532 339920 3748612 8% /var/lib/kubelet
overlay 4067188 4067188 0 100% /var/lib/docker/overlay/e0e52117d90b6f738c9078cc392802ce556aedefeeb55c680a0903a1ac4a2e2b/merged
shm 65536 0 65536 0% /var/lib/docker/containers/9314362b5ecbf46b4a6b635938414e15ed5e50c064202f66ded9aabb048bec07/shm
overlay 4067188 4067188 0 100% /var/lib/docker/overlay/86714be636451e431b0ac789d538b9e6bcf5b64853b5492f6305926a3b443122/merged
shm 65536 0 65536 0% /var/lib/docker/containers/9a9b6fde279d7c34b53f9a35aebce7153ebf0c6699cbc33aa688288c18860004/shm
overlay 4067188 4067188 0 100% /var/lib/docker/overlay/a5e24c3fc90cc4eadc6023975ca87f4e6bc0f316eb957e37e0edbc2619164992/merged
shm 65536 0 65536 0% /var/lib/docker/containers/175d78e7a016b166ef12d6be0e5cd30d7987bea4d1b7382b65831328bef15746/shm
overlay 4067188 4067188 0 100% /var/lib/docker/overlay/3cc50d8312efa8d43e709ce9571302deccc6cc85845defc6983e687c538feedd/merged
shm 65536 0 65536 0% /var/lib/docker/containers/c43a01d50467a752aca2a1305ab6721a7067f38c2977f21110a698208201390d/shm
overlay 4067188 4067188 0 100% /var/lib/docker/overlay/0556a4cba6186b1f36459ad2a8f1640ffd21b5670c4a65e3601a74a8980661ce/merged
shm 65536 0 65536 0% /var/lib/docker/containers/b35085aafcd01157533a27ab2fbaab63593c9fae65439f54fd28f5897d873c60/shm
.
Adding a zram swap only makes it worse since none of the swap is used by ROS:
[root@rancherserver rancher]# cat /proc/meminfo
MemTotal: 8177060 kB
MemFree: 2153284 kB
MemAvailable: 2125408 kB
Buffers: 8 kB
Cached: 4556180 kB
SwapCached: 0 kB
Active: 2224884 kB
Inactive: 3278164 kB
Active(anon): 2224816 kB
Inactive(anon): 3277928 kB
Active(file): 68 kB
Inactive(file): 236 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 6132792 kB
SwapFree: 6132792 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 932948 kB
Mapped: 199288 kB
Shmem: 4555884 kB
Slab: 443800 kB
SReclaimable: 193420 kB
SUnreclaim: 250380 kB
KernelStack: 8672 kB
PageTables: 6212 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 10221320 kB
Committed_AS: 7910000 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 483328 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 116672 kB
DirectMap2M: 8271872 kB
[root@rancherserver rancher]# df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 4088528 487468 3601060 12% /
tmpfs 4067196 0 4067196 0% /dev
tmpfs 4088528 0 4088528 0% /sys/fs/cgroup
none 4088528 1228 4087300 0% /run
tmpfs 4088528 0 4088528 0% /mnt
tmpfs 4088528 0 4088528 0% /media
none 4088528 1228 4087300 0% /var/run
devtmpfs 4067196 0 4067196 0% /host/dev
shm 65536 0 65536 0% /host/dev/shm
tmpfs 4088528 487468 3601060 12% /etc/hostname
shm 65536 0 65536 0% /dev/shm
devtmpfs 4067196 0 4067196 0% /dev
shm 65536 0 65536 0% /dev/shm
10.10.33.20:/MSF_Backup/rancher-configs/rancherserver
209715200 2476096 207239104 1% /home/rancher/conf
10.10.33.20:/MSF_Backup/rancher-configs/rancherserver/ssh
209715200 2476096 207239104 1% /etc/ssh
10.10.33.20:/MSF_Backup/rancher-db/conf
209715200 2476096 207239104 1% /home/rancher/rancher-db/conf
10.10.33.20:/MSF_Backup/rancher-states/rancherserver
209715200 2476096 207239104 1% /var/lib/rancher/state
overlay 4067188 4067188 0 100% /var/lib/docker/overlay/9f41358b567d8c03a05ab220b6efca1442d80c59a0e8bf0c3e772f1909999a1a/merged
shm 65536 0 65536 0% /var/lib/docker/containers/9dcea60e77dbf143bbde479ebfada33b3782395fab064154449988ec8e665d56/shm
overlay 4088528 487468 3601060 12% /var/lib/kubelet
overlay 4067188 4067188 0 100% /var/lib/docker/overlay/0e83acfa66ce6f217a9a461e59a025b1bae7168a5720a5d5530bc43521334b45/merged
shm 65536 0 65536 0% /var/lib/docker/containers/0dc7f3089f421154754853889107ea6b8eb287fa3caffb60161abf500e0ea42a/shm
overlay 4067188 4067188 0 100% /var/lib/docker/overlay/e100bffb0ea90c4ebd8129c22122470ec6e07e501248926b46571b9467ec2136/merged
shm 65536 0 65536 0% /var/lib/docker/containers/cccfe21ee0e5ec72c81822871fbb8933262a754a9f6b093723fcde1989ad0f74/shm
overlay 4067188 4067188 0 100% /var/lib/docker/overlay/ec33237a3f8f148476bc2853e9303917e7614291658e8716f4991998291f4108/merged
shm 65536 0 65536 0% /var/lib/docker/containers/8f1aa6bef08bbd692d8432b410fe833fbc574f33d72a33587b697780404e897a/shm
overlay 4067188 4067188 0 100% /var/lib/docker/overlay/eff73b568b6824c59b8272eea9c017efaefc648257bf96cc5ec6458e7a97242c/merged
shm 65536 0 65536 0% /var/lib/docker/containers/1a4aa4f4f842710988d977da611f2f1c7f4713c34d9f822b31254983d4c8bc17/shm
overlay 4067188 4067188 0 100% /var/lib/docker/overlay/7dff21dc04f6e3f670e9d948489470de84f6897151ca98cfaec41dc746b4b8ec/merged
shm 65536 0 65536 0% /var/lib/docker/containers/58dd0574848a7428ac99677d8000184104f97000af8bd95c518be8a06800de6d/shm
.
Is this a bug or expected behavior? What can I do to make ROS use the swap, and how can I make the overlay mounts larger?
Regards
HU