Rancheros.iso out of memory issue

Hi

I’m booting the rancheros.iso, v1.1.0 on a Vmware machine with plenty of ram. After running a Rancher Server container and a Mariadb container for a few minutes, everything stops working and I get out of memory messages. Cat /proc/meminfo shows that plenty of memory is still available but df shows that the subfolders of /var/lib/docker/ is full:

[root@rancherserver rancher]# cat /proc/meminfo 
MemTotal:        8177064 kB
MemFree:         3012180 kB
MemAvailable:    2977136 kB
Buffers:               0 kB
Cached:          4408668 kB
SwapCached:            0 kB
Active:          1506452 kB
Inactive:        3189440 kB
Active(anon):    1506384 kB
Inactive(anon):  3189212 kB
Active(file):         68 kB
Inactive(file):      228 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        250736 kB
Mapped:           118816 kB
Shmem:           4408372 kB
Slab:             429980 kB
SReclaimable:     186320 kB
SUnreclaim:       243660 kB
KernelStack:        5024 kB
PageTables:         3700 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     4088532 kB
Committed_AS:    6493232 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:     96256 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      139200 kB
DirectMap2M:     8249344 kB
[root@rancherserver rancher]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
overlay                4088532    339920   3748612   8% /
tmpfs                  4067200         0   4067200   0% /dev
tmpfs                  4088532         0   4088532   0% /sys/fs/cgroup
tmpfs                  4088532         0   4088532   0% /mnt
tmpfs                  4088532         0   4088532   0% /media
none                   4088532      1264   4087268   0% /run
devtmpfs               4067200         0   4067200   0% /host/dev
shm                      65536         0     65536   0% /host/dev/shm
none                   4088532      1264   4087268   0% /var/run
tmpfs                  4088532    339920   3748612   8% /etc/hostname
shm                      65536         0     65536   0% /dev/shm
devtmpfs               4067200         0   4067200   0% /dev
shm                      65536         0     65536   0% /dev/shm
10.10.33.20:/MSF_Backup/rancher-configs/rancherserver
                     209715200   2480384 207234816   1% /home/rancher/conf
10.10.33.20:/MSF_Backup/rancher-configs/rancherserver/ssh
                     209715200   2480384 207234816   1% /etc/ssh
10.10.33.20:/MSF_Backup/rancher-db/conf
                     209715200   2480384 207234816   1% /home/rancher/rancher-db/conf
10.10.33.20:/MSF_Backup/rancher-states/rancherserver
                     209715200   2480384 207234816   1% /var/lib/rancher/state
overlay                4088532    339920   3748612   8% /var/lib/kubelet
overlay                4067188   4067188         0 100% /var/lib/docker/overlay/e0e52117d90b6f738c9078cc392802ce556aedefeeb55c680a0903a1ac4a2e2b/merged
shm                      65536         0     65536   0% /var/lib/docker/containers/9314362b5ecbf46b4a6b635938414e15ed5e50c064202f66ded9aabb048bec07/shm
overlay                4067188   4067188         0 100% /var/lib/docker/overlay/86714be636451e431b0ac789d538b9e6bcf5b64853b5492f6305926a3b443122/merged
shm                      65536         0     65536   0% /var/lib/docker/containers/9a9b6fde279d7c34b53f9a35aebce7153ebf0c6699cbc33aa688288c18860004/shm
overlay                4067188   4067188         0 100% /var/lib/docker/overlay/a5e24c3fc90cc4eadc6023975ca87f4e6bc0f316eb957e37e0edbc2619164992/merged
shm                      65536         0     65536   0% /var/lib/docker/containers/175d78e7a016b166ef12d6be0e5cd30d7987bea4d1b7382b65831328bef15746/shm
overlay                4067188   4067188         0 100% /var/lib/docker/overlay/3cc50d8312efa8d43e709ce9571302deccc6cc85845defc6983e687c538feedd/merged
shm                      65536         0     65536   0% /var/lib/docker/containers/c43a01d50467a752aca2a1305ab6721a7067f38c2977f21110a698208201390d/shm
overlay                4067188   4067188         0 100% /var/lib/docker/overlay/0556a4cba6186b1f36459ad2a8f1640ffd21b5670c4a65e3601a74a8980661ce/merged
shm                      65536         0     65536   0% /var/lib/docker/containers/b35085aafcd01157533a27ab2fbaab63593c9fae65439f54fd28f5897d873c60/shm

.

Adding a zram swap only makes it worse since none of the swap is used by ROS:

[root@rancherserver rancher]# cat /proc/meminfo 
MemTotal:        8177060 kB
MemFree:         2153284 kB
MemAvailable:    2125408 kB
Buffers:               8 kB
Cached:          4556180 kB
SwapCached:            0 kB
Active:          2224884 kB
Inactive:        3278164 kB
Active(anon):    2224816 kB
Inactive(anon):  3277928 kB
Active(file):         68 kB
Inactive(file):      236 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       6132792 kB
SwapFree:        6132792 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        932948 kB
Mapped:           199288 kB
Shmem:           4555884 kB
Slab:             443800 kB
SReclaimable:     193420 kB
SUnreclaim:       250380 kB
KernelStack:        8672 kB
PageTables:         6212 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    10221320 kB
Committed_AS:    7910000 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:    483328 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      116672 kB
DirectMap2M:     8271872 kB
[root@rancherserver rancher]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
overlay                4088528    487468   3601060  12% /
tmpfs                  4067196         0   4067196   0% /dev
tmpfs                  4088528         0   4088528   0% /sys/fs/cgroup
none                   4088528      1228   4087300   0% /run
tmpfs                  4088528         0   4088528   0% /mnt
tmpfs                  4088528         0   4088528   0% /media
none                   4088528      1228   4087300   0% /var/run
devtmpfs               4067196         0   4067196   0% /host/dev
shm                      65536         0     65536   0% /host/dev/shm
tmpfs                  4088528    487468   3601060  12% /etc/hostname
shm                      65536         0     65536   0% /dev/shm
devtmpfs               4067196         0   4067196   0% /dev
shm                      65536         0     65536   0% /dev/shm
10.10.33.20:/MSF_Backup/rancher-configs/rancherserver
                     209715200   2476096 207239104   1% /home/rancher/conf
10.10.33.20:/MSF_Backup/rancher-configs/rancherserver/ssh
                     209715200   2476096 207239104   1% /etc/ssh
10.10.33.20:/MSF_Backup/rancher-db/conf
                     209715200   2476096 207239104   1% /home/rancher/rancher-db/conf
10.10.33.20:/MSF_Backup/rancher-states/rancherserver
                     209715200   2476096 207239104   1% /var/lib/rancher/state
overlay                4067188   4067188         0 100% /var/lib/docker/overlay/9f41358b567d8c03a05ab220b6efca1442d80c59a0e8bf0c3e772f1909999a1a/merged
shm                      65536         0     65536   0% /var/lib/docker/containers/9dcea60e77dbf143bbde479ebfada33b3782395fab064154449988ec8e665d56/shm
overlay                4088528    487468   3601060  12% /var/lib/kubelet
overlay                4067188   4067188         0 100% /var/lib/docker/overlay/0e83acfa66ce6f217a9a461e59a025b1bae7168a5720a5d5530bc43521334b45/merged
shm                      65536         0     65536   0% /var/lib/docker/containers/0dc7f3089f421154754853889107ea6b8eb287fa3caffb60161abf500e0ea42a/shm
overlay                4067188   4067188         0 100% /var/lib/docker/overlay/e100bffb0ea90c4ebd8129c22122470ec6e07e501248926b46571b9467ec2136/merged
shm                      65536         0     65536   0% /var/lib/docker/containers/cccfe21ee0e5ec72c81822871fbb8933262a754a9f6b093723fcde1989ad0f74/shm
overlay                4067188   4067188         0 100% /var/lib/docker/overlay/ec33237a3f8f148476bc2853e9303917e7614291658e8716f4991998291f4108/merged
shm                      65536         0     65536   0% /var/lib/docker/containers/8f1aa6bef08bbd692d8432b410fe833fbc574f33d72a33587b697780404e897a/shm
overlay                4067188   4067188         0 100% /var/lib/docker/overlay/eff73b568b6824c59b8272eea9c017efaefc648257bf96cc5ec6458e7a97242c/merged
shm                      65536         0     65536   0% /var/lib/docker/containers/1a4aa4f4f842710988d977da611f2f1c7f4713c34d9f822b31254983d4c8bc17/shm
overlay                4067188   4067188         0 100% /var/lib/docker/overlay/7dff21dc04f6e3f670e9d948489470de84f6897151ca98cfaec41dc746b4b8ec/merged
shm                      65536         0     65536   0% /var/lib/docker/containers/58dd0574848a7428ac99677d8000184104f97000af8bd95c518be8a06800de6d/shm

.
Is this a bug or expected behavior? What can I do to make ROS use the swap, and how can I make the overlay mounts larger?

Regards
HU

I found that it is possible to resize /var/lib/docker/overlay:

mount -o remount,size=100% /var/lib/docker/overlay