Best Practices for managing disk space on a cluster


I’m running a Rancher cluster on bare metal, and just had to add new drives and symlinked the Docker folder to the new mounts when the disk space filled up.

I used the steps discussed on the docker github issues.

` First stop Docker

service docker stop
Copy everything from /var/lib/docker to my new destination, which is /mnt/extra in my case

cd /var/lib
sudo rsync -a docker /mnt/extra/
Remove the docker directory

sudo rm -rf docker
Create a static link

sudo ln -s docker /mnt/extra/docker
Edit the /etc/default/docker file

sudo vi /etc/default/docker
Modify this line

#DOCKER_OPTS="--dns --dns"
Changing it to this

DOCKER_OPTS="--dns --dns -g /mnt/extra/docker"
Restart Docker

service docker start
To test perms etc use this command

docker run -it ubuntu bash
If it all works, you're good to go!`

Whilst this works for a in-house cluster that you can schedule downtime for. I was wondering how others used Rancher on Cloud and other environments handle such issues? What would you say is the best practice to avoid disk space fill up caused by Docker?

In my mind I think that you would just slowly migrate the services to another node and take the problem node out of commission, or extend it with a new Block Store Mount. Though this seems like a reactive measure.

I was hoping to get some insight on how best to setup a proactive way to manage disk space issues. Any tips will be appreciated!

1 Like