I’m running a Rancher cluster on bare metal, and just had to add new drives and symlinked the Docker folder to the new mounts when the disk space filled up.
I used the steps discussed on the docker github issues.
` First stop Docker
service docker stop Copy everything from /var/lib/docker to my new destination, which is /mnt/extra in my case cd /var/lib sudo rsync -a docker /mnt/extra/ Remove the docker directory sudo rm -rf docker Create a static link sudo ln -s docker /mnt/extra/docker Edit the /etc/default/docker file sudo vi /etc/default/docker Modify this line #DOCKER_OPTS="--dns 188.8.131.52 --dns 184.108.40.206" Changing it to this DOCKER_OPTS="--dns 220.127.116.11 --dns 18.104.22.168 -g /mnt/extra/docker" Restart Docker service docker start To test perms etc use this command docker run -it ubuntu bash If it all works, you're good to go!`
Whilst this works for a in-house cluster that you can schedule downtime for. I was wondering how others used Rancher on Cloud and other environments handle such issues? What would you say is the best practice to avoid disk space fill up caused by Docker?
In my mind I think that you would just slowly migrate the services to another node and take the problem node out of commission, or extend it with a new Block Store Mount. Though this seems like a reactive measure.
I was hoping to get some insight on how best to setup a proactive way to manage disk space issues. Any tips will be appreciated!