Hi,
I’m running a Rancher cluster on bare metal, and just had to add new drives and symlinked the Docker folder to the new mounts when the disk space filled up.
I used the steps discussed on the docker github issues.
` First stop Docker
service docker stop
Copy everything from /var/lib/docker to my new destination, which is /mnt/extra in my case
cd /var/lib
sudo rsync -a docker /mnt/extra/
Remove the docker directory
sudo rm -rf docker
Create a static link
sudo ln -s docker /mnt/extra/docker
Edit the /etc/default/docker file
sudo vi /etc/default/docker
Modify this line
#DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"
Changing it to this
DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 -g /mnt/extra/docker"
Restart Docker
service docker start
To test perms etc use this command
docker run -it ubuntu bash
If it all works, you're good to go!`
Whilst this works for a in-house cluster that you can schedule downtime for. I was wondering how others used Rancher on Cloud and other environments handle such issues? What would you say is the best practice to avoid disk space fill up caused by Docker?
In my mind I think that you would just slowly migrate the services to another node and take the problem node out of commission, or extend it with a new Block Store Mount. Though this seems like a reactive measure.
I was hoping to get some insight on how best to setup a proactive way to manage disk space issues. Any tips will be appreciated!