I have been testing out Rancher for a few days now, and twice I have had to restart the server or container for various reasons. Tonight I was having an issue and my loadbalancer refused to go past initializing. After execing in to each agent and validating I can ping each host, restarting each agent, destroying and adding my hosts back, I decided to maybe restarted the rancher server. I did it through the infra interface, and the container didn’t restart. Started it back up manually but all of my configuration is gone. This happened to me earlier in the week on a different host too. So is this just a known thing to never ever let rancher go do, or you will lose everything?
Are you storing your Rancher config (DB) via an external DB or by bind mounting a volume at start-up?
I generally use ‘bind mounting’ a host volume to store the Rancher DB (See here)
You can also store to an external DB as well. (That is here)
I generally setup my hosts with a root volume called /dockervols/ with subfolders for the individual container names i.e.
/dockervols/rancher
/dockervols/nginx
So with bind mounting the volume the syntax for the Rancher container might be something like:
docker run -d -v /dockervols/rancher:/var/lib/mysql --restart=always -p 8080:8080 rancher/server:latest
Currently I was just running:
docker run -d --restart=always -p 8080:8080 rancher/server per this link
I will test out running it with volumes right now.
When you are testing out if your LB is not going past initializing, you should be testing that you can ping network agents on other hosts, not just the host itself.
What did you do to start it back up manually if you weren’t able to restart the rancher/server container? If you started the same container that was originally launched, the DB and configs would be preserved in the exact same container.
Can you do a docker ps -a to show how many rancher/server containers you have running?
The database is in the rancher/server container by default, and it’s just a regular Docker container that follows the regular lifecycle, so the data exists as long as the container does.
So losing it on reboot suggests that you’re deleting the container somehow, or that the filesystem docker data is being stored on is ephemeral and cleared on reboot.
Vincent - thanks for adding to the clarity of the rancher/server database and that it’s storage remains as long as the container does.
So right - losing it on reboot does point to some other issue that the container itself is going away or there is an issue with the location of where the docker data is being stored.
Also just for the ability to help a bit more - can we get a bit of info about your docker environment?
i.e. Host OS, Docker version, Rancher version
I have the same problem here. I restarted my VM and they are gone. 1 day i restarted my VM and next day they weren’t gone. Now i restarted it again and all my host and everything are gone.
When u create the rancher server, use the bind mount mysql volume, so that data will be persistent even if the rancher server get recreated
sudo docker run -d -v /mysql:/var/lib/mysql --restart=unless-stopped -p 8080:8080 rancher/server
1 Like
@Listener_me
I get this error when im running that command:
docker: Error response from daemon: driver failed programming external connectivity on endpoint cond escending_leavitt (66238d7c3d887889a9134b6e2800d7d7b124ab513bb59dd7accb4a27f9e828e4) Bind for 0.0.0.0:8080 failed: port is already allocated.
You may need to stop the old rancher server before creating new one as the port 8080 is already exposed by the old rancher
1 Like
My rancher server doesn’t even stop anymore? it says docker daemon is running?
You mean you cant stop ur old rancher any more? Something went wrong with docker then. Stop the docker, kill all the processes and then start the docker again.
1 Like