Currently using Rancher in Development, looking to go into production in September. It’s a fantastic piece of automation and orchestration software. I have a bit of an issue though!
Whenever I upgrade a container, I lose all data stored in named volumes. Why is this? Am I missing something?
As an example, I have several database nodes running. I upgraded the entire stack (Forced upgrade for all containers), then all data was lost across the board!
Why is this? Should named volumes not be persisted through upgrades?
Are you certain the volumes are gone? I had a similar issue awhile back, but it only happened to rancher/server itself, I upgraded rancherOS and when it came back up it ‘forgot’ my named rancher-data container that had my data volumes attached. The volumes were still there they just were not associated with my named rancher-data data container (which was missing). I ended up using busybox and just attaching all unused volumes until I found what I was looking for (not that hard in this case as rancher/server was the only thing running on that node), then built a new data volume using docker create. Details here
The first thing you should do is docker volumes ls and look through that to see if you can find your data. using something like this:
docker run -v 73c07cf3671:/tmp/test -it busybox /bin/sh
# ls /tmp/test
Are your volumes on a shared storage like NFS? If not, and the volumes are created on the host itself, then the problem might be that when you upgrade, the new container is started on a different host, and the volume gets created on the new host with no data on it.