So, I have rancher/server started with /var/lib/mysql as a static volume, so that nothing is lost when I retsart it. This all works fine.
I need to be able to backup the Rancher data in oder to rebuild the cluster and stacks if necessary.
However, I can’t really back up the mysql data directory and be confident of consistency. What is the recommended way to do this? It would be nice to have a button or schedule in Rancher itself to have periodic dumps of the database to a location that could be backed up, rather than having to do something ugly. I suppose I could put the mysql database into a separate container, but then I still have the question of how to back that one up.
How do other people do this?
I was thinking of finding a way to slave it to my non-dockerized infrastructure, but that would require modifications to the server container which I want to avoid.
I do personally feel you could do a docker stop, rsync the database dir off-server, then docker start again as part of a maintenance routine and be safe with data consistency I would /think/.
I’ve tried a couple of things -
- First, do “docker pause rancher-server” and then tar up the mysql data (since it will be consistent) and then “docker unpause rancher-server”.
- Alternatively, “docker exec rancher-server mysqldump cattle | gzip -c -q - > backupfile.sql.gz”
In both cases, the archive file can be dropped onto an NFS drive for backup.
Since I’m running everything under CoreOS, I have to use a systemd timer in the YAML to schedule the job to run daily.
I don’t believe a pause would cause mysql to finish any cached writes however, not that it is a high-risk with something that hits the db as little as Rancher.
I was interested in the same subject and I want to find a way to do “hot backups”, meaning backup of online databases without blocking them. I found an open source Percona XtraBackup which seems to support all versions of mysql … there seems to be some docker builds available
So one option would be to use one such backup software or other similar. I haven’t tried it yet.
Given how small the rancher db is (well, relative to our other databases) and how simple a recovery should be, I tihnk the ‘docker exec rancher-server mysqldump’ solution might be the easiest, particularly since (in our case) the entire server can be rebuilt quickly by just a CoreOS image and the YAML file. The only bit missing is the data…
I would go with the basic
mysqldump. @deitch created a container that schedules it and can write to some remote storage, How do I back up the database for Rancher?
I’ve tried a XtraBackup method using an image (I think it was https://github.com/martin-helmich/docker-xtrabackup). The main issue with this is that the
my.cnf for Rancher and XtraBackup need to be in sync or it fails. I didn’t go the extra mile to try and sync / merge the configs together yet.
I’ll try out @deitch method next! Though, minor nitpick, you should base from a qualified
alpine image (such as
alpine:3.2) to avoid getting updated to 3.3 or other version without noticing (in case it breaks something).
You know, @andyshinn, right you are. That would be a pain if it broke. I will fix it. If you don’t mind opening an issue on github…
@deitch thank you! I will try your solution looks appropriate for the current setup, and indeed XtraBackup is kind of overkill here.