What is the recommended way to backup Rancher?

Rancher 1.x
1

So, I have rancher/server started with /var/lib/mysql as a static volume, so that nothing is lost when I retsart it. This all works fine.

I need to be able to backup the Rancher data in oder to rebuild the cluster and stacks if necessary.

However, I can’t really back up the mysql data directory and be confident of consistency. What is the recommended way to do this? It would be nice to have a button or schedule in Rancher itself to have periodic dumps of the database to a location that could be backed up, rather than having to do something ugly. I suppose I could put the mysql database into a separate container, but then I still have the question of how to back that one up.

How do other people do this?

1 Like
2

I was thinking of finding a way to slave it to my non-dockerized infrastructure, but that would require modifications to the server container which I want to avoid.

I do personally feel you could do a docker stop, rsync the database dir off-server, then docker start again as part of a maintenance routine and be safe with data consistency I would /think/.

3

I’ve tried a couple of things -

  • First, do “docker pause rancher-server” and then tar up the mysql data (since it will be consistent) and then “docker unpause rancher-server”.
  • Alternatively, “docker exec rancher-server mysqldump cattle | gzip -c -q - > backupfile.sql.gz”

In both cases, the archive file can be dropped onto an NFS drive for backup.

Since I’m running everything under CoreOS, I have to use a systemd timer in the YAML to schedule the job to run daily.

4

I don’t believe a pause would cause mysql to finish any cached writes however, not that it is a high-risk with something that hits the db as little as Rancher.

5

I was interested in the same subject and I want to find a way to do “hot backups”, meaning backup of online databases without blocking them. I found an open source Percona XtraBackup which seems to support all versions of mysql … there seems to be some docker builds available
So one option would be to use one such backup software or other similar. I haven’t tried it yet.

6

Given how small the rancher db is (well, relative to our other databases) and how simple a recovery should be, I tihnk the ‘docker exec rancher-server mysqldump’ solution might be the easiest, particularly since (in our case) the entire server can be rebuilt quickly by just a CoreOS image and the YAML file. The only bit missing is the data…

7

I would go with the basic mysqldump. @deitch created a container that schedules it and can write to some remote storage, How do I back up the database for Rancher?

2 Likes
8

Yes, I did, and open-sourced it. Available at Docker hub https://hub.docker.com/r/deitch/mysql-backup/ with the source at https://github.com/deitch/mysql-backup

1 Like
9

I’ve tried a XtraBackup method using an image (I think it was https://github.com/martin-helmich/docker-xtrabackup). The main issue with this is that the my.cnf for Rancher and XtraBackup need to be in sync or it fails. I didn’t go the extra mile to try and sync / merge the configs together yet.

I’ll try out @deitch method next! Though, minor nitpick, you should base from a qualified alpine image (such as alpine:3.2) to avoid getting updated to 3.3 or other version without noticing (in case it breaks something).

10

You know, @andyshinn, right you are. That would be a pain if it broke. I will fix it. If you don’t mind opening an issue on github…

11

@deitch thank you! I will try your solution looks appropriate for the current setup, and indeed XtraBackup is kind of overkill here.