Backup etcd Cluster of Kubernetes built with Rancher RKE

Rancher
1

Hi all

What are the best practice to back up the etcd cluster of a kubernetes cluster built with rancher RKE?
Does Rancker backup the etcd Cluster automaticly?
In /var/lib/etcd/member, there is a directory “snap” this some file which look like snapshots:

# ls -ahl /var/lib/etcd/member/snap/
[...]
-rw-r--r--. 1 root root 7.1K May 25 04:09 0000000000000002-000000000005f39a.snap
-rw-r--r--. 1 root root 7.1K May 25 05:16 0000000000000002-0000000000061aab.snap
-rw-r--r--. 1 root root 7.1K May 25 06:23 0000000000000002-00000000000641bc.snap
-rw-r--r--. 1 root root 7.1K May 25 07:30 0000000000000002-00000000000668cd.snap
-rw-r--r--. 1 root root 7.1K May 25 08:37 0000000000000002-0000000000068fde.snap
-rw-------. 1 root root  17M May 25 09:04 db

Are these files ectc backups and if yes, how can I configure the backup time and retention?

With Rancher 1.6 you can configure the etcd backups and these backups are stored under /var/etcd/backups.

# ls -ahl /var/etcd/backups/
[...]
drwx------.  3 root root   19 May 17 18:28 2018-05-17T16:28:46Z_etcd_2
drwx------.  3 root root   19 May 17 18:43 2018-05-17T16:43:46Z_etcd_2
drwx------.  3 root root   19 May 17 18:58 2018-05-17T16:58:46Z_etcd_2

How is that solved with Rancher 2.0?

2

Hi,
Look at that link of new docs about backup and restore of ETCD release by Rancher:

https://rancher.com/docs/rancher/v2.x/en/installation/backups-and-restoration/ha-backup-and-restoration/

3

Hi,

does that include the kubernetes-etcd or only the rancher2-etcd?

regards,
Roman

4

Hi all

I found out something, that I did not find in the docs:
If you you choose your Cluster and then click on “Edit”, open the “Cluster Options” and then click on “Edit as YAML”, you can see something like that:

[...]
services: 
  etcd: 
    extra_args: 
      election-timeout: "5000"
      heartbeat-interval: "500"
    snapshot: false
[...]

You can turn “snapshots” to “true”:

[...]
services: 
  etcd: 
    extra_args: 
      election-timeout: "5000"
      heartbeat-interval: "500"
    snapshot: true
[...]

Shapshots of the etcd are then stored on each etcd node unter /opt/rke/etcd-snapshots.
It seeams that etcd is snapshotted every 15 minutes and the snapshots are kept for 24 hours.

Does anyone know, if that is also possible for the etcd of Rancher itself, if Rancher is installed as single node?

Regards,
Domenic

5

So to backup our clusters created with Rancher is it the way to go?

Also, how would you copy the created snapshots externally as a recurring job or something else ? Since it’s created by Rancher I don’t know how could this job run.