We are looking into switching to Rancher from our current Kops deployment model for ease of multi cluster management.
I have a few questions around how etcd persistence is handled in the case that partial/all of the etcd nodes are terminated. In Kops we have an EBS volume that is backing all of our etcd state and is automatically added to the master nodes on a restart. In this case the data for etcd appears to be ephemeral and upon restart is simply synced from any available node that is currently the master etcd member.
Are there any best practices for handling this in a production deployment of a rancher cluster? Is this assumption that if all of the etcd volumes are terminated for any reason then one should simple rely on the snapshot features to recover the cluster state? Is there a way to have the etcd data store backed by a persistence layer?
Thank you for your time.