Restore etcd quorum after data center outage

Rancher
1

Hi,

Assuming there is a 6-nodes k8s cluster spanning across 2 data centers, 3 nodes in each. Both DCs has some amount of etcd and control plane nodes. I would like to know how to restore the cluster functionally if one of the DCs fails permanently. In this scenario nodes in remaining DC will continue functioning, but etcd will be in an unhealthy state because it won’t be able to form the quorum. How to restore quorum in the remaining DC and have fully-functional cluster? Rancher itself is deployed to some other third DC (cloud) and is not affected by the outage.

Thanks,
Andrii

2

After I figured out how to do it by removing existing etcd container and staring new one with same configuration and --force-new-cluster flag, I found this article that explains same approach https://blog.devget.net/devops/rancher-etcd-disaster-recovery/.