Cluster Snapshot failing (failed to connect to etcd host) attempting to connect to node address not in the cluster

Rancher
1

I have a 4 node cluster maintained by a Rancher HA (v2.3.6). The cluster is in AWS and we are using SpotInst.

I select the “Snapshot Now” on the cluster from the Rancher UI and it starts. When I open the Tools>Snapshots menu I see the following error:
Cluster must have at least one etcd plane host: failed to connect to the following etcd host(s) [172.31.55.175]

Unfortunately this address is not a part of my cluster, however I believe that this address was a part of the cluster recently.

I do have snapshots that were successful (2 months ago) but the more recent ones have all failed.

Using the etcd troubleshooting: I have confirmed that there is an etcd leader in my cluster and all of the nodes are connected. The leader is 172.31.49.196

Can someone direct me towards solving this problem?
Is there other information you need from me?

Cheers,
Shaun

2

I guess no one has had this issue before. I have seen that someone had a Snapshot issue and they restarted the Master Node (which I am assuming is the ETCD leader), but their IP address was still valid (it did point to the leader).

3

Well my plan here is to use the Move Leader command in etcd and hope that this lets Rancher know the IP address of the actual leader.