RKE2 HA not working

Hello all -

I installed RKE2 v1.26.1, three master nodes and two workers. Install and “normal” operation are working as expected. However, wherever I reboot any one of the three master nodes, kubectl command do not function until all the masters are up. I see the following in my /var/log/messages:

Feb 21 14:11:36 kubehost-02 rke2[1434]: time="2023-02-21T14:11:36-05:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.222.178:9345: connect: connection refused"
Feb 21 14:11:36 kubehost-02 rke2[1434]: time="2023-02-21T14:11:36-05:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.222.178:9345: connect: connection refused"
Feb 21 14:11:41 kubehost-02 rke2[1434]: time="2023-02-21T14:11:41-05:00" level=info msg="Connecting to proxy" url="wss://192.168.222.178:9345/v1-rke2/connect"
Feb 21 14:11:41 kubehost-02 rke2[1434]: time="2023-02-21T14:11:41-05:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 192.168.222.178:9345: connect: connection refused"
Feb 21 14:11:41 kubehost-02 rke2[1434]: time="2023-02-21T14:11:41-05:00" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.222.178:9345: connect: connection refused"
Feb 21 14:11:46 kubehost-02 rke2[1434]: time="2023-02-21T14:11:46-05:00" level=info msg="Connecting to proxy" url="wss://192.168.222.178:9345/v1-rke2/connect"

I’m completely confused as to what’s going on here. My .kube/config (on each master) is set to connect to https://127.0.0.1:6443. RKE2 seems to have inserted itself somethere between kubectl and my cluster…?

  • What is this “proxy” mentioned in the above logs?
  • Why does the proxy not work if any one of my three masters is down?
  • How can I fix this so it works as expected (commands continue to work even if one master is down)?

Thanks!