K3s control planes behind an loadbalancer

I am trying to bring up 3 k3s control planes that are all using a load balancer vIP that load balances the 3 control planes.

we are trying to configure the first control plane like this

curl -sfL https://get.k3s.io | K3S_TOKEN=SECRET sh -s - server --server https://loadbalancerIP:6443 --cluster-init --disable=traefik --tls-san=loadbalancerIP

the loadbalancerIP is setup and has a pool that includes the 3 control plane ips on port 6443

this is throwing an error when starting k3s.

If I run without the --server https://loadbalancerIP:6443 it will install and start k3s but its not communicating via the loadbalancer IP.

If I get the first control plane up without the --server https://loadbalancerIP:6443

then add the 2nd and 3rd control plane with

curl -sfL https://get.k3s.io | K3S_TOKEN=uniquetoken sh -s - server --server https://loadbalancerIP:6443 --disable=traefik --tls-san=loadbalancerIP

those will install and startup.

This feels like the 2nd and 3rd are communicating via the loadbalancerIP but the first one is not.

what am i missing?