Issues having HA properly setup with Let's Encrypt


I’ve been trying to setup a K3S cluster in HA with the following setup. (for the sake of simplicity fake IP’s)
1 proxy server - my-dns ==
2 master servers - &
2 agent servers - &

I tried having both HAProxy and Nginx proxies and also tried using Cloudflare LB where I point ports 6443, 443 and 80 to the masters using round robin.

To run the k3s masters I ran: curl -sfL | sh -s - server --datastore-endpoint=‘mysql://STRING_HERE’ --tls-san=‘my-dns’ --cluster-domain=‘my-dns’

I then follow the steps in the docs and eventually I get to the point to install rancher with Let’sEncrypt, so I ran the following:

helm install rancher rancher-latest/rancher \
  --namespace cattle-system \
  --set hostname=my-dns \
  --set ingress.tls.source=letsEncrypt \

When I then run: kubectl -n cattle-system get ingress
I get the following:
rancher my-dns 80, 443 82s
cm-acme-http-solver-hljwn my-dns 80 60s

and it just stays like this for ever, the acme doesn’t seem to be working at all…
If I just skip the proxy and change my-dns to point to it just works right away.

I’m trying to understand what could be wrong here, I feel like the address in the ingress should be the proxy and not the master01 ip but I don’t understand it much so I can’t be sure.

any how, I can’t seem to get this to work… any clues?
I’ve exhausted all my ideas and researched everything I could ideas would be welcome.

Actually I am having the same problem.
Additionally, in my case, when the acme is performed without the LB the ssl is correctly generated.

Any help is greatly appreciated. thanks

running “kubectl get all --all-namespaces” returns only one service with external IP’s:
kube-system service/traefik LoadBalancer,,, 80:30183/TCP,443:31751/TCP 16m

which seems correct, I’m still trying to understand why letsEncrypt still doesn’t work.

I found some github’s that seem to have about the same issue for example: Pods in different networks are unable to communicate between them. · Issue #1824 · k3s-io/k3s · GitHub

On this particular case the guy had an issue with the external IP’s, I checked my cluster, indeed I had the same no ExternalIP’s for the nodes and my InternalIP’s where the public ones.
I changed that so now my master nodes have an external IP aswell which equals to the InternalIP.
Even after this change, still nothing Let’sEncrypt is not getting generated.

Also I noticed in some docs/refs that when people use k8s in HA and install Rancher, when they run “kubectl -n cattle-system get ingress” they actually see all the master’s IP’s in the acme ADDRESS. In my case I only see one, and it’s master01 could that be part of the issue?

I’m going to continue researching but if u have any ideas… would be greatly appreciated

I’ll try to setup a new cluster using RKE this time and try the all thing again and see if it works.
Even if there’s no answers here at least it serves as future reference.