Issues having HA properly setup with Let's Encrypt

k3s, k3OS, and k3d
1

Hello,

I’ve been trying to setup a K3S cluster in HA with the following setup. (for the sake of simplicity fake IP’s)
1 proxy server - my-dns == 122.122.122.1
2 master servers - 122.122.122.2 & 122.122.122.3
2 agent servers - 122.122.122.4 & 122.122.122.5

I tried having both HAProxy and Nginx proxies and also tried using Cloudflare LB where I point ports 6443, 443 and 80 to the masters using round robin.

To run the k3s masters I ran: curl -sfL https://get.k3s.io | sh -s - server --datastore-endpoint=‘mysql://STRING_HERE’ --tls-san=‘my-dns’ --cluster-domain=‘my-dns’

I then follow the steps in the docs and eventually I get to the point to install rancher with Let’sEncrypt, so I ran the following:

helm install rancher rancher-latest/rancher \
  --namespace cattle-system \
  --set hostname=my-dns \
  --set ingress.tls.source=letsEncrypt \
  --set letsEncrypt.email=me@example.org

When I then run: kubectl -n cattle-system get ingress
I get the following:
NAME CLASS HOSTS ADDRESS PORTS AGE
rancher my-dns 80, 443 82s
cm-acme-http-solver-hljwn my-dns 122.122.122.2 80 60s

and it just stays like this for ever, the acme doesn’t seem to be working at all…
If I just skip the proxy and change my-dns to point to 122.122.122.2 it just works right away.

I’m trying to understand what could be wrong here, I feel like the address in the ingress should be the proxy and not the master01 ip but I don’t understand it much so I can’t be sure.

any how, I can’t seem to get this to work… any clues?
I’ve exhausted all my ideas and researched everything I could ideas would be welcome.

2

Actually I am having the same problem.
Additionally, in my case, when the acme is performed without the LB the ssl is correctly generated.

Any help is greatly appreciated. thanks

3

Also,
running “kubectl get all --all-namespaces” returns only one service with external IP’s:
kube-system service/traefik LoadBalancer 10.43.30.122 122.122.122.2,122.122.122.3,122.122.122.4,122.122.122.5 80:30183/TCP,443:31751/TCP 16m

which seems correct, I’m still trying to understand why letsEncrypt still doesn’t work.

4

I found some github’s that seem to have about the same issue for example: Pods in different networks are unable to communicate between them. · Issue #1824 · k3s-io/k3s · GitHub

On this particular case the guy had an issue with the external IP’s, I checked my cluster, indeed I had the same no ExternalIP’s for the nodes and my InternalIP’s where the public ones.
I changed that so now my master nodes have an external IP aswell which equals to the InternalIP.
Even after this change, still nothing Let’sEncrypt is not getting generated.

Also I noticed in some docs/refs that when people use k8s in HA and install Rancher, when they run “kubectl -n cattle-system get ingress” they actually see all the master’s IP’s in the acme ADDRESS. In my case I only see one, and it’s master01 could that be part of the issue?

I’m going to continue researching but if u have any ideas… would be greatly appreciated

5

I’ll try to setup a new cluster using RKE this time and try the all thing again and see if it works.
Even if there’s no answers here at least it serves as future reference.