What I would like to do is to change the external ports from 80 to 8080 and from 443 to 8443.
Why? Because there is already another cluster listening on those ports in same nginx configuration.
Something like this:
stream {
server {
listen 8080; # < 8080 instead of 80
proxy_pass rancher_servers_http; # Upstreams are 80
}
server {
listen 8443; # < 8443 instead of 443
proxy_pass rancher_servers_https; # Upstreams are 443
}
server {
listen 80;
proxy_pass cluster_servers_http;
}
server {
listen 443; # another cluster
proxy_pass cluster_servers_https;
}
}
There are no options for port in the helm chart because rancher isn’t the one listening… it’s just creating an ingress rule in the cluster it is being installed into, and the ingress controllers runs on whatever it has already been configured for.
The only challenge in LetsEncrypt that supports alternate ports is dns01, where you give it credentials to talk to your DNS provider and manipulate a record for the target domain.
(If you are going to use alternate ports, there’s no point exposing 80. It’s just for redirecting and nobody is going to seek it out to get redirected to the tls port)
Thanks. I wonder a bit because in the past I had to open port 8089 for the webhook of cert manager to
be able to get the certificates (rke provisioned cluster with certmanager and ha installation with helm 3).
I couldn’t figure out how to change this port to use same L4 LB for rancher-cluster and the user-cluster. For this reason I moved the tls termination to external (nginx with letsencrypt).
But I like the simplicity of using L4 load balancer and letting ingress do the tls job.
Certs are just for a domain (foo.com), not for a domain+port (foo.com:8443) combination. LE’s validation policy basically acknowledges that the primary/“most dangerous” use of certs are for web servers, and therefore proving that you control just some random port on a domain name is not good enough to give you a cert, which could then be (ab)used to impersonate the web server on 443.
So you must prove that you control 80 (the http-01 challenge), 443 (alpn-01), or the DNS records for the domain itself (dns-01).
8089 appears to be (or have been in the past) the port cert-manager’s http-01 solver pod internally listens on. You would then have an ingress rule to route requests (from port 80) to it.
Sure, port doesn’t play a role in certificate generation, only the domain.
I think I can’t use same LB for both clusters, right?
This is my main goal here. I want to avoid setting up a new HA loadbalancer or let the rancher cluster ingress care for the traffic to the user cluster.
My provider (hetzner.de) doesn’t provide any load balancing in cloud by now (only floating IPs). I use two nginx server in a ha installation (nginx, keepalived, letsencrypt) to route the traffic to the internal nodes.
At now tls is performed in nginx on the edge nodes. But I don’t like the solution because I have to store the certificates somewhere to share them between the master and backup nodes. A simple L4 loadbalancing moves this part to the ingress in cluster. Appears for me as a much cleaner solution.
@superseb recommended an authorized cluster endpoint for the user cluster but to be honest I didn’t get how this solves my problem. (See: Rancher HA and User-Cluster in LAN)
Ok, found a solution. I need only an additional floating IP for the rancher cluster to bind the streams to two different IPs. All other things are one liners in routing.
stream {
server {
listen IP1:443
...
}
server {
listen IP2:443
...
}
}
