Is the Rancher Documentation for Nginx Layer 4 LB Right?

Rancher
1

Regarding the documentation example here

https://ranchermanager.docs.rancher.com/v2.8/how-to-guides/new-user-guides/infrastructure-setup/nginx-load-balancer>

worker_processes 4;
worker_rlimit_nofile 40000;

events {
    worker_connections 8192;
}

stream {
    upstream rancher_servers_http {
        least_conn;
        server <IP_NODE_1>:80 max_fails=3 fail_timeout=5s;
        server <IP_NODE_2>:80 max_fails=3 fail_timeout=5s;
        server <IP_NODE_3>:80 max_fails=3 fail_timeout=5s;
    }
    server {
        listen 80;
        proxy_pass rancher_servers_http;
    }

}

http {

    upstream rancher_servers_https {
        least_conn;
        server <IP_NODE_1>:443 max_fails=3 fail_timeout=5s;
        server <IP_NODE_2>:443 max_fails=3 fail_timeout=5s;
        server <IP_NODE_3>:443 max_fails=3 fail_timeout=5s;
    }
    server {
        listen 443 ssl;
        ssl_certificate /path/to/tls.crt;
        ssl_certificate_key /path/to/key.key;
        location / {
            proxy_pass https://rancher_servers_https;
            proxy_set_header Host <rancher UI URL>;
            proxy_ssl_server_name on;
            proxy_ssl_name <rancher UI URL>
        }
    }
}

We have a test environment running with 2 nodes, k3s and all is up and working.

As the example had SSL termination on Nginx we did this to install rancher

helm install rancher rancher-latest/rancher
–namespace cattle-system
–version 2.8.0
–set hostname=lb1.example.uk
–set bootstrapPassword=admin
–set tls=external

But then we had an issue where we were trying to open a kubectl shell, but it would not.

1/ Check 1
Point DNS IP at Rancher directly < Works. Kubectl shell works, so issue is not with Rancher.

2/ Check 2
Point DNS IP at Loadbalancer < Rancher web appears to work, but on investigation with browser we can see stuff like this

Firefox can’t establish a connection to the server at wss://lb1.example.uk/k8s/clusters/local/v1/subscribe?sockId=33. index.8592d9dc.js:8:2439574

Now I think wss is across port 443, but the nginx is clearly set up to proxy https.

Is this a bug/doc error, or am I barking up the wrong tree?

Any suggestions to resolve?

1 Like
2

I am still testing, but at the moment this appears to have resolved from
https://www.serverlab.ca/tutorials/linux/web-servers-linux/how-to-proxy-wss-websockets-with-nginx/

And becomes

worker_processes 4;
worker_rlimit_nofile 40000;

events {
    worker_connections 8192;
}

stream {
    upstream rancher_servers_http {
        least_conn;
        server <IP_NODE_1>:80 max_fails=3 fail_timeout=5s;
        server <IP_NODE_2>:80 max_fails=3 fail_timeout=5s;
        server <IP_NODE_3>:80 max_fails=3 fail_timeout=5s;
    }
    server {
        listen 80;
        proxy_pass rancher_servers_http;
    }

}
http {
    upstream rancher_servers_https {
        least_conn;
        server <IP_NODE_1>:443 max_fails=3 fail_timeout=5s;
        server <IP_NODE_2>:443 max_fails=3 fail_timeout=5s;
        server <IP_NODE_3>:443 max_fails=3 fail_timeout=5s;
    }
    server {
        listen 443 ssl;
        ssl_certificate /path/to/tls.crt;
        ssl_certificate_key /path/to/key.key;
        location / {
            proxy_pass https://rancher_servers_https;
            proxy_set_header Host <rancher UI URL>;
            proxy_ssl_server_name on;
            proxy_ssl_name <rancher UI URL>
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }
    }
}

Note the addition of
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection “upgrade”;

3

Check also: Rancher 2 HA setup Error connecting to WebSocket - #2 by amioranza

1 Like