HAProxy Reverse Proxy in front of Rancher Server issue


Because this is my first post here, I owe a thanks, Rancher is a great product and I’ve enjoyed using it for the past few months.

I have been running Rancher public facing for a little over a month now but with only Local Auth as my protection I decided to put SSL in front to ensure that no snoops would grab my password over the air.

This is where the issues start, rancher/agent is no longer able to connect to my rancher/server.
Both server and agent run in virtual machines at my home where I’ve been testing Rancher. It’s been working fine until putting HAProxy in front.

Checking Rancher Agent logs I have noticed this error, which I suspect is the source of the issue:
ERROR root [140406802299792] [_logging.py:54] Received websocket error: [Handshake status 503]

503 is likely handed out by HAProxy if there is no matching backend server, I know WebSockets work because I can see the handshake in the browser when visiting the server UI.

HAProxy routes multiple domains by Host header after SSL termination, is there something I’m missing about how the agent talks to the server?

I am able to curl -i https://server-address/v1 and it returns the expected response. So I know the agent host is able to reach the server, its just the socket that is failing.

There is something different in implementation between how the Rancher UI and Rancher Agent handle WebSockets?

Edit 1
I started a node container and installed wscat (websocket cat), I am able to connect to the websocket from the rancher-agent host.

rancher/agent is still getting 503, does rancher/agent’s implementation of wss:// send a host header?

Edit 2
I allocated a unique port on HAProxy that does SSL and hands over directly to the rancher backend, without checking Host headers. This seems to have worked. rancher/agent is now showing up in my hosts. Though I wouldn’t consider this a fix

Cheers, Josh.