HAProxy as ELB for Rancher High Availability (HA)

Rancher 1.x
1

Hi all

I managed to get a 3-node rancher setup to work after much handholding and about three restarts from fresh.

My main question now is concerning the ELB for rancher. As has been noted on other threads in the forum, the amount of documentation there is rather scarce. This is still a test setup and I’m using HAProxy as ELB for rancher HA. So far, I configured it with three frontends on ports 80, 443 and 18080, all TCP. I also configured three matching backends, also all TCP.

My main questions:

  1. How would I best set up HAProxy to do SSL termination? Would I have to set up a “http” frontend in HAproxy with still a TCP back end? And would I still use 443 at the back end?
  2. How do folks configure the health checks for rancher in HAProxy?
  3. Why do I have to expose port 80? What uses it?

In my current setup, I simply updated the “system-ssl” certificate within rancher to use my public cert. I am not yet sure about the consequences of that, but it seems to be doing the trick.

I found mention of specifically having to enable the PROXY protocol when using an external load balancer. I am not all to sure how this maps to HAProxy:
http://docs.rancher.com/rancher/v1.0/en/installing-rancher/installing-server/basic-ssl-config/#configuration-requirements-for-elb-to-enable-rancher

Once I have this HAProxy setup figured out, I’d love to contribute to the ELB-specific part of the multi-node setup document.

Thank you!

For what it’s worth, here a copy of my current haproxy.cfg file for rancher:

global
        log /dev/log local2
        maxconn 4096
        maxpipes 1024
        uid 99
        gid 99
        daemon

defaults
        log     global
        mode    tcp
        option  tcplog
        option  dontlognull
        option  redispatch
        option http-server-close
        retries 3
        timeout connect 5000
        timeout client 50000
        timeout server 50000

frontend rancher_80_frontend
    bind *:80
    mode http

    default_backend rancher_80_backend

frontend rancher_443_frontend
    bind *:443
    mode tcp

    default_backend rancher_443_backend

frontend rancher_18080_frontend
    bind *:18080
    mode http

    default_backend rancher_18080_backend

backend rancher_80_backend
        mode http
        server rancher_01 10.47.64.25:80
        server rancher_02 10.47.64.47:80
        server rancher_03 10.47.64.54:80

backend rancher_443_backend
        mode tcp
        server rancher_01 10.47.64.25:443 check
        server rancher_02 10.47.64.47:443 check
        server rancher_03 10.47.64.54:443 check

backend rancher_18080_backend
        mode http
        server rancher_01 10.47.64.25:18080
        server rancher_02 10.47.64.47:18080
        server rancher_03 10.47.64.54:18080
2

Hi @moensch !

Have you get some feedbacks about this question ?

We try to configure HAProxy for Rancher HA. I suppose we are facing the same questions as you. In fact, the bad configuration of the ssl certificate in HAProxy generate some troubles with websoket, and other features in Rancher.

If you have some news, would you like to share it ?

Cheers.

3

The only customization allowed in the current release is what you can put into the defaults or global section of the HAProxy config. https://github.com/rancher/rancher/issues/2179 will be included in 1.2.0 (and some pre-release before that, likely pre3) and will let you change any part of the config.

4

@vincent: The question here is not about LB setup within rancher and rancher-compose, but about how to configure an external load balancer for a rancher HA cluster.

Usually, this external load balancer would not be configured within Rancher itself, but outside of it.

The documentation is too scarce for the rancher-HA setup on the ELB configuration. I’d love to find out how folks configure an ELB for their rancher-HA clusters, and apparently @cclairmont has the same problem…

Unfortunately, whilst I do have a working setup, I am not at all convinced it is correct. WebSockets seem to work, but I have no explanation as to why they work. Also, I don’t know how to set up health checks on HAProxy for the different rancher backend services.

And having the certificate handled at the HAproxy layer would be much more preferable than the option I took now. I just replaced the “system” certificate within rancher from the self signed one to my public one.

And now I’m blatantly going to be cheeky and tag @denise in this. She’s been very helpful in the past and I hope if we get some traction, maybe we can all work together to figure out the best way to configure HAProxy as an ELB for rancher-ha, feed this information back to rancher, and help improve the rancher-ha documentation for everyone.

5

Hi all !

Me and some colleagues spent many time trying to install Rancher in HA mode last months !

Firts we started trying with CentOS 7 and met many difficulties, with devicemapper (we configured direct-lvm mode instead of loop-lvm for Docker). This configuration was not the good one. Perhaps, the kernel version was not appropriated for docker.

Finally we adopted Debian 8. We have a 3.14 kernel version. With this kernel we have some troubles with docker and devicemapper with direct-lvm mode. If we have to delete racher-ha-cattle we faced some kernel error like that:

unregister_netdevice: waiting for lo to become free. Usage count = -1

After that, we was not sure about how good was our installation. So we continue with Debian 8, but with the 3.16 kernel, and the Docker 1.10.3 ! This new installation uses AUFS. So we didn’t tried the devicemapper with direct-lvm.
In our installation we also faced a bug of the rancher HA installer script. We have to put our certificate in the script as it’s described here:

Loadbalancer

To ansewer to @moensch we use Nginx as a loadbalancer. And it’s work.

upstream rancher {
ip_hash;
server xxx.xxx.xxx.xxx;
server xxx.xxx.xxx.xxx;
server xxx.xxx.xxx.xxx;
}

server {

listen 80;
listen 443 ssl;
server_name <<sub domain>>.<<domain>>.<<tld>>;

ssl on;
ssl_certificate /etc/rancher/ssl/cert.pem;
ssl_certificate_key /etc/rancher/ssl/privkey.pem;
ssl_trusted_certificate /etc/rancher/ssl/chain.pem;

ssl_session_cache shared:SSL:20m;
ssl_session_timeout 10m;

ssl_prefer_server_ciphers       on;
ssl_protocols                   TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers                     ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS;

add_header Strict-Transport-Security "max-age=31536000";

access_log /var/log/nginx/access.log;
error_log  /var/log/nginx/error.log;

location / {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass http://rancher;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    # This allows the ability for the execute shell window to remain open for up to 15 minutes. Without this parameter, the default is 1 minute and will automatically close.
    proxy_read_timeout 900s;
}

}

I spent time, and time with Haproxy. I will investigate more on it. Now Rancher is working.

a new error

I face a last error when I try to add a host that will be manage by Rancher. It’s the same trouble as it’s described here:

Our certificate is generated with Let’s Encrypt.

Our stack:

  • Debian 8.5 (3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u3 (2016-07-02) x86_64 GNU/Linux)
  • Docker 1.10.3
  • Rancher Server 1.1.2
  • Rancher Agent-instance v0.8.3
  • Rancher Agent v1.0.2

Cheers

6

Hi,

We finaly found a solution for our Let’s Encrypt validation trouble. We explained it here in this other thread:

Cheers