Trouble setting up http-to-https redirection

Rancher 1.x
1

I have the following simplified load balancer/container setup:

docker-compose.yml:

http: 
  ports: 
  - '443' 
  - '80' 
  labels: 
    io.rancher.loadbalancer.ssl.ports: '443' 
    io.rancher.loadbalancer.target.to-https: 80=80 
    io.rancher.loadbalancer.target.jenkins: ci.thewordnerd.info:443=8080 
  tty: true 
  image: rancher/load-balancer-service 
  links: 
  - jenkins:jenkins 
  - to-https:to-https 
  stdin_open: true 
jenkins: 
  image: jenkins:latest 
  volumes: 
  - /var/run/docker.sock:/var/run/docker.sock 
  - /home/jenkins:/var/jenkins_home 
to-https: 
  image: geldim/https-redirect:latest

rancher-compose.yml:

http: 
  scale: 1 
  certs: 
  - violetcrown.software 
  load_balancer_config: 
    haproxy_config: 
      defaults: '' 
      global: '' 
  default_cert: self-signed 
jenkins: 
  scale: 1 
to-https: 
  scale: 1

In summary, I’m trying to redirect all HTTP requests to HTTPS.

Many times this works. However, I run into a number of situations where hitting the http URL results in a 503 error from the Rancher load balancer. The HTTPS redirector uses Nginx so it’s fairly easy to see whether the request dies there or in Rancher. Sometimes the redirect works, while others it dies in Rancher’s HAProxy. to-https seems to stay up so shouldn’t ever become unavailable.

My HTTPS is fronted by Cloudflare but this behavior also appears when I hit the server IP directly. It also seems to happen somewhat randomly in that if I curl the server and manually set the Host: header, some requests will go through and others immediately following won’t. It’s almost as if there are 2 load balancers running on the port, and one doesn’t have the right configuration.

How can I either fix this, or provide additional debugging information? I’m about to take a Rancher-hosted app into production and it’d be nice to have this sorted.

Thanks.

2

Heh, forgot that I submitted the exact same questions a few months back.

Anyhow, I’ve already included my compose files. Here is the generated haproxy.cfg:

global
    log 127.0.0.1 local0
        log 127.0.0.1 local1 notice
        maxconn 4096
        maxpipes 1024
    tune.ssl.default-dh-param 2048
    ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
    ssl-default-server-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
    ssl-default-bind-options no-sslv3 no-tlsv10
    chroot /var/lib/haproxy
    user haproxy
    group haproxy
    daemon
    

defaults
    log    global
    mode    tcp
    option    tcplog
        option  dontlognull
        option  redispatch
        option http-server-close
        option forwardfor
        retries 3
        timeout connect 5000
        timeout client 50000
        timeout server 50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http
    

frontend b6971eed-936a-4209-818d-8171ae467f99_443_frontend
        bind 10.42.204.250:443 ssl crt /etc/haproxy/certs/
        mode http

                acl 8_host hdr(host) -i matrix.thewordnerd.info
                acl 8_host hdr(host) -i matrix.thewordnerd.info:443
            acl 8_path path_beg -i /_matrix
        use_backend b6971eed-936a-4209-818d-8171ae467f99_443_8_backend if 8_host 8_path
                acl 7_host hdr(host) -i syncthing.thewordnerd.info
                acl 7_host hdr(host) -i syncthing.thewordnerd.info:443
        use_backend b6971eed-936a-4209-818d-8171ae467f99_443_7_backend if 7_host 
                acl 6_host hdr(host) -i docker.violetcrown.software
                acl 6_host hdr(host) -i docker.violetcrown.software:443
        use_backend b6971eed-936a-4209-818d-8171ae467f99_443_6_backend if 6_host 
                acl 5_host hdr(host) -i matrix.thewordnerd.info
                acl 5_host hdr(host) -i matrix.thewordnerd.info:443
        use_backend b6971eed-936a-4209-818d-8171ae467f99_443_5_backend if 5_host 
                acl 4_host hdr(host) -i ci.thewordnerd.info
                acl 4_host hdr(host) -i ci.thewordnerd.info:443
        use_backend b6971eed-936a-4209-818d-8171ae467f99_443_4_backend if 4_host 
                acl 3_host hdr(host) -i dev.thewordnerd.info
                acl 3_host hdr(host) -i dev.thewordnerd.info:443
        use_backend b6971eed-936a-4209-818d-8171ae467f99_443_3_backend if 3_host 
                acl 2_host hdr(host) -i rundeck.thewordnerd.info
                acl 2_host hdr(host) -i rundeck.thewordnerd.info:443
        use_backend b6971eed-936a-4209-818d-8171ae467f99_443_2_backend if 2_host 
        default_backend b6971eed-936a-4209-818d-8171ae467f99_443_0_backend

backend b6971eed-936a-4209-818d-8171ae467f99_443_8_backend
        mode http
        server f05a8cdd-2258-485e-bf72-34b57ceb69a6 10.42.190.115:8008
        http-request add-header X-Forwarded-Proto https if { ssl_fc }
        
backend b6971eed-936a-4209-818d-8171ae467f99_443_7_backend
        mode http
        server a22ce2fd-9b29-4f35-ab7a-9a899785712e 10.42.19.179:8080
        http-request add-header X-Forwarded-Proto https if { ssl_fc }
        
backend b6971eed-936a-4209-818d-8171ae467f99_443_6_backend
        mode http
        server 970e1a1c-678c-459b-8c7a-3cff3399bff5 10.42.37.171:5000
        http-request add-header X-Forwarded-Proto https if { ssl_fc }
        
backend b6971eed-936a-4209-818d-8171ae467f99_443_5_backend
        mode http
        server f05a8cdd-2258-485e-bf72-34b57ceb69a6 10.42.190.115:8080
        http-request add-header X-Forwarded-Proto https if { ssl_fc }
        
backend b6971eed-936a-4209-818d-8171ae467f99_443_4_backend
        mode http
        server 00d1b04a-489c-474c-98f4-0db480ceaaef 10.42.226.228:8080
        http-request add-header X-Forwarded-Proto https if { ssl_fc }
        
backend b6971eed-936a-4209-818d-8171ae467f99_443_3_backend
        mode http
        server dee07bb8-2ce4-4702-be19-fa7614c454bc 10.42.75.65:8080
        http-request add-header X-Forwarded-Proto https if { ssl_fc }
        
backend b6971eed-936a-4209-818d-8171ae467f99_443_2_backend
        mode http
        server e057a796-29f5-4fa3-a57c-757aa201a11d 10.42.189.146:4440
        http-request add-header X-Forwarded-Proto https if { ssl_fc }
        
backend b6971eed-936a-4209-818d-8171ae467f99_443_0_backend
        mode http
        server 224f6b4a-1264-4683-9b5f-e3cdefe634ce 10.42.160.173:443
        http-request add-header X-Forwarded-Proto https if { ssl_fc }
        
frontend b6971eed-936a-4209-818d-8171ae467f99_80_frontend
        bind 10.42.204.250:80
        mode http

        default_backend b6971eed-936a-4209-818d-8171ae467f99_80_1_backend

backend b6971eed-936a-4209-818d-8171ae467f99_80_1_backend
        mode http
        server e057a796-29f5-4fa3-a57c-757aa201a11d 10.42.189.146:80
        server 970e1a1c-678c-459b-8c7a-3cff3399bff5 10.42.37.171:80
        server 00d1b04a-489c-474c-98f4-0db480ceaaef 10.42.226.228:80
        server 224f6b4a-1264-4683-9b5f-e3cdefe634ce 10.42.160.173:80
        server dee07bb8-2ce4-4702-be19-fa7614c454bc 10.42.75.65:80
        server f05a8cdd-2258-485e-bf72-34b57ceb69a6 10.42.190.115:80
        server a22ce2fd-9b29-4f35-ab7a-9a899785712e 10.42.19.179:80
        

listen default 
    bind 0.0.0.0:42
3

I should note that while my docker/rancher-compose files are simplified, haproxy.cfg isn’t. It is the full configuration for the entire LB.

It’s interesting, though, that I only have a single server listening on port 80, but the front-end for port 80 has a bunch of back-ends. All other services have a source port of 443, so I don’t know why they’re all included there. I’d expect only a single back-end server for the listener on 80.

4

Rancher doesn’t support multiple frontends directing to specific services. By default, any source port will access any target port.

A workaround if you want to use only 1 load balancer is that you could use hostname routing rules to create a bogus hostname route to prevent any services using a specific port.

See this issue for a specific example:

5

Do as described here: https://github.com/rancher/rancher/issues/3505#issuecomment-264308277

  1. Add selector rule in your balancer with protocol “HTTP”, request host “*”, port “80” and target “foo=bar” (actually any target that won’t select any service). It will force balancer to expose 80 port, but it won’t actually change haproxy.cfg, because there is no service with the target.
  2. In “Custom haproxy.cfg” add this
    frontend http-frontend
    bind *:80
    mode http
    redirect scheme https if !{ ssl_fc }
  3. Save balancer configs. All http requests will be redirected to https :wink: