nolan
February 14, 2016, 7:34pm
1
I have the following simplified load balancer/container setup:
docker-compose.yml:
http:
ports:
- '443'
- '80'
labels:
io.rancher.loadbalancer.ssl.ports: '443'
io.rancher.loadbalancer.target.to-https: 80=80
io.rancher.loadbalancer.target.jenkins: ci.thewordnerd.info:443=8080
tty: true
image: rancher/load-balancer-service
links:
- jenkins:jenkins
- to-https:to-https
stdin_open: true
jenkins:
image: jenkins:latest
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /home/jenkins:/var/jenkins_home
to-https:
image: geldim/https-redirect:latest
rancher-compose.yml:
http:
scale: 1
certs:
- violetcrown.software
load_balancer_config:
haproxy_config:
defaults: ''
global: ''
default_cert: self-signed
jenkins:
scale: 1
to-https:
scale: 1
In summary, I’m trying to redirect all HTTP requests to HTTPS.
Many times this works. However, I run into a number of situations where hitting the http URL results in a 503 error from the Rancher load balancer. The HTTPS redirector uses Nginx so it’s fairly easy to see whether the request dies there or in Rancher. Sometimes the redirect works, while others it dies in Rancher’s HAProxy. to-https seems to stay up so shouldn’t ever become unavailable.
My HTTPS is fronted by Cloudflare but this behavior also appears when I hit the server IP directly. It also seems to happen somewhat randomly in that if I curl the server and manually set the Host: header, some requests will go through and others immediately following won’t. It’s almost as if there are 2 load balancers running on the port, and one doesn’t have the right configuration.
How can I either fix this, or provide additional debugging information? I’m about to take a Rancher-hosted app into production and it’d be nice to have this sorted.
Thanks.
nolan
February 14, 2016, 11:08pm
2
Heh, forgot that I submitted the exact same questions a few months back.
Anyhow, I’ve already included my compose files. Here is the generated haproxy.cfg:
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
maxpipes 1024
tune.ssl.default-dh-param 2048
ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
ssl-default-server-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
ssl-default-bind-options no-sslv3 no-tlsv10
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
defaults
log global
mode tcp
option tcplog
option dontlognull
option redispatch
option http-server-close
option forwardfor
retries 3
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
frontend b6971eed-936a-4209-818d-8171ae467f99_443_frontend
bind 10.42.204.250:443 ssl crt /etc/haproxy/certs/
mode http
acl 8_host hdr(host) -i matrix.thewordnerd.info
acl 8_host hdr(host) -i matrix.thewordnerd.info:443
acl 8_path path_beg -i /_matrix
use_backend b6971eed-936a-4209-818d-8171ae467f99_443_8_backend if 8_host 8_path
acl 7_host hdr(host) -i syncthing.thewordnerd.info
acl 7_host hdr(host) -i syncthing.thewordnerd.info:443
use_backend b6971eed-936a-4209-818d-8171ae467f99_443_7_backend if 7_host
acl 6_host hdr(host) -i docker.violetcrown.software
acl 6_host hdr(host) -i docker.violetcrown.software:443
use_backend b6971eed-936a-4209-818d-8171ae467f99_443_6_backend if 6_host
acl 5_host hdr(host) -i matrix.thewordnerd.info
acl 5_host hdr(host) -i matrix.thewordnerd.info:443
use_backend b6971eed-936a-4209-818d-8171ae467f99_443_5_backend if 5_host
acl 4_host hdr(host) -i ci.thewordnerd.info
acl 4_host hdr(host) -i ci.thewordnerd.info:443
use_backend b6971eed-936a-4209-818d-8171ae467f99_443_4_backend if 4_host
acl 3_host hdr(host) -i dev.thewordnerd.info
acl 3_host hdr(host) -i dev.thewordnerd.info:443
use_backend b6971eed-936a-4209-818d-8171ae467f99_443_3_backend if 3_host
acl 2_host hdr(host) -i rundeck.thewordnerd.info
acl 2_host hdr(host) -i rundeck.thewordnerd.info:443
use_backend b6971eed-936a-4209-818d-8171ae467f99_443_2_backend if 2_host
default_backend b6971eed-936a-4209-818d-8171ae467f99_443_0_backend
backend b6971eed-936a-4209-818d-8171ae467f99_443_8_backend
mode http
server f05a8cdd-2258-485e-bf72-34b57ceb69a6 10.42.190.115:8008
http-request add-header X-Forwarded-Proto https if { ssl_fc }
backend b6971eed-936a-4209-818d-8171ae467f99_443_7_backend
mode http
server a22ce2fd-9b29-4f35-ab7a-9a899785712e 10.42.19.179:8080
http-request add-header X-Forwarded-Proto https if { ssl_fc }
backend b6971eed-936a-4209-818d-8171ae467f99_443_6_backend
mode http
server 970e1a1c-678c-459b-8c7a-3cff3399bff5 10.42.37.171:5000
http-request add-header X-Forwarded-Proto https if { ssl_fc }
backend b6971eed-936a-4209-818d-8171ae467f99_443_5_backend
mode http
server f05a8cdd-2258-485e-bf72-34b57ceb69a6 10.42.190.115:8080
http-request add-header X-Forwarded-Proto https if { ssl_fc }
backend b6971eed-936a-4209-818d-8171ae467f99_443_4_backend
mode http
server 00d1b04a-489c-474c-98f4-0db480ceaaef 10.42.226.228:8080
http-request add-header X-Forwarded-Proto https if { ssl_fc }
backend b6971eed-936a-4209-818d-8171ae467f99_443_3_backend
mode http
server dee07bb8-2ce4-4702-be19-fa7614c454bc 10.42.75.65:8080
http-request add-header X-Forwarded-Proto https if { ssl_fc }
backend b6971eed-936a-4209-818d-8171ae467f99_443_2_backend
mode http
server e057a796-29f5-4fa3-a57c-757aa201a11d 10.42.189.146:4440
http-request add-header X-Forwarded-Proto https if { ssl_fc }
backend b6971eed-936a-4209-818d-8171ae467f99_443_0_backend
mode http
server 224f6b4a-1264-4683-9b5f-e3cdefe634ce 10.42.160.173:443
http-request add-header X-Forwarded-Proto https if { ssl_fc }
frontend b6971eed-936a-4209-818d-8171ae467f99_80_frontend
bind 10.42.204.250:80
mode http
default_backend b6971eed-936a-4209-818d-8171ae467f99_80_1_backend
backend b6971eed-936a-4209-818d-8171ae467f99_80_1_backend
mode http
server e057a796-29f5-4fa3-a57c-757aa201a11d 10.42.189.146:80
server 970e1a1c-678c-459b-8c7a-3cff3399bff5 10.42.37.171:80
server 00d1b04a-489c-474c-98f4-0db480ceaaef 10.42.226.228:80
server 224f6b4a-1264-4683-9b5f-e3cdefe634ce 10.42.160.173:80
server dee07bb8-2ce4-4702-be19-fa7614c454bc 10.42.75.65:80
server f05a8cdd-2258-485e-bf72-34b57ceb69a6 10.42.190.115:80
server a22ce2fd-9b29-4f35-ab7a-9a899785712e 10.42.19.179:80
listen default
bind 0.0.0.0:42
nolan
February 14, 2016, 11:29pm
3
I should note that while my docker/rancher-compose files are simplified, haproxy.cfg isn’t. It is the full configuration for the entire LB.
It’s interesting, though, that I only have a single server listening on port 80, but the front-end for port 80 has a bunch of back-ends. All other services have a source port of 443, so I don’t know why they’re all included there. I’d expect only a single back-end server for the listener on 80.
denise
March 1, 2016, 10:24pm
4
Rancher doesn’t support multiple frontends directing to specific services. By default, any source port will access any target port.
A workaround if you want to use only 1 load balancer is that you could use hostname routing rules to create a bogus hostname route to prevent any services using a specific port.
See this issue for a specific example:
rsilva4
December 12, 2016, 3:28pm
5
Do as described here: https://github.com/rancher/rancher/issues/3505#issuecomment-264308277
Add selector rule in your balancer with protocol “HTTP”, request host “*”, port “80” and target “foo=bar” (actually any target that won’t select any service). It will force balancer to expose 80 port, but it won’t actually change haproxy.cfg, because there is no service with the target.
In “Custom haproxy.cfg” add this
frontend http-frontend
bind *:80
mode http
redirect scheme https if !{ ssl_fc }
Save balancer configs. All http requests will be redirected to https