HAProxy + SSL issues

stormerider · January 3, 2017, 1:55am

I’ve got a generic config for my load balancer like so (notice that only 3 sites have HTTPS enabled inside Rancher at the moment): https://www.dropbox.com/s/wnpd1raectzx07r/Screenshot%202017-01-02%2015.45.00.png?dl=0

When I go and run a test script against http it works fine:

[nclemons@hawke docker-wordpress-nginx (master)]$ ~/bin/wptest http windsofstorm.com 80  
+ CURL='curl -kI'
+ curl -kI --header 'Host: windsofstorm.com' http://wolfsbane.windsofstorm.net:80
HTTP/1.1 301 Moved Permanently
Server: nginx/1.4.6 (Ubuntu)
Date: Tue, 03 Jan 2017 01:51:44 GMT
Content-Type: text/html; charset=UTF-8
X-Powered-By: PHP/5.5.9-1ubuntu4.20
Set-Cookie: wfvt_2412430881=586b03b032771; expires=Tue, 03-Jan-2017 02:21:44 GMT; Max-Age=1800; path=/; httponly
Location: https://windsofstorm.com/
Set-Cookie: asex2_wos2=10.42.181.10; path=/

+ curl -kI --header 'Host: windsofstorm.com' http://hemlock.windsofstorm.net:80
HTTP/1.1 301 Moved Permanently
Server: nginx/1.4.6 (Ubuntu)
Date: Tue, 03 Jan 2017 01:51:45 GMT
Content-Type: text/html; charset=UTF-8
X-Powered-By: PHP/5.5.9-1ubuntu4.20
Set-Cookie: wfvt_2412430881=586b03b1d52d9; expires=Tue, 03-Jan-2017 02:21:45 GMT; Max-Age=1800; path=/; httponly
Location: https://windsofstorm.com/
Set-Cookie: asex2_wos2=10.42.177.171; path=/

+ curl -kI --header 'Host: windsofstorm.com' http://nightshade.windsofstorm.net:80
HTTP/1.1 301 Moved Permanently
Server: nginx/1.4.6 (Ubuntu)
Date: Tue, 03 Jan 2017 01:51:48 GMT
Content-Type: text/html; charset=UTF-8
X-Powered-By: PHP/5.5.9-1ubuntu4.20
Set-Cookie: wfvt_2412430881=586b03b47827b; expires=Tue, 03-Jan-2017 02:21:48 GMT; Max-Age=1800; path=/; httponly
Location: https://windsofstorm.com/
Set-Cookie: asex2_wos2=10.42.13.111; path=/

[nclemons@hawke docker-wordpress-nginx (master)]$

When I try it against HTTPS, it fails (the curl command here drops the -I flag so that I can see what the actual error is):

[nclemons@hawke docker-wordpress-nginx (master)]$ ~/bin/wptest https windsofstorm.com 443
+ CURL='curl -k'
+ curl -k --header 'Host: windsofstorm.com' https://wolfsbane.windsofstorm.net:443
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx/1.4.6 (Ubuntu)</center>
</body>
</html>
+ curl -k --header 'Host: windsofstorm.com' https://hemlock.windsofstorm.net:443
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx/1.4.6 (Ubuntu)</center>
</body>
</html>
+ curl -k --header 'Host: windsofstorm.com' https://nightshade.windsofstorm.net:443
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx/1.4.6 (Ubuntu)</center>
</body>
</html>
[nclemons@hawke docker-wordpress-nginx (master)]$

Any thoughts as to what’s wrong here? Here’s the generated haproxy.cfg:

global
    chroot /var/lib/haproxy
    daemon
    group haproxy
    maxconn 4096
    maxpipes 1024
    ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
    ssl-default-bind-options no-sslv3 no-tlsv10
    ssl-default-server-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:D-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
    tune.ssl.default-dh-param 2048
    user haproxy

defaults
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http
    maxconn 4096
    mode tcp
    option forwardfor
    option http-server-close
    option redispatch
    retries 3
    timeout client 50000
    timeout connect 5000
    timeout server 50000
             
resolvers rancher
 nameserver dnsmasq 169.254.169.250:53

listen default
bind *:42

frontend 80
bind *:80
mode http
acl asex2_host hdr_end(host) -i accessiblesex.com
acl asex2_host hdr_end(host) -i accessiblesex.com:80
use_backend asex2 if asex2_host
acl alden1_host hdr_end(host) -i alden.nu
acl alden1_host hdr_end(host) -i alden.nu:80
use_backend alden1 if alden1_host
acl cia2_host hdr_end(host) -i cialeah.com
acl cia2_host hdr_end(host) -i cialeah.com:80
use_backend cia2 if cia2_host
acl dbr2_host hdr_end(host) -i davidbridger.com
acl dbr2_host hdr_end(host) -i davidbridger.com:80
use_backend dbr2 if dbr2_host
acl storm4_host hdr_end(host) -i jasblackthorne.com
acl storm4_host hdr_end(host) -i jasblackthorne.com:80
use_backend storm4 if storm4_host
acl storm6_host hdr_end(host) -i jasmeralia.com
acl storm6_host hdr_end(host) -i jasmeralia.com:80
use_backend storm6 if storm6_host
acl mpp2_host hdr_end(host) -i mypatronpress.com
acl mpp2_host hdr_end(host) -i mypatronpress.com:80
use_backend mpp2 if mpp2_host
acl nonny2_host hdr_end(host) -i nonnyblackthorne.com
acl nonny2_host hdr_end(host) -i nonnyblackthorne.com:80
use_backend nonny2 if nonny2_host
acl rd2_host hdr_end(host) -i romancedivas.com
acl rd2_host hdr_end(host) -i romancedivas.com:80
use_backend rd2 if rd2_host
acl rs2_host hdr_end(host) -i robert-sloan.com
acl rs2_host hdr_end(host) -i robert-sloan.com:80
use_backend rs2 if rs2_host
acl shend2_host hdr_end(host) -i shendilavri.com
acl shend2_host hdr_end(host) -i shendilavri.com:80
use_backend shend2 if shend2_host
acl storm2_host hdr_end(host) -i stormerider.com
acl storm2_host hdr_end(host) -i stormerider.com:80
use_backend storm2 if storm2_host
acl shend4_host hdr_end(host) -i twilightsdawn.com
acl shend4_host hdr_end(host) -i twilightsdawn.com:80
use_backend shend4 if shend4_host
acl vae2_host hdr_end(host) -i vaedrennan.com
acl vae2_host hdr_end(host) -i vaedrennan.com:80
use_backend vae2 if vae2_host
acl wos2_host hdr_end(host) -i windsofstorm.com
acl wos2_host hdr_end(host) -i windsofstorm.com:80
use_backend wos2 if wos2_host
frontend 443
bind *:443 ssl crt /etc/haproxy/certs/current
mode http
acl shend1_host hdr_end(host) -i shendilavri.com
acl shend1_host hdr_end(host) -i shendilavri.com:443
use_backend shend1 if shend1_host
acl vae1_host hdr_end(host) -i vaedrennan.com
acl vae1_host hdr_end(host) -i vaedrennan.com:443
use_backend vae1 if vae1_host
acl wos1_host hdr_end(host) -i windsofstorm.com
acl wos1_host hdr_end(host) -i windsofstorm.com:443
use_backend wos1 if wos1_host

backend asex2
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
    http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
    http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
    cookie asex2_asex2 insert
mode http
server 10.42.81.0 10.42.81.0:80  cookie 10.42.81.0
server 10.42.243.144 10.42.243.144:80  cookie 10.42.243.144
server 10.42.221.190 10.42.221.190:80  cookie 10.42.221.190

backend alden1
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
    http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
    http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
    cookie asex2_alden1 insert
mode http
server 10.42.98.28 10.42.98.28:80  cookie 10.42.98.28
server 10.42.53.220 10.42.53.220:80  cookie 10.42.53.220
server 10.42.194.181 10.42.194.181:80  cookie 10.42.194.181

backend cia2 
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
    http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
    http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
    cookie asex2_cia2 insert
mode http
server 10.42.82.97 10.42.82.97:80  cookie 10.42.82.97
server 10.42.176.200 10.42.176.200:80  cookie 10.42.176.200
server 10.42.109.212 10.42.109.212:80  cookie 10.42.109.212

backend dbr2
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
    http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
    http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
    cookie asex2_dbr2 insert
mode http
server 10.42.17.22 10.42.17.22:80  cookie 10.42.17.22
server 10.42.152.125 10.42.152.125:80  cookie 10.42.152.125
server 10.42.142.232 10.42.142.232:80  cookie 10.42.142.232
             
backend storm4
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
    http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
    http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
    cookie asex2_storm4 insert
mode http
server 10.42.23.116 10.42.23.116:80  cookie 10.42.23.116
server 10.42.168.50 10.42.168.50:80  cookie 10.42.168.50
server 10.42.134.90 10.42.134.90:80  cookie 10.42.134.90

backend storm6
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
    http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
    http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
    cookie asex2_storm6 insert
mode http
server 10.42.23.116 10.42.23.116:80  cookie 10.42.23.116
server 10.42.168.50 10.42.168.50:80  cookie 10.42.168.50
server 10.42.134.90 10.42.134.90:80  cookie 10.42.134.90

backend mpp2
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
    http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
    http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
    cookie asex2_mpp2 insert
mode http
server 10.42.74.58 10.42.74.58:80  cookie 10.42.74.58
server 10.42.145.167 10.42.145.167:80  cookie 10.42.145.167
server 10.42.125.224 10.42.125.224:80  cookie 10.42.125.224

backend nonny2
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
    http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
    http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
    cookie asex2_nonny2 insert
mode http
server 10.42.227.94 10.42.227.94:80  cookie 10.42.227.94
server 10.42.226.81 10.42.226.81:80  cookie 10.42.226.81
server 10.42.167.103 10.42.167.103:80  cookie 10.42.167.103

backend rd2
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
    http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
    http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
    cookie asex2_rd2 insert
mode http
server 10.42.23.91 10.42.23.91:80  cookie 10.42.23.91
server 10.42.182.75 10.42.182.75:80  cookie 10.42.182.75
server 10.42.15.154 10.42.15.154:80  cookie 10.42.15.154

backend rs2
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
    http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
    http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
    cookie asex2_rs2 insert
mode http
server 10.42.91.72 10.42.91.72:80  cookie 10.42.91.72
server 10.42.78.68 10.42.78.68:80  cookie 10.42.78.68
server 10.42.228.210 10.42.228.210:80  cookie 10.42.228.210

backend shend2
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
    http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
    http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
    cookie asex2_shend2 insert
mode http
server 10.42.30.95 10.42.30.95:80  cookie 10.42.30.95
server 10.42.252.62 10.42.252.62:80  cookie 10.42.252.62
server 10.42.250.33 10.42.250.33:80  cookie 10.42.250.33

backend storm2
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
    http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
    http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
    cookie asex2_storm2 insert
mode http    
server 10.42.23.116 10.42.23.116:80  cookie 10.42.23.116
server 10.42.168.50 10.42.168.50:80  cookie 10.42.168.50
server 10.42.134.90 10.42.134.90:80  cookie 10.42.134.90

backend shend4
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
    http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
    http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
    cookie asex2_shend4 insert
mode http
server 10.42.30.95 10.42.30.95:80  cookie 10.42.30.95
server 10.42.252.62 10.42.252.62:80  cookie 10.42.252.62
server 10.42.250.33 10.42.250.33:80  cookie 10.42.250.33

backend vae2
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
    http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
    http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
    cookie asex2_vae2 insert
mode http
server 10.42.4.148 10.42.4.148:80  cookie 10.42.4.148
server 10.42.217.173 10.42.217.173:80  cookie 10.42.217.173
server 10.42.208.140 10.42.208.140:80  cookie 10.42.208.140

backend wos2
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
    http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
    http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
    timeout check 2000
    cookie asex2_wos2 insert
mode http
server 10.42.181.10 10.42.181.10:80  check port 80 inter 2000 rise 2 fall 3 cookie 10.42.181.10
server 10.42.177.171 10.42.177.171:80  check port 80 inter 2000 rise 2 fall 3 cookie 10.42.177.171
server 10.42.13.111 10.42.13.111:80  check port 80 inter 2000 rise 2 fall 3 cookie 10.42.13.111

backend shend1
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
    http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
    http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
    cookie asex2_shend1 insert
mode http
server 10.42.30.95 10.42.30.95:443  cookie 10.42.30.95
server 10.42.252.62 10.42.252.62:443  cookie 10.42.252.62
server 10.42.250.33 10.42.250.33:443  cookie 10.42.250.33

backend vae1
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
    http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
    http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
    cookie asex2_vae1 insert
mode http
server 10.42.4.148 10.42.4.148:443  cookie 10.42.4.148
server 10.42.217.173 10.42.217.173:443  cookie 10.42.217.173
server 10.42.208.140 10.42.208.140:443  cookie 10.42.208.140

backend wos1 
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
    http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
    http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
    timeout check 2000
    cookie asex2_wos1 insert
mode http
server 10.42.181.10 10.42.181.10:443  check port 80 inter 2000 rise 2 fall 3 cookie 10.42.181.10
server 10.42.177.171 10.42.177.171:443  check port 80 inter 2000 rise 2 fall 3 cookie 10.42.177.171
server 10.42.13.111 10.42.13.111:443  check port 80 inter 2000 rise 2 fall 3 cookie 10.42.13.111

stormerider · January 3, 2017, 1:58am

Also, is there a way to configure HAProxy to offload HTTPS requests to an HTTP config, so that I don’t have to configure nginx to listen on both 80 and 443?

And why is my first backend (asex2) being chosen for the header of the cookie name (like asex2_vae1)? That seems strange…

yoram · January 3, 2017, 2:48pm

Up,
seems like the exact same issue I raised here

yoram · January 3, 2017, 4:06pm

Ok Mate just had answer for my issue.
HTTPS is only for SSL Termination.

Regarding your screenshot you should use SNI instead of HTTPS and it should work

vincent · January 3, 2017, 11:17pm

The error you’re getting is because a decrypted (plaintext) HTTP request is being sent to a HTTPS listener on the target container (on port 443). Which matches the generated config you posted (“wos1” targets say :443), but that config does not look like what the picture from the UI should have generated, as there are none which have a target port of 443. So either this is a bug in the config generation somehow, or that screenshot isn’t of the current/saved state of the balancer…

stormerider · January 21, 2017, 11:11pm

So after testing with the newest image, I found you have to redirect 443 -> 80, you can’t do 443 -> 443.

Topic		Replies	Views
SSL termination and wordpress Rancher 1.x	9	2336	January 31, 2017
Trouble setting up http-to-https redirection Rancher 1.x	4	5944	December 12, 2016
HAproxy Support HTTPS Rancher 1.x	3	1805	September 4, 2015
Load Balancer not allowing both HTTP and HTTPS rules Rancher 1.x	7	2628	April 21, 2017
Load Balancer with SSL Termination and x509 Client Certificates Rancher 1.x	1	3758	March 22, 2016

HAProxy + SSL issues

Related topics