Private Rancher EC2 instance behind AWS ELB with SSL termination

Rancher 1.x
1

Hi,

According to https://docs.rancher.com/rancher/v1.5/en/installing-rancher/installing-server/basic-ssl-config/#elb this should be supported in v1.5.3+, so I have setup Rancher server on EC2 Ubuntu-16.04:

# docker ps
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS                                                      NAMES
bbbcc14dd895        rancher/server:v1.5.3   "/usr/bin/entry --..."   2 hours ago         Up 2 hours          0.0.0.0:8080->8080/tcp, 3306/tcp, 0.0.0.0:9345->9345/tcp   upbeat_kalam

behind an ELB with SSL termination and ProxyProtocol enabled for ports 443 and 8080 as per the instructions:

            "ListenerDescriptions": [
                {
                    "Listener": {
                        "InstancePort": 8080, 
                        "SSLCertificateId": "arn:aws:iam::xxxxxxxxxxxx:server-certificate/cert", 
                        "LoadBalancerPort": 443, 
                        "Protocol": "SSL", 
                        "InstanceProtocol": "TCP"
                    }, 
                    "PolicyNames": [
                        "ELBSecurityPolicy-2016-08"
                    ]
                }
            ], 
            "HealthCheck": {
                "HealthyThreshold": 2, 
                "Interval": 10, 
                "Target": "HTTP:8080/ping", 
                "Timeout": 5, 
                "UnhealthyThreshold": 3
            }, 
            "VPCId": "vpc-xxxxxxxx", 
            "BackendServerDescriptions": [
                {
                    "InstancePort": 443, 
                    "PolicyNames": [
                        "TFEnableProxyProtocol"
                    ]
                }, 
                {
                    "InstancePort": 8080, 
                    "PolicyNames": [
                        "TFEnableProxyProtocol"
                    ]
                }
            ], 
            "Instances": [
                {
                    "InstanceId": "i-xxxxxxxxxxxxxxxxx"
                }
            ],

The problem is I can’t access the Rancher UI. The health checks work fine though and the instance shows as in service in the ELB, below is tcp dump on the instance port 8080:

# tcpdump -ni eth0 tcp port 8080
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

03:18:25.424389 IP 10.99.5.9.31174 > 10.99.5.155.8080: Flags [P.], seq 967946804:967946922, ack 2983485949, win 761, options [nop,nop,TS val 40009704 ecr 2031881], length 118: HTTP: GET /ping HTTP/1.1
03:18:25.425332 IP 10.99.5.155.8080 > 10.99.5.9.31174: Flags [P.], seq 1:190, ack 118, win 227, options [nop,nop,TS val 2034381 ecr 40009704], length 189: HTTP: HTTP/1.1 200 OK
03:18:25.425654 IP 10.99.5.9.31174 > 10.99.5.155.8080: Flags [.], ack 190, win 761, options [nop,nop,TS val 40009705 ecr 2034381], length 0
03:18:26.001231 IP 10.99.3.50.33976 > 10.99.5.155.8080: Flags [P.], seq 514864193:514864311, ack 3767786171, win 761, options [nop,nop,TS val 40009168 ecr 2032026], length 118: HTTP: GET /ping HTTP/1.1
03:18:26.002124 IP 10.99.5.155.8080 > 10.99.3.50.33976: Flags [P.], seq 1:190, ack 118, win 227, options [nop,nop,TS val 2034525 ecr 40009168], length 189: HTTP: HTTP/1.1 200 OK
03:18:26.002858 IP 10.99.3.50.33976 > 10.99.5.155.8080: Flags [.], ack 190, win 761, options [nop,nop,TS val 40009168 ecr 2034525], length 0

And locally the Rancher server seams responsive:

# curl -ksSNIL -I http://10.99.5.155:8080/v1
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: Mon, 27 Mar 2017 03:57:00 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Server: Jetty(9.2.11.v20150529)
Set-Cookie: PL=rancher;Path=/
X-Api-Account-Id: 1a1
X-Api-Client-Ip: 10.99.5.155
X-Api-Schemas: http://10.99.5.155:8080/v1/schemas
X-Api-User-Id: 1a1
X-Rancher-Version: v1.5.3

# curl -ksSNIL -I http://localhost:8080/v1
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: Mon, 27 Mar 2017 03:57:17 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Server: Jetty(9.2.11.v20150529)
Set-Cookie: PL=rancher;Path=/
X-Api-Account-Id: 1a1
X-Api-Client-Ip: 172.17.0.1
X-Api-Schemas: http://localhost:8080/v1/schemas
X-Api-User-Id: 1a1
X-Rancher-Version: v1.5.3

What can be the problem?

Thanks

2

Igorc,

I ran into this as well. I had originally created a AWS Classic ELB and it failed. I was seeing errors in the Rancher logs about websockets. AWS Application ELB, the newer flavor, uses Level 7 and works with websockets.

I’m don’t understand the JSON that you sent about the ELB. Unsure if this the Application or Classic AWS ELB.

Can you sent the logs from the rancher container, with your private information removed?

Loyal Bassett

3

Hi,

Just leaving a feedback, the issue was that I created the ELB with reference to the private subnets instead of public ones. Ones I corrected that all is working as expected.