Rancher HA pains

Archive Rancher 1.x
1

Hi,
this is my setup in AWS (I don’t want to use made cloudformation script):

DATABASE: AWS RDS MySQL
LB FRONT END: ELS TCP 443:8080 -> node02,node03

Node 1

  • REDIS Master
  • ZOOKEEPER node in cluster with other nodes
  • docker run -d --restart=always
    -e CATTLE_URL=https://rancher..com/v1
    -e CATTLE_ACCESS_KEY=    **** \
    -e CATTLE_SECRET_KEY=**************** \
    rancher/server go-machine-service

Node 2

  • REDIS slave
  • ZOOKEEPER node in cluster with other nodes
  • docker run -d --restart=always -p 8080:8080
    -e CATTLE_HOST_API_PROXY_MODE=“ha”
    -e CATTLE_HOST_API_PROXY_HOST=“rancher.*********.com:443”
    -e CATTLE_DB_CATTLE_MYSQL_HOST=rancherdb.*********.rds.amazonaws.com
    -e CATTLE_DB_CATTLE_MYSQL_PORT=3306
    -e CATTLE_DB_CATTLE_MYSQL_NAME=cattle
    -e CATTLE_DB_CATTLE_USERNAME=*********
    -e CATTLE_DB_CATTLE_PASSWORD=***************
    -e DEFAULT_CATTLE_MACHINE_EXECUTE=false
    -e CATTLE_ZOOKEEPER_CONNECTION_STRING=“node1,node2,node3”
    -e CATTLE_REDIS_HOSTS=“node1:6379,node2:6379,node3:6379”
    rancher/server:latest

Node 3

  • REDIS slave
  • ZOOKEEPER node in cluster with other nodes
  • docker run -d --restart=always -p 8080:8080
    -e CATTLE_HOST_API_PROXY_MODE=“ha”
    -e CATTLE_HOST_API_PROXY_HOST=“rancher.*********.com:443”
    -e CATTLE_DB_CATTLE_MYSQL_HOST=rancherdb.*********.rds.amazonaws.com
    -e CATTLE_DB_CATTLE_MYSQL_PORT=3306
    -e CATTLE_DB_CATTLE_MYSQL_NAME=cattle
    -e CATTLE_DB_CATTLE_USERNAME=*********
    -e CATTLE_DB_CATTLE_PASSWORD=***************
    -e DEFAULT_CATTLE_MACHINE_EXECUTE=false
    -e CATTLE_ZOOKEEPER_CONNECTION_STRING=“node1,node2,node3”
    -e CATTLE_REDIS_HOSTS=“node1:6379,node2:6379,node3:6379”
    rancher/server:latest

Problems:

  • web interface doesn’t work in https for XMLHttpRequest cannot load http://rancher.********.com/v1/userpreferences?limit=1000. Response to preflight request doesn’t pass access control check: No ‘Access-Control-Allow-Origin’ header is present on the requested resource. Now I’ve opened 80/tcp on ELS to allow to reach and view web interface but I want to fix it asap…

  • websocket-proxy - I don’t understand what it should do (I’ve read the Rancher HA page) and what it would proxy… please help…

A part for these problems, I’ve added hosts and deployed some stack (etcd, elasticsearch etc…) without problem.

Suggestion: please update Rancher HA page cause for me is totally unclear … I suggest to specify roles, layer (FE/BE/DB…) and how many instance for each.

Thanks for your help!

2
  1. Rancher does not support HTTPS natively, there is a section in the Rancher Docs regarding SSL termination, example NGINX config provided
    http://docs.rancher.com/rancher/installing-rancher/installing-server/basic-ssl-config/
    or

Running Rancher Server Behind an ELB in AWS with SSL
By default, ELB in HTTP/HTTPS mode does not support websockets. Since Rancher uses websockets, ELB must be configured specifically in order for Rancher’s websockets to work.

Configuration Requirements for ELB to enable Rancher:

Enabling proxy protocol mode
Configuring TLS/SSL for the frontend and TCP for the backend

  1. Depending on your authentication method i.e. via GitHub, the access url needs to be updated to reflect your security setup (HTTP/HTTPS) - explained when choosing authentication method

  2. Rancher Web-Proxy as per documentation:

NOTE:
The IP that you configure for the websocket proxy needs be accessible from your browser and from the hosts. The hosts make outbound connections to it and browsers connect directly to it as well. CATTLE_HOST_API_PROXY_HOST for rancher/server would need to have the public IP of the server running the websocket proxy and the websocket proxy’s -listen-address=ip:port param would need to be 0.0.0.0:.

NOTE:
In order to start websocket-proxy, Rancher server needs to be up and running. Therefore, Rancher server needs to be started with the CATTLE_HOST_API_PROXY_MODE and CATTLE_HOST_API_PROXY_HOST variables to be able to use websocket-proxy.
This is where all your others nodes connect to ie. proxy, single point , yes its not HA, but it is being worked on.

its appears complex, but its not really, and been assured HA mode is being worked on and will improve shortly.

3

Thanks for your kind response.

  1. About SSL and ELB, I’ve already touched that matter on that post AWS ELB WS problems but in this post I’ve reported that through https only I’ve got some XMLHttpRequest error cause part of web pages are refering to http address and not https - this cause the problem. So I guess to have some suggestion or conf to change for fix it and use ssl only version of rancher manager website.

  2. From what I’ve understood from documentation (and I accept that could be a personal limit of my brain) websocket-proxy is used as entrypoint for all request coming from public website and rancher node. If this correct it means that I’ve to use it in front of load balancer … PUBLIC > websocket-proxy > private load balancer > rancher server … is it right?

What I’m trying to understand is the entire layout of components for HA setup.

4

Have you figured this out yet?

I am having the issues with HTTPS. TCP 80-81 works fine, but SSL to TCP 81 or 444 result in the loop you have described.

5

@whiteadam This was regards to our previous HA setup which is completely different than what you are using today (in v1.0.1). I’d recommend opening a new question if possible.