Cluster takes a few minutes to service requests when node goes down

Rancher 1.x
1

I’ve setup a simple three node elasticsearch cluster.

Here is the docker-compose file:

elasticsearch-masters:
  image: rancher/elasticsearch-conf:v0.5.0
  labels:
    io.rancher.container.hostname_override: container_name
    io.rancher.sidekicks: elasticsearch-base
    io.rancher.container.start_once: true
elasticsearch-base:
  labels:
    elasticsearch.config.version: '0'
    io.rancher.container.hostname_override: container_name
    io.rancher.container.start_once: true
  net: "container:elasticsearch-masters"
  volumes_from:
    - elasticsearch-masters
  volumes:
    - /usr/share/elasticsearch/data
    - /usr/share/elasticsearch/config
    - /usr/share/elasticsearch/config/scripts
  image: elasticsearch:2.4.3-alpine

search:
  environment:
    ELASTICSEARCH_HOST: es-clients
    ELASTICSEARCH_PORT: '9200'
  labels:
    io.rancher.container.pull_image: always
  tty: true
  image: test/api:1.0.186
  links:
  - elasticsearch-masters:es-clients
  stdin_open: true
  ports:
    - "9000:8080"

kopf:
  image: rancher/kopf:v0.4.0
  ports:
    - "80:80"
  environment:
    KOPF_SERVER_NAME: 'es.dev'
    KOPF_ES_SERVERS: 'es-clients:9200'
  labels:
    io.rancher.container.hostname_override: container_name
  links:
    - elasticsearch-masters:es-clients

rancher-compose:
elasticsearch-masters:
scale: 3
metadata:
elasticsearch:
yml:
cluster.name: "es"
node.name: "$${HOSTNAME}"
node.data: "true"
node.master: "true"
network.host: “site”

I have three questions.

  1. When I shut down a node, it takes a few minutes for the “search” container to connect to elastic. Is there anyway to speed up this process when a node is lost while the cluster is being reconfigured?

  2. I’m currently testing the resilience of the cluster by setting io.rancher.container.start_once: true so when I stop a node it won’t fire up again, this will obviously be removed when going live. Is there a better way to do this?

  3. I’m very new to rancher and docker this is my first setup, am I going about the right way with regard to setting up an es cluster as above or am I way off??

Thank you