How do you setup a cluster?

Rancher 1.x
1

Hello

I’m trying to understand how, or if it’s possible, to setup a cluster of containers where the application requires that the 2nd+ instances “join” the first. For example, RethinkDB, you start the first instance normally, than all subsequent instances “join” the ip/port of the first.

Is there a way to do this with rancher and still have the ability to Scale Up dynamically?

I also know RabbitMQ follows a similar pattern.

2

Maybe define your stack using one service for a single-instance Master, and a second service for the multiple-instances of the additional containers? Use rules to ensure that your master always starts on a specific host/IP. Or, use an external F5 load balancer to find your master and point all minions at the F5 IP. Or, it might be possible to configure the minion service to have a connection to the master service and connect that way, though I don’t quite know how you define it.

3

This kind of sounds like sidekicks, where services are joined together with other services. Though I"m not sure how they would “join” the ip/port of the first. Sidekicks scale up together so if you scale up the primary configuration all secondary configurations (aka sidekicks) are scaled up as well.

http://docs.rancher.com/rancher/rancher-compose/#sidekicks

Sidekicks are available through rancher-compose or through the UI (As of the latest version, v0.43.1).

4

I think I have a possible solution, sharing here for validation/input and in case it helps someone else in the future:

This should apply equally for both RethinkDB and RabbitMQ clusters which follow the pattern of:

  • First node that starts is “primary”, is started with a normal command
  • All other nodes start as “secondary” and are started with a “–join” (or whatever) parameter which specifies the primary’s address

My solution:
Create a docker image that has a start script. This script first uses the Meta-Data services from Rancher and looks up its “Service” entry. That result will tell it if there are any other containers in the service. If there aren’t, then run command as primary.

If there are other containers get all their names and generate a Secondary command.

With RethinkDB, it will allow you to specify multiple --join parameters and the first that works will take. The generated command may look something like:
rethinkdb --some option --join db1 --join db2 --join db5
(assuming db1, db2, and db5 are container names returned from the Services Meta-Data call)

5

I created a Docker image for using clustered RabbitMQ containers within a Rancher service. It uses Rancher’s Metadata Service to gather scale size, container names and stack name. Then it does some specific RabbitMQ stuff to allow an container to be added, restarted and added back after it was deleted (or crashed).

You can find it here: https://hub.docker.com/r/nuarch/rancher-rabbitmq

Source is at: https://github.com/dsvmacdonald/rancher-rabbitmq

6

Thanks @dsvmacdonald — that looks like exactly what I need! I"m going to give this a go in the next day or two.

7

Were you able to get ‘rancher-rabbitmq’ to start properly?
As soon as the first container starts, it fails and immediately goes to ‘removed’.
I used the example docker-compose and rancher-compose yaml files since I’m just testing clustering in general with Rancher.