Deploy different nodes of an Hazelcast cluster in different hosts?

Rancher 1.x
1

Hello,

We are having a problem with container scheduling for a stack in our rancher environment (that has 4 hosts): our stack is composed by an hazelcast server container (deployed with scale=3) and a container running our service (deployed with scale=2 and using the hazelcast cluster). What happens sometimes is that rancher deploys the whole stack in the same host, which might result in the whole hazelcast cluster gone (therefore losing all its content) if the host would go down.

What ideally I would like to do is to instruct cattle to deploy preferably all the hazelcast instances on different hosts and all the service instances in different hosts, allowing a server instance and an hazelcast instance on the same host.

In older versions of rancher there was the ability to specify affinity rules but there is not anymore any mention of it in the documentation of the latest version, so I wonder is it still possible to specify affinity rules or something equivalent?

Thanks in advance!

2

@robyf we didn’t drop any scheduling rule functionality. Is this the documentation you’re looking for? http://docs.rancher.com/rancher/v1.4/en/cattle/scheduling/#example-1

3

@cjellick Thanks, I completely missed that one. I was looking under rancher-compose and not under rancher… noob mistake!

However I tried the affinity rules in my docker-compose and for some reason they are not working for me. For example I have this in my docker-compose (simplified for this example):

version: '2'
services:
  hazelcast:
    image: hazelcast:3.7.4
    labels:
      io.rancher.container.network: true
      io.rancher.scheduler.affinity:container_label_soft_ne: io.rancher.stack_service.name=keyvaluestore-service/hazelcast

but I still have all the 3 hazelcast containers (as it is deployed with scale=3) running in the same rancher host. And tip on what else can I try? Do I need to completely remove the stack and recreate it or are those labels taken into account just by updating a running stack?

Thanks in advance!

4

@robyf the ${} can stay. So, you can have exactly:

io.rancher.scheduler.affinity:container_label_soft_ne: io.rancher.stack_service.name=$${stack_name}/$${service_name}

Dollar signs and all. Can you try that?

5

@cjellick Now it worked, thanks. However it didn’t work when I updated the stack, I had to delete it and only then the affinity settings kicked in.