Scaling a complete stack (keeping ratios)

Rancher 1.x
1

I finally got the time to build out some stacks on Rancher (instead of just playing with dummy stacks) and am very pleased with the result.

I am hit with one hurdle though, which is dictated by the software design which I cannot easily change.

Background:
I have a stack consisting of an event bus (pub/sub) and multiple subscribers. Each event bus must have one, and only one of each type of subscribers. Having two subscribers of the same type on the same event bus would cause data duplication, and having none of a given type would cause data loss.

I see that, within my stack, I can scale each service independently, but this would not provide the desired result. The subscribers use Rancher’s DNS to find their event bus, and I cannot guarantee which one they are connected to.

So far, I only found an option to guarantee the “no more than one sub per event bus” requirement:
Instead of using the “eventbus” hostname, I would:
SVC_INDEX=$(curl -s http://rancher-metadata/2015-12-19/self/container/service_index)
EVENT_BUS=$(curl -s http://rancher-metadata/2015-12-19/self/stack/services/eventbus/containers/$SVC_INDEX/primary_ip)

And then make my subscribers use $EVENT_BUS to connect to the event bus, instead of the “eventbus” DNS name.

However, the problem I still have is that I cannot easily scale the whole stack. If I have more subscribers than event bus, that curl for EVENT_BUS above would not return anything - which I can catch and error on, but not desirable. I’d love to be able to just turn the volume on a complete stack to eleven, and not just on a per-service level.

2

Ah! And now I just read about affinity.

In theory, the below could work?

Assuming my stack is called “events”, my subscribers are “sub1” and “sub2”, my event bus is “eventbus”:

docker-compose.yml (incomplete):

eventbus:
  image: something
sub1:
  image: something1
  labels:
    io.rancher.scheduler.affinity:container_label: io.rancher.stack_service.name=$${stack_name}/eventbus
    io.rancher.scheduler.affinity:container_label_ne: io.rancher.stack_service.name=$${stack_name}/$${service_name}
sub2:
  image: something2
  labels:
    io.rancher.scheduler.affinity:container_label: io.rancher.stack_service.name=$${stack_name}/eventbus
    io.rancher.scheduler.affinity:container_label_ne: io.rancher.stack_service.name=$${stack_name}/$${service_name}

The first label would guarantee that my subscriber is on the same host as my eventbus, and the second guarantees there are never two containers of the same service on the same host.

3

Yes, I believe the scheduling rules should handle what you are looking to do. There is no concept of scaling a whole stack though.

Another option would be to use sidekicks. You could create the event bus as a primary service and have the subscribers as sidekick services. Typically, sidekicks are used so that all the sidekicks would be scheduled on the same host as the primary service as one launch configuration onto a host.

http://docs.rancher.com/rancher/rancher-compose/#sidekicks

You could then add a scheduling rule to say that you don’t want any event buses scheduled on the same host and you’d be able to scale that 1 service up each time and all subscribers would be launched as well.

eventbus:
  image: something
  labels:
    io.rancher.sidekicks: sub1, sub2
    io.rancher.scheduler.affinity:container_label_ne: io.rancher.stack_service.name=$${stack_name}/$${service_name}
sub1:
  image: something1
 sub2:
  image: something2
4

Ah perfect - I didn’t even think about using sidekicks for this. I will give this a try, thank you.

How do sidekicks address the main instance they are connected to? I assume the “eventbus..rancher.internal” DNS name would still resolve to multiple A records, depending on scale? How does a side kick find their main service instance?

I don’t want to go too far off topic, but how do I upgrade a sidekick container? In the UI, I seemingly only have the option of upgrading a main container, but not a sidekick independently.

5

Here’e the docs on how the internal DNS works with sidekicks.

http://docs.rancher.com/rancher/rancher-services/internal-dns-service/#pinging-sidekick-services

You should be able to update just a sidekick. In the UI, you will find the Upgrade button on the primary service, but inside the upgrade option, you have the ability to select only a sidekick to upgrade.

Screen Shot 2016-04-19 at 11.15.33 AM.png791×129 13.2 KB

6

Thank you denise - this already helped me a ton!

The link you posted is helpful too, but I’m missing information on pinging the other direction (from sidekick to main service).

I notice that I can ping “eventbus” and reliably get the IP of the associated main service. What seems nice too is that I no longer get DNS round robin - so if I scale the stack up, if I ping “eventbus” from sidekick_2, I get the IP for eventbus_2 reliably, and never the one for eventbus_1, which is exactly what I want.

I don’t have `dig’ inside my minimal alpine containers to verify, but I assume my assumptions are correct?

7

I retract my earlier statement - there is still DNS round robin in place.

Must find myself a way of reliably getting the “sidekick-owner” from the sidekick. Maybe back to using rancher-metadata and create_index?
Though I think there is no guarantee that a sidekick will always have the same create_index as its owner - as I can upgrade them independently.

8

So far, this is what I came up with to find the IP of the primary service within a stack when run from a sidekick container (error handling removed for brevity):

#!/bin/bash
EC_SERVICE="eventbus"
PREFIX="rancher-metadata/2015-12-19/self"

SIDEKICK_SERVICE_INDEX=$(curl -s --fail $PREFIX/container/service_index)

SERVICE_INSTANCES=$(curl -s --fail $PREFIX/stack/services/$EC_SERVICE/containers)

for CONTAINER in $SERVICE_INSTANCES; do
  CONT_NUMBER=$(echo $CONTAINER| cut -d'=' -f 1)
  INSTANCE_SVC_INDEX=$(curl -s --fail $PREFIX/stack/services/$EC_SERVICE/containers/$CONT_NUMBER/service_index)

  if [ "$INSTANCE_SVC_INDEX" -eq "$SIDEKICK_SERVICE_INDEX" ]; then
    echo "Found eventbus service instance with same service index"
    SERVICE_IP=$(curl -s --fail $PREFIX/stack/services/$EC_SERVICE/containers/$CONT_NUMBER/primary_ip)
    export SERVICE_IP
    break
  fi
done

Would be awesome for this to be either accessible via DNS, or via rancher-metadata on a sidekick container (Maybe a property on the container called “parent”, referencing the main service container object?

So I could just do:

curl rancher-metadata/2015-12-19/self/parent/primary_ip

That would be awesome.

9

I tried looking for an issue in Github as this doesnt’ exist today, but was unable to find a duplicate. Please feel free to create a new issue.

10

Done: https://github.com/rancher/rancher/issues/4505