What is the service hash / is it important & some other rancher-compose questions?

Rancher 1.x
1

Hi,

I have a couple of doubts regarding the rancher-compose file, and its requirements etc… I have noticed that when I create a stack from the rancher UI, I get a pretty simple docker-compose and rancher-compose file, with basically just the name, image, etc… However, when I use rancher-compose CLI to add a stack from a docker-compose file (without a rancher-compose file present), I am getting some other stuff added to both rancher-compose.yml and docker-compose.yml (when I click on View Config from the UI)…

I’d like to understand what they do, and whether I should keep some things when doing my own rancher-compose file…

the service hash and the metadata is what my question revolves mostly around…
rancher compose exerpt:

app:
  scale: 1
  metadata: &id001
    io.rancher.service.hash: 75eb960afaec0b728ce4dafa79170a41b5252dc3
datacontainer:
  scale: 1
  metadata: *id001

docker-compose exerpt:

app:
  ports:
  - 64174:80/tcp
  labels:
    io.rancher.sidekicks: datacontainer
    io.rancher.container.pull_image: always
    io.rancher.service.hash: e26441407c7c20211362fae3fede5d54b4061699
  image: app-a9c64d471210
  links:
  - php5:php5
  volumes_from:
  - datacontainer
datacontainer:
  labels:
    io.rancher.container.pull_image: always
    io.rancher.service.hash: 82c1360a7a9aea7481ff03761c79567cf1c8ad52
  command:
  - cat
  tty: true
  image: datacontainer-f90929ca57c4
  links:
  - php5:php5
  volumes:
  - /var/www
  stdin_open: true

In this example, the two containers are linked, actually datacontainer is a sidekick to app… Is this what the metadata &idxx and *idxx mean?

I also noticed that I dont have links in my datacontainer section of my original docker-compose file, but a link to another container which doesnt even use datacointainer was added when I submitted with rancher-compose…why? is it necessary/correct? (the datacontainer appears to “inherit” the links from the container it is a sidekick to)

Also, I notice that each container has a hash in docker-compose, but they dont correspond to the hash in the rancher-compose file… is this correct? Why are the hashes added/are they required?

Also, a bit unrelated, but is there a plan to allow editing of more options from a service? Or must we always clone the service in order to change things on the UI? (healthchecks, ports, advanced settings, scheduling options, etc)… is there a reason for this to be available at creation only?

–

I have also noted another issue… I’m trying to scale up a service that has a datacontainer and is linked from another service (basically its a php container, and its linked to the webserver container. I’m trying to scale up the php to 2 containers for 1 webserver…) When I try to scale up it creates an additional data-container, but doesnt create the php container itself and then move the datacontainer to sidekick… Is anyone else having this issue? Also, in my setup im currently testing with only 1 host, so in this case wouldnt it be better/possible to re-use the datacontainer?

EDIT: When obsrving the service try to scale up here is what is going on… it tries to start the container, but just gos to removed right away… I’m unable to pull logs for it…

Pasted image1526×501 64 KB
Also, from the host iself, a docker ps -a doesnt show any exited container… it shows 2 data containers but only 1 ‘service’ container…

For this sort of setup, is there any suggestion to how to place the containers/services? Ideally I want to use the containers from the same host, is there any way to accomplish this? I can put affinity rules to place containers on the same host (ex: php containers should co-exist with webserver containers), but how do I ensure that the routing is going to the local containers?

My idea is that I will have X webservers containers and Z php containers… I should be able to scale them independently from each other, and ideally should be able to use the closest possible php containers (i.e. from the same host if possible)… Is anything like this currently supported or planned? I think it would be a powerful usecase… I’d also in theory maybe want to prevent the linking from going to certain other hosts? (i.e. prefer local host, otherwise use another host in xxx datacenter, but dont use another datacenter as that would be too laggy - just an example…) Like I said I’m still having trouble scaling up the containers, but this will be my next question so I thought i’d already add it here…:stuck_out_tongue:

2

io.rancher.service.hash are used by rancher-compose to help determine which pieces changed when you run it again later.

YAML has “pointers” to reuse sections of the document. The metadata for both services are the same, so &id001 gives an “address” to that subtree and *id001 points to it to say “the same thing goes here too”.

Sidekicks automatically get the same links as the parent service.

Containers are immutable, so most things can’t be edited. If you’re doing clone & delete, you probably want the Upgrade action.

The PHP container is listening on a specific host port so only one of them can run per host. If you want more than one per host the container needs to not be bound to any host ports and use a load balancer (with scale <= $num_hosts) to distribute requests between them.

There is not any mechanism to prefer the same host (or higher-level construct) when talking across links at this time.

3

Hi @vincent… Thanks for the response, and thanks for clearing up the YAML “pointers”… So just in clearing up, the metadata pointers and service hashes wouldnt be something I’d normally include in my own YML files, right?

About the php scaling, I was actually not exposing php on a fixed port on the host, but did have the ports section in docker compose with the container port… in the UI it was attributing it a random high port so I didnt think that would be a conflict, however, when I did remove the ports defintion

ports: 
    - 9000

I am now able to scale it… I apologize, but I thought it was something with the sidekick… my bad…

One idea I had about using “local” services, would be to just point it to 127.0.0.1 and have a load balancer on that node… However, I’d need to somehow ensure that the “local” loadbalancer, only loadbalances to local containers… Is something like this supported somehow or might it be supported soon?

This is a great experimentation I’m having with rancher, however I think I’m probably most likely going to have a fixed number of php services per webserver, and run it that way… the loadbalancer would be most elegant, but it may just be too complex for this usecase… What do you think of this? Nonetheless, I’d still need a way of either knowing the IPs or ports of the multiple phps… I’m actually more confused as I write lol… Have you or anyone else used a similar setup or have any ideas of a good way of doing this?

Worst-case would be to create multiple services, but that doesnt sound right at all… I notice that if I continually ping from the webserver and have 2 containers running, I’m getting 2 ips in roundrobin, but this is being cached in the webserver and its only using one… Would there be a way of obtaining the current IPs of the containers from metadata and then I could dynamically insert these on the startup script? Another point might be to run the php containers as sidekicks, but then I still cant scale the sidekicks independently, so it wouldnt work…

Some vague ideas would include:
a) Loadbalancer on every host + php as sidekicks of the loadbalancer (to ensure they are on the same host), and use the host IP as the php upstream (which would point to the loadbalancer instead)… but sidekicks cant scale…
b) Loadbalancer on every host adding the local containers of a certain service as its backends, again I’d need to do this dynamically somehow by getting the local container ips…

Any help on a good way of planning this out would be appreciated :wink:

EDIT: one thing I found out is that if you have multiple instances of a server, and the DNS roundrobins, nginx will add all the IPs. So part of the problem can be solved by simply restarting the webserver after scaling up or down… I still havent found any way in my mind to keep the webserver talking only to services on the same host… I’m wondering if maybe some modification or plugin on the networkagent level, where you can address the container DNS with something like containername.local in order to resolve only to a container on the loal host or something…

4

I will add this here as its somewhat related. I found another issue discussing something similar (affinity rules), but it was locked so I cant comment there…

I added 2 services with 1 sidekick each, and a soft affinity rule to make sure they run on hosts “together” (at least 1 of each)… If I only have one host the stack works fine, but if I have 2 hosts, its sending the containers to separate hosts, and for some reason the containers cant ping each other… The IP is correct, but its not communicating…

This is the docker-compose, and I do see the affinity rules/labels in the UI of the containers, but its not respecting them…

I tried both hard and soft affinity with the same results…

nginx-static-files:
    build: nginx-static-files
    volumes:
        - /var/www
    labels:
        io.rancher.container.pull_image: always
    command: cat
    tty: true
    stdin_open: true
php-static-files:
    build: nginx-static-files
    volumes:
        - /var/www
    labels:
        io.rancher.container.pull_image: always
    command: cat
    tty: true
    stdin_open: true
php5:
    build: php5-fpm
    labels:
        neo.servicetype: php
        io.rancher.scheduler.affinity:container_label_soft: neo.servicetype=nginx
        io.rancher.sidekicks: php-static-files
        io.rancher.container.pull_image: always
    volumes_from:
        - php-static-files
nginxweb:
    build: nginx
    links:
        - php5
    labels:
        neo.servicetype: nginx
        io.rancher.sidekicks: nginx-static-files
        io.rancher.container.pull_image: always
    volumes_from:
        - nginx-static-files

** I also attempted using hard, and tried putting both rules in a “circular” fashion, and also on just one of the containers. All my attempts led to the containers being on separate hosts. / The file in viewconfig cntained the same affinity rules for the sidekicks, so I imagine rancher or rancher-compose is already adding those in there.

FYI: reducing the active hosts to 1 and restarting the stack makes it all go to the one (single) host, and the containers communicate normally.

EDIT: tried of 0.55.0rc3 and also on latest (v0.51.0) both exhibit the same

5

Affinity rules are working once sidekicks were removed…

I have just noticed that label names are “global”, so if you have services with the same name in 2 stacks, having a label on a container from stack A will satisfy affinity rules for stack B service… This may have unintended results if you dont plan accordingly. A “local” scope would be useful IMO (I’m exploring using 2 stacks for upgrading services as rolling updates and all other strategies are breaking our simple PHP-NGINX stack…)

EDIT: I figured out I can use $${stack_name} in the label… this should take care of this first part :slightly_smiling:

Another thing which is important is that stopped containers also satisfy affinity rules… This may or may not be desirable… In our case it was not…

I also noticed that scheduling is done based only on number of containers apparently, and not on “types”… Lets say I have 2 container flavors, Vanilla and Chocolate… A Chocolate container should go with (through soft affinity rule) a Vanilla container.

When scale = 1 for both containers, I get:
Host 1 - Vanilla + Chocolate
Host 2 - nothing

When scale = 2 for both containers, I get:
Host 1 - Vanilla + Chocolate
Host 2 - Vanilla + Chocolate

When scale = 3 for both containers, I get mixd results, but sometimes end up with:
Host 1 - Vanilla + Chocolate + Chocolate
Host 2 - Vanilla + Chocolate + Vanilla

It would make sense to take into account containers and spread them out not solely based on total number… as a way to “even out” containers which may consume more/less… not sure if this makes sense?

6

If you look strictly at the rule at the fact that your chocolate container needs to go on a host that has a vanilla container, it’s done exactly that. On both hosts, there are vanilla containers, so chocolate could go on either host.

I don’t think it makes sense to even out the containers. There is a request in Github to schedule based on host resources.

7

As for your scheduling issues, I was finally able to pinpoint why they weren’t coming up on the same host.

Since you have linked nginxweb to php, the order of service start will be php and then nginxweb. Therefore, there is no host that has the nginxweb service.

To fix it, you move the scheduler label to the nginxweb to look for the php service.

nginxweb:
   labels:
        io.rancher.scheduler.affinity:container_label: neo.servicetype=php
8

Hi @denise, thanks for your response!

I think we will have to agree to disagree about evening out containers… In a stateless, micro-service world (as is the case with most or many containers), I do believe that evening out containers makes sense, AS LONG AS resource/scheduling rules are “equal”… (i.e. I dont think it makes sense to even out containers solely based on number of containers, but if resources are equal, and scheduling rules are all met equally, it would make a lot of sense imo to even them out… its essentially a way of balancing them even to prevent irregular resource usage)… This strategy also improves high availability which is extremely important (maybe an objective even?) of stateless microservices.

Abt the scheduling issues I understand your suggestion and will probably use it for now, nonetheless, imho it isnt absolutely correct and may cause some other issues linked to containers piling up on a single host. I’ll make some tests with anti-affinity combined with this, as it may work as intended…

9

Denise, your example works, however I noticed that when I try to use variable such as stack_name/service_name, it doesnt work…

nginxweb:
    build: nginx
    links:
        - php5
#    ports:
#        - 8080:80
    labels:
        neo.servicetype: $${stack_name}/nginx
        io.rancher.container.pull_image: always
        io.rancher.scheduler.affinity:container_label: neo.servicetype=$${stack_name}/php
        io.rancher.scheduler.affinity:container_label_soft_ne: io.rancher.stack_service.name=$${stack_name}/$${service_name}
php5:
    build: php5-fpm
    labels:
        neo.servicetype: $${stack_name}/php

The above fails, not even bringing up any nginx containers, while if I remove the stackname and leave service names static, it works… Any ideas if this is a bug or if Im fudging it up? We need to run multiple stacks with the same name so stack_name is quite useful…

10

If you look at the service after you launched, you’ll notice that the label on the service is ${stack_name}/nginx.

We only support ${stack_name} in scheduling rules. It was changed to $${stack_name} due to the introduction of environment interpolation.

http://docs.rancher.com/rancher/rancher-compose/environment-interpolation/

You might benefit from environment interpolation. You could set an environment variable on the host that you’re running your rancher-compose, and use ${STACKNAME} in the docker-compose.yml and it would replace the stack name with your environment variable throughout your compose.

11

Hmm… Understood… I thought I could use stack_name in a variable, and then use that variable for scheduling… a bit confusing where it can and cant be applied, but now that you explain it I understand :wink:

I’m going to have a think about this as I was thinking of using different stacks for upgrading services, etc, adding env variables on the host could get confusing…

I’ll try to change my scheduling rules to just use rancher variables such as stack_service.name, without redefining a “custom” variable…

12

Yep, the stack_name variable is only referenced in the scheduling section in the docs and nowhere else. :slightly_smiling: