Catalog port mapping is not the same as rancher-compose CLI

Archive Rancher 1.x
1

Team,

I have an app that I’ve deployed using the rancher-compose CLI and the stack is defined in a docker-compose.yml and rancher-compos.yml. I even have automated upgrades of this app plugged in to our Jenkins pipeline…so it works well using rancher-compose and the Rancher API. This stack runs 2 services and 2 load balancers on 2 VM hosts, so pretty standard, and simple stuff.

So I thought I’d add a catalog entry for this stack…just for fun.

The problem I’m seeing is that the load balancers don’t come up properly. The first to come up seems to listen on a random port, and the other never comes up…I assume because it can’t communicate with first one.

My original docker-compose.yml had the ports: set to - 6666, and it worked. When changed it to - “6666:6666” the first load balancer to come up listened on port 6666 properly, but the second one still won’t come up. It’s not trying to run on the same host, so there is not a port conflict. And there are no logs from the failing load balancer, because it’s not getting that far.

What am I missing? What can I look at to get a clue?

2

Can you post the compose files, or a dummy variant of them that shows the same problem?

The obvious thing would be that scale is too big for both LBs to use 6666 or there is an unrelated container on a host that is unexpectedly using it. The rancher/server log has info about the attempted scheduling and which requirement is not met.

The UI to launch a catalog item literally ends up calling rancher-compose to create the services.

3

There is no other container trying to use port 6666 on the host in question. There is another container listening on port 6666 privately. But the same condition exists on the other VM host, so I don’t believe there is a port conflict.

docker-compose.yml

===============================================================

myapp:
name: myapp
image: registry.cisco.com/myapp
command: myapp.py --rancher_server ${RANCHER_URL}
labels:
io.rancher.scheduler.affinity:container_label_soft_ne:
io.rancher.stack_service.name=$${stack_name}/$${service_name}
io.rancher.container.pull_image: always
io.rancher.scheduler.affinity:host_label: ${HOST_LABEL}

lb:
image: rancher/load-balancer-service
ports:

Listen on public port 6666 and direct traffic to private port 6666 of the service

  • "6666:6666"
    labels:
    io.rancher.scheduler.affinity:host_label: ${HOST_LABEL}
    links:

Target services in the same stack will be listed as a link

  • myapp:myapp

rancher-compose.yml

===============================================================

myapp:
scale: 2
health_check:
port: 6666
# Interval is measured in milliseconds
interval: 13333
unhealthy_threshold: 3
request_line: GET / HTTP/1.0
healthy_threshold: 2
# Response timeout is measured in milliseconds
response_timeout: 3333

lb:
scale: 2
health_check:
port: 42
interval: 3333
unhealthy_threshold: 3
healthy_threshold: 2
response_timeout: 3333

Nothing in the rancher/server logs that looks suspicious.

4

With the introduction of random port mapping, anytime that you want the same public port mapped to your private container port, you’ll need to use the following format

service:
  ports:
    80:80

As you have discovered, without both ports specified, Rancher is assuming you want it to assign the public port for you. I have updated the docs to reflect the change.

I didn’t have any issues trying to use the same docker-compose.yml and rancher-compose.yml, though I dummied it down to this

docker-compose.yml 
lb:
  ports:
  - 6666:6666
  image: rancher/load-balancer-service
  links:
  - myapp:myapp
myapp:
  labels:
    io.rancher.scheduler.affinity:container_label_soft_ne: io.rancher.stack_service.name=$${stack_name}/$${service_name}
  image: nginx

and

lb:
  scale: 2
  health_check:
    port: 42
    interval: 2000
    unhealthy_threshold: 3
    healthy_threshold: 2
    response_timeout: 2000
myapp:
  scale: 2

Note: I noticed tha tyou tried to change the interval/response_timeout of the load balancer. At this time, we don’t allow users to change the health check settings of a load balancer. I’ve added an enhancement for it.

When launching your files, can you look in the UI and click on the service to see if there are any errored containers being spun up or is it just stuck there? If there was a port conflict, there would be an errored out container with the message “Failed to find placement”.

5

Thanks for the response. I’ll wait for the next updater and see how things work. In the meetup it was mentioned that the network code in Rancher has changed, so maybe my stack will work better with the new release.