Service disruption on scale-down

Rancher 1.x
1

I’m running a java app behind a rancher-load balancer and am seeing disturbing behavior on when I scale-down the application.

(1.2.0-pre2, 4 compute nodes)

docker-compose:

myapp:
  labels:
    io.rancher.container.pull_image: always
  image: myapp:poc2
myapp-lb:
  ports:
  - 8075:8080
  labels:
    io.rancher.scheduler.global: 'true'
  tty: true
  image: rancher/load-balancer-service
  links:
  - myapp:myapp
      stdin_open: true

rancher-compose:

myapp:
  scale: 5
  health_check:
    port: 8080
    interval: 2000
    recreate_on_quorum_strategy_config:
      quorum: 2
    initializing_timeout: 60000
    unhealthy_threshold: 3
    strategy: recreateOnQuorum
    request_line: GET "/myapp/monitor/verify" "HTTP/1.0"
    healthy_threshold: 2
    response_timeout: 2000
    reinitializing_timeout: 60000
myapp-lb:
  load_balancer_config:
    haproxy_config: {}
  health_check:
    port: 42
    interval: 2000
    unhealthy_threshold: 3
    healthy_threshold: 2
    response_timeout: 2000

When I decrement the scale of myapp from 5 to 4, a jmeter script I have running shows:
Throughput:

Screen Shot 2016-09-02 at 11.02.46 AM.png868×472 38.2 KB

Response time jumps up during that period too.

JMeter (10 threads, also seen w/100) is hitting a loadbalancer that spreads the requests across all of the myapp-lb instances
This is a pretty significant interruption on a single instance scaledown. Am I missing something about how to orchestrate this?

2

It looks like the load balancer isn’t told which containers are going to disappear before they actually disappear. If you look at your LB config, it does a healthcheck every 2s, and takes 2 failures to remove it from the pool. This means 4s of bad requests to remove the container from the pool which is what we see in your graph.

It could also be that a request was in process during the scaledown and returned with an error, your graph doesn’t show when it was scaled down or how long these requests are taking, so it’s impossible to tell if its an error in your application or an error in the loadbalancer implementation.