Service upgrade challenge

Archive Rancher 1.x
1

So I build a stack which handles 250 client accounts or 250 client subdomains like client1.myapp, client2.myapp and so on… We have 2 containers: web and mysql.

‘web’ is the one that gets all incoming HTTP requests.

When new docker image is pushed (which is always ‘web’), we upgrade the service by simple upgrade strategy.
The new container is started and once it does, we turn off the old container. During the start we migrate each database, and for 250 instances, it can take upto 5-10 minutes - and that is where my question lies.

During migration, since all instances will not be ready as we’re processing them 1 by 1. Is there a way to redirect some traffic to old container for the subdomain pending migration?

At this point I’m thinking: there must be something I don’t know or this could be architectural flaw in our stack. Whatever you think it is, feel free to address.

2

This is an interesting question that my implementation will certainly encounter, have you been able to determine the best way to handle incoming traffic and also quiescence?

Im just beginning my research on this topic and have yet to find anything that addresses this directly. Also havent found anything in the rancher docs that describes what does happen to incoming traffic during an upgrade.

3

I still haven’t found a good solution.

4

As described, not really. Only one running container can be bound to the port. If you add a load balancer though in front, then the web container can not publish any actual port. Then you can use “start before stop” on upgrade and the new containers will not go into the balancer rotation until the healthcheck is passing, which you can make not pass until you’re migration is done.