I have a service which runs a Java application under Tomcat. The application takes about 30 seconds to start up and actually be available (I wish it was instant but it’s not!) The service is served via a Rancher HAProxy load balancer instance.
When I do a service upgrade, I select “start before stop” in the UI because I want my service to remain on-line. However, Rancher stops the old container before the new one is fully started - well the container is started but the Java application isn’t fully started.
I thought this was because I didn’t have a health check so I added one. But that didn’t resolve the issue.
Rancher won’t be able to tell the difference between the container starting and the applications inside your container starting. We just track that the container has started and is active.
Do you have only 1 container in your service or more than 1? You could try making it into a service of 2 containers and then upgrade and keep the batch size as 1 and increase the batch size interval to 45 sec. This might solve it, but not sure.
Another one is that you could try using rolling upgrade using rancher-compose. And use a wait that’s long enough?
Hum, I see. I was under the impression that with the Health Check, Rancher knows whether the service is up or not inside the container. The status of the container is not yet “Running”, but rather in “Initializing” state. When in initializing, I would think it should not yet stop the old container (in “start before stop” mode).
@etlweather I see your concern. Today we consider instance upgraded based on its state (running), not its health state. The least we should do - make this option configurable. I’m going to file an enhancement request for it