Start stacks when hosts are recyled or newly registered

Rancher 1.x
1

Rancher v1.3.2
Platform : AWS

We use Terraform to create the infrastructure for Rancher. When we create a new environment (using the Rancher provider in Terraform) we also create a number of core stacks which we want to run in every env. We use the rancher_stack resource for that and it works nicely EXCEPT …

When you are standing up a brand new environment it takes a while for the hosts (in this case EC2 instances) to come up and register themselves. The stacks, both my user stacks and the 4 infrastructure stacks are sitting their in an unhealthy state as you would expect, and when the hosts register themselves the infrastructure stacks all automatically start. But my user stacks dont.

I can start them manually via the UI and all is fine (so I know theres nothing wrong with the stacks themselves), but in the world that I live in, pretty much everything needs to fully automated, so doing this manually is a non starter

I could possibly write something to detect when the hosts are registered and only launch the stacks at that point, but, since the infrastructure stacks have the very behaviour I need, I hoped that I could configure my user stacks similarly. But I can’t see anything (like a label) that I can use for that purpose that isn’t there already.

Anyone know if this is possible ?

I note that @vincent has said this in previous posts that touch on this topic but not sure if this is the reason why …

The restarting of containers created by services is managed by the rancher/server, for various reasons like ensuring that the network agent and related services are up before the container is started.

Regards

Fraser.

2

I’ve never used the Terraform integration but what state are the [user] stacks in after being created (click View in API action on one)? It almost sounds like startOnCreate is just off on them.

That quote refers to something unrelated… When you reboot a host we don’t set `–restart=always‘ on the existing actual docker containers because the orchestration system decides if/when to start them when they host comes back. So immediately after reboot none of the old containers on a host are running except the Rancher agent itself.

3

Hey @vincent , you were of course correct. I had neglected to set the start_on_create property on the rancher_stack resource, and when I did, the users stacks sat patiently waiting for the hosts to register themselves, and when they did, started successfully.

We like using terraform as our automation for Rancher, however we note that when spinning up a new environment (including re-cycling all the hosts from one that already exists) because those hosts come available at different times you often end up with a imbalanced distribution of containers across them. There is a Rancher issue that has been raised for this which talks to how you can re-balance the distribution, and we are very much looking forward to that being included in a minor release sometime soon.

Thanks for your help

Fraser.