I’ve been testing Rancher out for a few weeks now. I had been only using a single host for all the services that I had running under rancher. I just added a 2nd host, and shortly after adding it, Rancher immediately auto-migrated my stack to the new host without any kind of warning. Problem is, my stack has dedicated disks on the original host. Moreover, the service in now inaccessible because the firewall is still directing traffic to the original host.
Is this container/host auto migration considered expected behavior?
Yes, it is considered the expected behavior unless you did setup some scheduling rules to prevent that.
You have a couple of options to “glue” the service to the particular host:
(the most explicit one) in “Scheduling” section choose “Run all containers on a specific host” and simply select the host
(more flexible option) add labels to the hosts with dedicated disks, let’s say “disk_type=ssd”. You can do that while adding the host to the cluster or later on in “Infrastructure->Hosts->Edit”. While creating (or upgrading) the service, you can set scheduling like this: "The host must have a host label of disk_type = ssd.
Thanks for the quick reply. How did it also get the volumes migrated? I’ve been trying to trigger this behavior to move a stack to a new host, and the process is really unintuitive, which is to say, I have been unable to migrate the stack. Is there a documented process for moving the stack between hosts without recreating it and losing data?
It seems like, from a UX standpoint, if from the hosts screen you could drag/drop a complete stack from host column to host column, then have rancher handle the details of actually moving stuff around would be amazing. Since it already clearly has the capability to migrate between hosts, seems like it’s only a UI addition.
Rancher is supposed to create a cloud-like environment for you so you should not rely that much on the fact where/when/what is located. If you add two hosts to the environment and create service with scale=1, it means that this service is supposed to work all the time, no matter where (so if your one host is down, it should be automigrated to the other one). If you have dedicated hosts for some services (for example: faster disks, better CPU etc.), you should properly attach them labels so your Rancher instance will be more restricted. I cannot say I don’t like the idea of drag-and-drop in hosts menu but at the same time, I doubt people really need that in cloud-aware software. If you need to move something out of the host, there must be a reason:
wrong host = wrong scheduling rules (fix them)
host will be deleted soon -> use “Evacuate” from hosts menu
Talking about volumes: they are not automatically migrated. From my experience, you have a couple of options here:
use db software with proper HA setup so you don’t have to migrate it together with app containers
use some kind of persisted volume (on Cattle, you can achieve that by using volume plugins like Rancher NFS/Rancher EBS)
To sum up: it all depends on the software you want to deploy.
When I added that 2nd host, a wordpress stack was moved from my primary host to the new one. When I accessed the service at the new host’s IP address, all the data was still there. If I hadn’t had to use a new IP to access it, I may not have even noticed it was migrated. If the volumes don’t automatically migrate, how then did WP/MariaDB still function correctly? Was it accessing the volumes from the original host?
I am probably missing a Rancher key concept here, but how is persisted Cattle data managed in the UI? I can’t even see what persistent volumes are on which hosts. If the services all migrate around and persistent volumes aren’t moved with the stacks, I’m not sure how you would keep track of where you data actually is.
Say I need to evacuate a host. If the services are migrated to another host by rancher, but the volumes aren’t what just happened to my data?