Creating thousands of container requests on Rancher

1

Hello,
I wanted to find out if this is valid usage of Rancher. I have a job which creates thousands of containers requests on a fixed set of machines (say 10). These containers are short lived and run a command before exiting.

Using REST API calls I plan to create these thousands of container requests inside Rancher, expecting it to queue and schedule them for execution when resources became available. I am hoping Rancher can execute multiple (3-4) containers on any given host. Also certain containers should go to certain hosts.

Is this a valid use of Rancher? What scheduler (Cattle, Kubernetes, Mesos) would you recommend for something like this?
Thanks in advance.

2

Most of the orchestration engines are designed to be running all the specified containers simultaneously, not a job queue… In Mesos you have to say how much of each resource is available to each host and how much each container needs so that may work a little better, but you’d have to write your own framework for it.

But you would probably be better off with something that is designed to be a job queue, e.g. gearman or some of the exiting Mesos frameworks. And then those jobs may happen to run containers, if the work you actually want to to do involves containers.

1 Like