I’m interested in using Rancher to manage dynamic clusters running Spark on top of HDFS. I was hoping to use the experimental Hadoop + YARN stack that is included in Rancher, and deploy Spark on top of that. However, I’m running into problems with both (and my inexperience with Rancher likely isn’t helping either).
First: when I select “Launch” for the Hadoop + YARN setup in the Catalog, the new stack appears but no services ever spawn. I left my cluster (20 physical machines) like this for over a day with no change. Was there something else I needed to do?
Second: if/when I get the Hadoop/HDFS service up and running, what would be the best way to go about deploying Spark on top? I noticed the Rancher Dockerhub has a Spark package, but I could not find a corresponding dockerfile anywhere to examine. Please advise. Thanks!
Hi @magsol, The Hadoop + Yarn stack startup behavior varies depending on your rancher version. v0.47.0 and later have start on launch and prior to that you had to click “Start all services” from the stack drop down menu on the stacks page. Or individual stacks can be started from their drop downs. In v0.49.0 we have received reports of some issues launching catalog entries and are working to get a fix out, hopefully in the next day. The stack also requires docker 1.9 to launch.
Are you seeing the service spinning? Or are they just sitting there… if they are spinning in an activating state, then something is wrong and we will need logs. Sometimes on the hosts page you can see dead containers, and can click on one of them to see an error message.
We do have a Spark template, but it currently has to be launched from rancher-compose command line. You can find the compose/Dockerfiles in the rancher/compose-templates github repo. Thats where we do all of our hacking before they become catalog entries
Spark is a standalone cluster at the moment, and would need to be tweaked to run on top of HDFS and/or HDFS+YARN. We are curious to get some feedback on the Hadoop stack from users, my hope was that you could deploy that, then link or share a common config data volume between all of the services and build out solutions that way.
Feedback welcome, I hope we can get you started in the next little bit.
Oh %$&#, somehow I missed that Docker 1.9 can’t run on RHEL 6 systems; I’m running kernel version 2.6.32 Pretty sure that’s the source of all my issues. Though how I was able to get the base Rancher infrastructure up and running on Docker 1.7.1 is interesting.
For completeness: I’m running Rancher 0.47. The stack just spins indefinitely when I click Launch (with the option “start on launch” selected).
Looks like I have some major OS overhauling to do before I can get this up and running as I’d hoped; I have 16 physical nodes I’ll have to re-image. That’s all part of the learning process, I suppose Thanks for your help; I’ll post again when I have something.
Pretty sure that’s the source of all my issues. Though how I was able to get the base Rancher infrastructure up and running on Docker 1.7.1 is interesting.