I’m interested in using Rancher to manage dynamic clusters running Spark on top of HDFS. I was hoping to use the experimental Hadoop + YARN stack that is included in Rancher, and deploy Spark on top of that. However, I’m running into problems with both (and my inexperience with Rancher likely isn’t helping either).
First: when I select “Launch” for the Hadoop + YARN setup in the Catalog, the new stack appears but no services ever spawn. I left my cluster (20 physical machines) like this for over a day with no change. Was there something else I needed to do?
Second: if/when I get the Hadoop/HDFS service up and running, what would be the best way to go about deploying Spark on top? I noticed the Rancher Dockerhub has a Spark package, but I could not find a corresponding dockerfile anywhere to examine. Please advise. Thanks!