Starting Docker Containers from other Containers with Data Volumes

Rancher 1.x
1

I know… a Docker in Docker question. They’re a pain.

I am running Jenkins in Rancher. I have Jenkins agents/slaves setup in Rancher as well that are, obviously, containers in themselves. These agents need to run builds. The builds themselves are done in Docker containers.

There are a number of approaches to running Docker while in a container, but the current wisdom seems to suggest mounting the /var/run/docker.sock as the safest approach, so that’s what we’re trying to do. This mostly works, but as described in several places, a big issue to overcome is data volumes.

In the Jenkins agent container, I check out the code, then launch the “build” container and mount the checked out code into the build container so that it can be processed. Well, in the bind-mount approach to Docker in Docker, the docker you’re actually running is the Docker from the Host machine so when you pass a volume to be mounted, it tries to do that FROM the host machine. This fails, of course, because the code files are actually checked out in the agent container, not the host.

The articles linked above suggest data volumes to solve this, but I can’t figure out how to get that working in Rancher. Creating the data volume is no problem, and making it available to the agent container is trivial (using a Sidekick at the moment, but I could use a storage pool as well). The problem is making that same data volume accessible to the build container that I’m launching from within the agent container. Technically, I DO have access to the same data volume, as the Docker I’m using is the one from the host, however the problem is identifying the data volume so that I can link it with “–volumes-from” in the build container. I cannot figure out a consistent way to identify the correct volume. Sometimes it is something like r-jenkins-agent-workspace-volumes-1 but other times it doesn’t have a readable name but a UUID instead (after an upgrade, i think).

Any ideas here? Maybe I’m going at this in the completely wrong manner and making things way more complex than they need to be?

2

I have solved this, but I’m not particularly thrilled with my solution. To my team I am constantly preaching the benefits of “simplicity, simplicity, simplicity!”, but the contortions I am having to go through to get all of this working violates that principle a hundred fold.

In essence, what I ended up doing is: in the Jenkins build configuration step itself, I am doing a docker inspect to find the “.Mounts” and then passing those to the script which kicks off the build. I don’t have this lookup in the script itself as I’d prefer to keep the jenkins/rancher specific functionality out of our generic build script. That build script then uses the passed information to mount the volumes in the eventual build container, and then the entry script in the container finishes a few things off.

What I end up with is 1) not complex, but not intuitive commands in the jenkins job, 2) a fairly complex build script to kick off the container, and then finally 3) more logic in the entry script to smooth out the differences between building in the Jenkins/rancher environment and building anywhere else (e.g. the developer’s machine). Clearly this is far from ideal. What’s worse, this is only the first of about 15 projects that I’d like to move over into this new build environment. Having to add so much custom logic just because I’m working in Rancher (although I guess it would be the same in any environment where the builder is a container itself) is frustrating.

Am I missing something that would have made all of this much simpler and would have let me forget the fact that I’m doing a container build inside a parent container?

4

The way we are doing it is running the Jenkins master in Docker, but have multiple slaves as VMs. The slaves are basic CentOS machines that have Docker installed, and the Jenkins slave agent. We configured the Master to have zero executors, so all of the builds happen on the slaves.

5

Since there is a new reply on this quite old thread, I figured I’d update it with where we are now. For our new projects, I’ve actually taken us off Jenkins and tried to simply things using Gitlab for the entire pipeline. It’s actually working pretty well. Gitlab is takes docker into account at almost every step of the way. We have Gitlab in Rancher using gitlab-runner containers building images themselves. There are definitely hurdles (one of which is getting the runner-spawned containers onto the rancher managed network), but int he end we have fairly small and simple docker/rancher compose files and gitlab-ci.yml files running pretty large scale builds.

As always, I’m looking to simplify, simplify, simplify…

1 Like
6

When running Jenkins, the hub.docker.com page mentions to bind mount /var/jenkins_home, but the question is:

  1. How do you get the container to accept it? It seems that there are permission issues, because it’ll be mounted as root:root
  2. Do you use CMD or ENTRYPOINT? Which one can you use to run the command chown -R jenkins:jenkins /var/jenkins_home? How do you format that for Rancher’s UI to accept it?

I’ve been failing at being able to run CMD’s

7

I am using Rancher-NFS to mount the /var/jenkins_home and the /etc ssl directory, and it works fine for me.

version: '2'
volumes:
  jenkins-var-jenkins:
    external: true
    driver: rancher-nfs
  jenkins-etc-ssl-certs:
    external: true
    driver: rancher-nfs
services:
  Jenkins:
    image: jenkins/jenkins:lts
    stdin_open: true
    volumes:
    - jenkins-var-jenkins:/var/jenkins_home
    - jenkins-etc-ssl-certs:/etc/ssl/certs
    tty: true
8

Is Rancher-NFS the default volume driver, or do you need to specify it?

9

You need to specify it. By default, it tries to use local filesystem on the host that runs the container.

1 Like