Using Marathon health checks over managed network segments

Archive Rancher 1.x
1

Hello,

I am running a somewhat complicated setup on Rancher, and am having some trouble with Marathon being able to perform health checks on applications when it is running in a managed network. Let me give you some background…

For reasons I can’t get into here, I am running my Mesos/Marathon cluster from within a container, but NOT the Rancher-provided one-click install. If I start my Mesos/Marathon containers in host networking mode, things work as I’d expect. That means my Mesos cluster is healthy and the Marathon framework is deployed. I am able to start Docker containers via Marathon, have the ports randomly assigned, and monitor the health of the application. In this configuration, the Docker client has been installed in the Mesos/Marathon container as well. Its purpose is to talk to Docker on the host to spawn the containers (sounds a bit like Inception 2, eh?). So all said and done, the containers that get started by Marathon live on the host right next to the Rancher agent and the Mesos/Marathon container…

This is all good for host networking mode. However, when I run my Mesos/Marathon container cluster in Rancher’s managed network, things start getting a little funky.

First, my Mesos cluster starts up fine. Marathon gets deployed without any problems. I am able to spawn a container and, using the docker logs, see that the container gets started correctly. However, because the container lives outside of Rancher’s managed network, the health check in Marathon fails (which eventually kills the app). I tried to force the container spawned by Marathon onto the managed network my adding this into the Marathon json deploy:

    "docker": {
      "parameters": [
        {
          "key": "label",
          "value": "io.rancher.container.network=true"
        }
      ],

This seems to get the container on the 10.42.x.x managed network… however, Marathon still can’t monitor the health check. I’m wondering if there’s any way to either

  • Tell Mesos/Marathon how to find the application on the host, or the new managed network
  • Figure out how to launch my Docker containers on the same managed IP as the Mesos/Marathon

Here’s an example of my Marathon port mappings:

      "portMappings": [
        { "containerPort": 3000, "hostPort": 0, "protocol": "tcp" }
      ]

and health check:

 "healthChecks": [
   {
         "path": "/",
         "portIndex": 0,
         "protocol": "HTTP",
         "gracePeriodSeconds": 10,
         "intervalSeconds": 10,
         "timeoutSeconds": 10,
         "maxConsecutiveFailures": 2
   }
 ]

Does anyone have any experience with this type of setup? Thanks in advance.

EDIT: I read this excerpt from Rancher Docs: Networking in Rancher

NOTE:
The Rancher managed IP address will not be present in Docker meta-data and as such will not appear in the result of a Docker “inspect.” This sometimes causes incompatibilities with certain tools that require a Docker bridge IP. We are already working with the Docker community to make sure a future version of Docker can handle overlay networks more cleanly.

I am thinking my problem is related, since Marathon tries to setup a bridged network.