Rancher architecture and Communication

1

For a real live deploy we need to explain to the security team of a customer project how rancher communication works and the high level architecture works. In particular we need to know the direction of the communication between the rancher server and the host, the required open ports on the hosts. Further which process are running as part of the privileged rancher-agent and agent-instance containers.

So far we understand a Hosts ONLY needs the following Ports Open to work:

  • IPSec Ports for the overlay network (4500, 500 UDP)
  • Ports Required for the Host Exposed Services via -p xxxx:yyyy or the build in load balancer

As far we understand NO further Ports needs to be open (like the Docker API Port) at the
host as the API communication goes trough IPSec?

So Far I assume the communication model is the following:

  • The ranger-agent connects to the Rancher Server over HTTP(S) from the host to the Rancher Server
  • The Rancher Server sends requests (Long-Poll or WS?) to the rancher-agent over the existing connection?!
  • The Rancher Server Issues the Commands over this HTTP API to the rancher-agent and the rancher-agent
    interacts with the Local docker host instance?!?!

The following Questions are still open to answer to the security team:

  • For creating the services on the host, the rancher-agent is using the standard docker-compose tools to evaluate the service.yml files and interact with the docker-engine. Or is rancher-server or agent is re-implementing this functionality
  • Who interacts with the docker-engine on the host? the rancer-server over the IPSec overlay or the rancher-agent locally?
  • how can be communication between different services deployed in different zones (i.e. defined by labels) firewalled ? the current assumptions is: every container in an environment can talk to any other container to all ports. (which BTW: would no be accepted and can become a deal breaker so far)

Whats in the privileged Container?

I assume we simply look here:


It would be really great to move to smaller base images like alpine Linux for that, as a full blown ubuntu
has quite a broad surface to attack. But seems normal, even after the announcement the official docker images will, be migrated to alpine, not much alpine to see in the official repos.

Is there any resource explaining the architecture, preferable with a nice picture, some boxes, lines and arrows?
We think it would help adoption in “enterprise” environments if the OPS and Security Teams understand how
rancher works, as it is already quire complex and “hidden” behind a quite good UX.

2

Agents open a WebSocket connection to the Server, which provides the 2-way communication & allows them to work with no listening/agent port exposed. Only the IPSec ports need to be exposed, (and only to the other host IPs if you want to restrict it). Plus of course your application ports too.

IPSec is not involved in the control plane and the Server container is not part of the managed network; IPSec is only host<->host for container communication.

The agent manipulates containers and reports changes using the “docker remote” API (via the docker.sock bound in). Docker-compose is not involved.

There is one overlay network per Environment and no current mechanism to restrict communication between containers inside of that. Depending on needs there is probably enough info in metadata for one to build a service that manipulates iptables to restrict communication.

agent-instance is a separate image unrelated to the main agent. It is what is run for Network Agent and Load Balancer service containers.

1 Like
3

Hey @vincent thanks a lot for the quick reply.

4

Based on this, I’m wondering where the compose files are evaluated and processed? Based on your answers I would assume in the rancher-server which then send the messages to the agent.

5

There is a rancher-compose-executor microservice which runs inside rancher/server (in the default install… For HA it’s a standalone container).

It runs rancher-compose with the input, parses it with libcompose and then turns that into API calls to manipulate services.

1 Like
6

cool, therefore it’s likely to stay compatible with the docker libcompose.

Thanks for the insights, the picture gets more clear.

7

Yes, we originally contributed libcompose to docker for that purpose… But they don’t actually do very much with it themselves because docker-compose is still Python and based on their original purchase of Fig.

8

Is there any resource explaining the architecture, preferable with a nice picture, some boxes, lines and arrows?
We think it would help adoption in “enterprise” environments if the OPS and Security Teams understand how
rancher works, as it is already quire complex and “hidden” behind a quite good UX.

Yes, please point me to one as well…

Thanks
Saju

9

+1 Would love to see an architecture overview of rancher.

10

+1 Would like to have a rancher architecture diagram

11

+1 would love to have a rancher low level architecture diagram with network and port flows. I’m having to do this myself for a customer and wish this resource already existed.