If I set up Rancher 2 with canal (default HA setup, 3 nodes, v2.1.0, Kubernetes v1.11.3, default k8s ingress https://github.com/kubernetes/ingress-nginx), how secure is the system? Do I need to secure connections? Do I need a separate firewall?
I am not asking about the Rancher authentication system which is well documented. I am asking about the default setup of ingress controllers, canal, k8s API. Can this system be reached from the outside somehow, or does Rancher use best practices for securing the system?
Is anybody aware of a detailed discussion of the issue.
You pods are on a private network, so they’ re not directly accessible from outside. That means, a web server on a pod can’t get accessed from the internet (unless you specifically instruct k8s to let it happen).
The traffic between your pods is encapsulated with ipsec (so, it won’t be sniffed). If you were using Calico (like I do), there would be no encapsulation and you would have to take care of that part (if required).
The traffic between your hosts is not secured in any sense. That’s on you. You may or may not need to do something about it depending on your setup and the security requirements you have.
what you mean from not secured between hosts?
Is a simple firewall configuration to deny access from outside of these hosts, sufficient or we must setup another secure ipsec for them?
Depends on your setup. If your servers are on a single provider inside private network you don’t have to do anything. If (like mine) they’re on multiple commodity hosting companies, you’d use ipsec or wireguard (I use wireguard). All that said, I do not know exactly what kind of traffic gets sent between the host nodes and how sensitive that is.
So for me, I use wireguard to connect the hosts for 2 reasons:
a) My CNI of choice is Calico (so no encapsulation/ipsec for the pods) and
b) I want to play it safe for the rest of the traffic that I don’t know how sensitive it may (or not) be.
@localhost If I am not mistaken, Canal is a wrapper around Calico plus some other tools. Your response would mean, that traffic isn’t secured unless Canal secures it or Rancher configures Canal to do so (which I would expect from Rancher to do). Does Rancher secure such traffic by default or not?
Let’s say we have 2 nodes on the public internet, with an ingress controller on each host. In k8s, a pod does not necessarily need to be on the same host as the ingress controller. Imagine the ingress accesses the pod on another host across the internet. If I understand this correctly, by default, Rancher spans an internal network between hosts using Canal. The pod serves unsecured http traffic over the internal network, and the ingress controller injects the certificate and serves as https to the caller. The channel between pod and ingress would be unsecured then.
@oliverhausler Canal does encapsulation with ipsec, Calico does not. So, if you use Canal, you’re set w.r.t. to the inter-pod traffic security from sniffing. The confusion is that project calico originally intended for Canal to be the new Calico (and so, merge these together) but then changed their minds and so they’re separate things.
Yes.
The internal network using Canal is encapsulated with ipsec. So in terms of sniffing from outside, the traffic is secure. In terms of sniffing from inside (a compromised pod) it is not secure. If that’s an attack surface you care about (hacked pods sniffing traffic), I do not know much on how to mitigate it.
Ingress<->pod is secure (from the outside attacker) with Canal. Not secure (from the outside attacker) with Calico. Not secure from the inside regardless of Canal of Calico.
I have been looking into similar issue lately. Plan is to run cluster in Hezner cloud, witch does not provide any firewall or internal ip network.
So there are few ports open to world, like 10250 - kubelet
I don’t even want to think about compromised pods and attacks from inside k8s. This, with heavy open source usage and auto-importing of libraries, might become a nightmare in the future. No idea how to approach this attack vector.
The scenario reminds me of the early days of Windows, where Microsoft didn’t secure LAN traffic because they thought everybody inside the organization could be trusted. Boy, where they wrong. I remember stats which claimed 80% of attacks came from the inside. We have something very similar here, and history tends to repeat itself. Scary, and not much we can do.
in case of inner attack, rancher offer Project Network Isolation in cluster settings. so one more layer of securing a rancher cluster is to set it to True. but then in that project there is nothing rancher do for securing your pods from its neighbors.
Are you sure about this? I’ve been looking into having encrypted networking between kubernetes nodes, and it looks like all traffic between nodes is unencrypted when using canal, just vxlan encapsulated with flannel, but not encrypted.
rancher 1.x did ipsec tunneling between hosts, but that doesn’t seem to be the case in rancher 2.x canal.
Using canal I can see unencrypted traffic (e.g. SQL statements) when I watch eth0 with tcpdump on the Hetzner Cloud node. After switching to weave CNI, traffic appears to be encrypted.
