K8S nodes communication - inter nodes communication on private network

[fcoulloudon]

Good morning/afternoon everyone!

I’m very happy deploying K8S using RKE2/Rancher.

Everything is working flawlessly except one thing: the physical network used for communication between nodes (ie. inter nodes communication).

I have, on one of my setups, two nodes: 1 server and 1 agent.
I installed both of them following the general documentation.

These two machines are connected on two different physical networks:

1. One interface goes to our router with a public IP address (directly accessible from the internet)
2. Another interface goes to a switch and a private network.

The private network has been designed to be way faster and only go through a switch without firewall rules.

When I declared the agent node and linked it to the server node, I used the private IP address. However, for a reason I can’t explain, now both nodes communicate using their public IPs. If believe Flannel is using the first interface by default but I’m not 100% sure.

How to have the nodes to communicate on the private network?
I read RKE2 server config:
And I’m not clear about the impact of changing node-external-ip/node-ip values:
–node-ip value, -i value
(agent/networking) IPv4/IPv6 addresses to advertise for node
–node-external-ip value
(agent/networking) IPv4/IPv6 external IP addresses to advertise for node

First of all, when I change these values, they are not necessarily directly visible even after restarting RKE2 agent/server. And I’m not sure they are the right elements (or the only elements) to modify.

Maybe it’s a Rancher or Flannel setting?

configuring flannel to use a non default interface in kubernetes

Can it be done from the Rancher UI?

I tried to find the solution but there are several answers on the web and it’s not clear:

Digital Ocean how to use private network to avoid bandwidth charges?

I installer rancher on a Digital Ocean droplet and created a Digital Ocean cluster with 3 nodes. All have private networking enabled and have a private and public IP. After looking at the bandwidth graphs in Digital Ocean it seems all the communication between the nodes and also between the nodes an rancher server is being done using the public IP address instead of the private network. According to Digital Ocean in a few days I have used 833GB of public bandwidth so will probably use several …

(I did search for more, but can only have two links in a post of this forum)

Any help giving me the right approach with RKE2 to have inter-node communication using my private network (and/or specific interface) would be helpful.

Thank you very much all!

[fcoulloudon]

I did continue to investgate and I tried to change iface from flannel:

Network Options | RKE2

# /var/lib/rancher/rke2/server/manifests/rke2-canal-config.yaml
---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-canal
  namespace: kube-system
spec:
  valuesContent: |-
    flannel:
      iface: "eth1"

However, when doing this, I can’t restart RKE2-server.
Also, I get this error message on the rke2-canal pod:

Readiness probe failed: Get “http://localhost:9099/readiness”: dial tcp 127.0.0.1:9099: connect: connection refused

I also tried playing with node-ip and node-external-ip.

Any hints would be appreciated :-).