Hello all!
I’m not sure this is specific to RKE2 but since it ships with a Nginx Ingress controller, I’ll ask here anyway.
I deployed a RKE2 cluster in a virtual environment with each VM node having 3 network interfaces. Let’s say one interface in the 192.0.2.0/24 network, one in the 198.51.100.0/24 network and one in the 203.0.113.0/24 network. The default gateway of each node is in the 192.0.2.0/24 network.
I deployed Kube VIP (cloud provider) with cidr-global param set to “192.0.2.10-192.0.2.11,198.51.100.10-198.51.100.11,203.0.113.10-203.0.113.11”. That gives me 2 assignable VIPs by network.
What I’m trying to achieve is to be able to choose which VIP address a Loadbalancer service has to listen on. I want some services to be exposed on the VIP address of the 192.0.2.0/24 network, some on the VIP address of the 198.51.100.0/24 network and some others on the VIP address of the 203.0.113.0/24 network.
Is there anyone out there who succeeded in doing something like that ?
My observations - correct me if I’m wrong - are the following:
First thing, Kube VIP seems to work on a single interface. After some research, when deploying KubeVIP you have to choose the network interface on which the VIP addresses will be managed. With a Kube VIP working with the interface in the 192.0.2.0/24 network, I tried to declare several ranges, one within each network. Doing so the VIP adresses belonging to each network and handled by Kube VIP are set on the same interface (the one Kube VIP is configured on) and that brings on an odd network situation. The network interface on the 192.0.2.0/24 network gets all the VIPs even the ones in the other networks (the ones handled by the 2 other network interfaces)
As a remedy to that, should I deploy 3 Kube VIP instances, one for each network? Is that even viable?
Let’s say that is the way to go (having 3 Kube VIP instances), on the Nginx Ingress part, It seems that it is impossible to declare a single interface to bind the 80/443 ports on (you can choose the IP though but this is not applicable here since the VIP addresses are not on every node at the same time). By default the Nginx Ingress bounds on the port 80/443 of all the network interfaces. So, since we can’t choose the interface to listen on, it excludes the possibility to have 3 Nginx Ingress, one for each network/VIP.
What do you think would be the best solution to fit my needs ? I even though about deploying 3 K8S clusters, one for each network but that seems a bit overkill.
Thanks in advance to anyone taking the time to read me.