Adding HA / VIP Ingress for non-VM deployments

Harvester
1

I have Harvester 1.3.0 on 3 node cluster. Each node has 2 Ethernet ports.
Installed in the office LAN network 10.0.4.0/24

The Harvester API and UI is configured to static IP 10.0.4.247

And I’m running deployments (few web servers) right on default Kubernetes (Same cluster where Harvester is running VMs).

At first I used default (included) Ingress, but wanted the apps traffic to flow through the non-management network.

So, here what I did:

  • created new cluster network (and VM net just in case)
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  annotations:
    network.harvesterhci.io/route: '{"mode":"auto","serverIPAddr":"","cidr":"","gateway":""}'
  labels:
    network.harvesterhci.io/clusternetwork: cn2
    network.harvesterhci.io/ready: 'true'
    network.harvesterhci.io/type: UntaggedNetwork
  name: net2
  namespace: default
spec:
  config: >-
    {"cniVersion":"0.3.1","name":"net2","type":"bridge","bridge":"cn2-br","promiscMode":true,"ipam":{}}
  • Installed Traefik via Helm Chart:
helm repo add traefik https://traefik.github.io/charts
helm repo update
helm install traefik traefik/traefik -f https://gitlab.acloud.app/system/harvester/-/raw/main/traefik/values.yaml

As you can see above, I have my values defined here:
https://gitlab.acloud.app/system/harvester/-/raw/main/traefik/values.yaml

The important part is:

service:
  enabled: true
  single: true
  type: LoadBalancer
  annotations:
    kube-vip.io/loadbalancerIPs: 10.0.4.248
  spec:
    loadBalancerClass: kube-vip.io/kube-vip-class
  loadBalancerClass: kube-vip.io/kube-vip-class

I configured Traefik to create LoadBalancer, which relies on kube-vip, included already with Harvester.

Everything deployed successfully. Tested with few deployments.

But it looks like the VIP, which kube-vip created for Traefik still uses same management network:

$ arp -a
? (10.0.4.247) at 02:11:32:2a:16:9a [ether] on enp1s0
? (10.0.4.248) at 02:11:32:2a:16:9a [ether] on enp1s0

So, how can I configure LoadBalancer to use non-management network?

3

Finally figured how to add additional network interface from non-management network to the pod.

For Traefik, here is how I do that:

deployment:
  podAnnotations:
    # https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/how-to-use.md#launch-pod-with-json-annotation
    k8s.v1.cni.cncf.io/networks: '[ { "name": "net2", "namespace":"default" } ]'

But… then I realized I don’t need that non-management NIC on the Traefik pod. I need it on VIP LoadBalancer!

So, looked at the kube-vip config. And the one deployed with Harvester by default doesn’t include additional NIC. Which probably makes sense, because kube-vip is part of the system installation, and there is no additional network exists during installation - they added later.

So, does that mean that I need to modify system-installed kube-vip to add non-management NIC to it? Or would be more appropriate to install second instance of kube-vip to work on non-management NIC?

Is there any other solution to enable HA Ingress on non-management NIC?