K3s MetalLB ingress-nginx not working properly

k3s, k3OS, and k3d
1

Hi,

I have a virtual machine build of Ubuntu 20.04 with k3s installed, --disable traefik and–disable servicelb set, CertManager, MetalLB and ingress-nginx installed on top.

My intent in all this is to use a configmap.yaml file to set several ingress IPs for the cluster and use instances of ingress-nginx to load balance and L7-route requests to service applications based upon the hostnames set in the applications.

The L7 routing works fine but I am having trouble with the ingress-nginx to MetalLB connection where the selection of the address-pool in MetalLB is not being performed in accordance with the metallb.universe.tf/address-pool: annotation on the controller-service.yaml selector.

Can anyone tell me whether I need to do anyting other than set the annotation above to the name of an address-pool in MetalLB for that address-pool to be used ? The allocated IP tahn ingress-nginx receives is always the first for which “auto-assign: false” is set in the address-pool descriptor.

There is a much fuller description of my problem here on stackoverflow giving all the version numbers and the full setup info with yamls and install code etc.

Thanks,

2

Hmm, I would agree, your config from Stackoverflow “should” get your desired result.

Anything of interest in the Metallb Controller Pod? Does it properly see your inetfa pool? When it goes to assign an IP to the service, anything interesting in there?

I’ll try to duplicate a similar setup and see if I can spot anything.

3

Update:

Duplicated similar buildout and I got an IP from the inetfa pool.

apiVersion: v1
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 172.16.5.11-172.16.5.20
    - name: inetfa
      protocol: layer2
      addresses:
      - 192.168.1.16/32
kind: ConfigMap

Note: The default pool is in no way full

apiVersion: v1
kind: Service
metadata:
  name: test
  annotations:
    metallb.universe.tf/address-pool: inetfa
  namespace: smtp
spec:
  ports:
    - name: smtp
      port: 25
      protocol: TCP
      targetPort: 25
  type: LoadBalancer

status:
  loadBalancer:
    ingress:
      - ip: 192.168.1.16

Some things to check is if the inetfa pool was added to the ConfigMap after the Controller pod was started, Pods do not get informed of an updated ConfigMap automatically, so you do need to restart the Controller pod (I usually just nuke it from orbit and let Kubernetes reschedule it… Probably not the best way, but it works :slight_smile:

Check the logs as the Controller is starting, make sure there are no errors regarding the pool and also logs as MetalLB is assigning an IP to the service.

4

Gan you tell me the version numbers you are using, please Casey ? I assume you’re on k3s not k8s or similar ?

I have been re-booting the complete machine to ensure that everything reloads as I am never certain which pods etc need to be restarted. Am an absolute kubernetes beginner so please be patient with me.

Will check out the logs…

5

My k3s cluster is currently on v1.21.4+k3s1, but it started at 1.19, then 1.20 all with this functioning.

Metallb is at v0.10.2

OS is Ubuntu 20.04

6

And your nginx version ?

The logs are as follows:

nginx:

$ kubectl logs ingress-nginx-plain-controller-64f79ddcc6-j7pp7
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.0.0
  Build:         041eb167c7bfccb1d1653f194924b0c5fd885e10
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.20.1

-------------------------------------------------------------------------------

W0927 15:42:08.949893       8 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0927 15:42:08.950126       8 main.go:221] "Creating API client" host="https://10.43.0.1:443"
I0927 15:42:09.052328       8 main.go:265] "Running in Kubernetes cluster" major="1" minor="21" git="v1.21.4+k3s1" state="clean" commit="3e250fdbab72d88f7e6aae57446023a0567ffc97" platform="linux/amd64"
I0927 15:42:09.505353       8 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0927 15:42:09.687343       8 ssl.go:531] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0927 15:42:09.833997       8 nginx.go:253] "Starting NGINX Ingress controller"
I0927 15:42:09.893116       8 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"default", Name:"ingress-nginx-plain-controller", UID:"f437c6a4-f303-421b-9977-e382beb4bc49", APIVersion:"v1", ResourceVersion:"16616", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap default/ingress-nginx-plain-controller
I0927 15:42:10.940087       8 store.go:365] "Found valid IngressClass" ingress="default/my-release-apache" ingressclass="nginx"
I0927 15:42:10.940766       8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"my-release-apache", UID:"ba739c7c-b166-4c14-ae9c-700c97f21d64", APIVersion:"networking.k8s.io/v1", ResourceVersion:"21537", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
W0927 15:42:10.940904       8 backend_ssl.go:46] Error obtaining X.509 certificate: no object matching key "default/apache1.local-tls" in local store
I0927 15:42:11.041039       8 nginx.go:295] "Starting NGINX process"
I0927 15:42:11.041198       8 leaderelection.go:243] attempting to acquire leader lease default/ingress-controller-leader...
I0927 15:42:11.041707       8 nginx.go:315] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
W0927 15:42:11.042608       8 controller.go:1222] Error getting SSL certificate "default/apache1.local-tls": local SSL certificate default/apache1.local-tls was not found. Using default certificate
I0927 15:42:11.042648       8 controller.go:150] "Configuration changes detected, backend reload required"
I0927 15:42:11.052708       8 leaderelection.go:253] successfully acquired lease default/ingress-controller-leader
I0927 15:42:11.053073       8 status.go:84] "New leader elected" identity="ingress-nginx-plain-controller-64f79ddcc6-j7pp7"
I0927 15:42:11.252831       8 controller.go:167] "Backend successfully reloaded"
I0927 15:42:11.256741       8 controller.go:178] "Initial sync, sleeping for 1 second"
I0927 15:42:11.256977       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"ingress-nginx-plain-controller-64f79ddcc6-j7pp7", UID:"c9729f4f-439c-4935-bd6b-aa99c83d0c73", APIVersion:"v1", ResourceVersion:"25874", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
W0927 15:42:14.646233       8 controller.go:1222] Error getting SSL certificate "default/apache1.local-tls": local SSL certificate default/apache1.local-tls was not found. Using default certificate
W0927 15:42:20.594176       8 controller.go:1222] Error getting SSL certificate "default/apache1.local-tls": local SSL certificate default/apache1.local-tls was not found. Using default certificate
W0927 15:42:23.928344       8 controller.go:1007] Service "default/my-release-apache" does not have any active Endpoint.
W0927 15:42:23.928457       8 controller.go:1222] Error getting SSL certificate "default/apache1.local-tls": local SSL certificate default/apache1.local-tls was not found. Using default certificate
W0927 15:42:27.261757       8 controller.go:1007] Service "default/my-release-apache" does not have any active Endpoint.
W0927 15:42:27.261795       8 controller.go:1222] Error getting SSL certificate "default/apache1.local-tls": local SSL certificate default/apache1.local-tls was not found. Using default certificate
W0927 15:42:30.594316       8 controller.go:1007] Service "default/my-release-apache" does not have any active Endpoint.
W0927 15:42:30.594424       8 controller.go:1222] Error getting SSL certificate "default/apache1.local-tls": local SSL certificate default/apache1.local-tls was not found. Using default certificate
W0927 15:42:41.821845       8 controller.go:1222] Error getting SSL certificate "default/apache1.local-tls": local SSL certificate default/apache1.local-tls was not found. Using default certificate

metallb:

$ kubectl logs -n metallb-system controller-6b78bff7d9-n6525
{"branch":"HEAD","caller":"main.go:138","commit":"v0.10.2","goversion":"gc / go1.16.5 / amd64","msg":"MetalLB controller starting version 0.10.2 (commit v0.10.2, branch HEAD)","ts":"2021-09-27T15:42:12.71020917Z","version":"0.10.2"}
{"caller":"k8s.go:325","msg":"secret already exists, nothing to do","op":"CreateMlSecret","ts":"2021-09-27T15:42:12.819197702Z"}
{"caller":"main.go:100","configmap":"metallb-system/config","event":"startUpdate","msg":"start of config update","ts":"2021-09-27T15:42:12.920097432Z"}
{"caller":"main.go:113","configmap":"metallb-system/config","event":"endUpdate","msg":"end of config update","ts":"2021-09-27T15:42:12.920172516Z"}
{"caller":"k8s.go:545","configmap":"metallb-system/config","event":"configLoaded","msg":"config (re)loaded","ts":"2021-09-27T15:42:12.920188613Z"}
{"caller":"main.go:49","event":"startUpdate","msg":"start of service update","service":"cert-manager/cert-manager","ts":"2021-09-27T15:42:12.920223505Z"}
{"caller":"service.go:33","event":"clearAssignment","msg":"not a LoadBalancer","reason":"notLoadBalancer","service":"cert-manager/cert-manager","ts":"2021-09-27T15:42:12.922517124Z"}
{"caller":"main.go:75","event":"noChange","msg":"service converged, no change","service":"cert-manager/cert-manager","ts":"2021-09-27T15:42:12.922622302Z"}
{"caller":"main.go:76","event":"endUpdate","msg":"end of service update","service":"cert-manager/cert-manager","ts":"2021-09-27T15:42:12.9226318Z"}
{"caller":"main.go:49","event":"startUpdate","msg":"start of service update","service":"cert-manager/cert-manager-webhook","ts":"2021-09-27T15:42:12.922646797Z"}
{"caller":"service.go:33","event":"clearAssignment","msg":"not a LoadBalancer","reason":"notLoadBalancer","service":"cert-manager/cert-manager-webhook","ts":"2021-09-27T15:42:12.922658294Z"}
{"caller":"main.go:75","event":"noChange","msg":"service converged, no change","service":"cert-manager/cert-manager-webhook","ts":"2021-09-27T15:42:12.922715982Z"}
{"caller":"main.go:76","event":"endUpdate","msg":"end of service update","service":"cert-manager/cert-manager-webhook","ts":"2021-09-27T15:42:12.922721781Z"}
{"caller":"main.go:49","event":"startUpdate","msg":"start of service update","service":"default/ingress-nginx-plain-controller-admission","ts":"2021-09-27T15:42:12.92272798Z"}
{"caller":"service.go:33","event":"clearAssignment","msg":"not a LoadBalancer","reason":"notLoadBalancer","service":"default/ingress-nginx-plain-controller-admission","ts":"2021-09-27T15:42:12.922735178Z"}
{"caller":"main.go:75","event":"noChange","msg":"service converged, no change","service":"default/ingress-nginx-plain-controller-admission","ts":"2021-09-27T15:42:12.922800265Z"}
{"caller":"main.go:76","event":"endUpdate","msg":"end of service update","service":"default/ingress-nginx-plain-controller-admission","ts":"2021-09-27T15:42:12.922806363Z"}
{"caller":"main.go:49","event":"startUpdate","msg":"start of service update","service":"default/ingress-nginx-plain-controller","ts":"2021-09-27T15:42:12.922812262Z"}
{"caller":"main.go:75","event":"noChange","msg":"service converged, no change","service":"default/ingress-nginx-plain-controller","ts":"2021-09-27T15:42:12.922977128Z"}
{"caller":"main.go:76","event":"endUpdate","msg":"end of service update","service":"default/ingress-nginx-plain-controller","ts":"2021-09-27T15:42:12.922990625Z"}
{"caller":"main.go:49","event":"startUpdate","msg":"start of service update","service":"default/my-release-apache","ts":"2021-09-27T15:42:12.922998223Z"}
{"caller":"service.go:33","event":"clearAssignment","msg":"not a LoadBalancer","reason":"notLoadBalancer","service":"default/my-release-apache","ts":"2021-09-27T15:42:12.923005522Z"}
{"caller":"main.go:75","event":"noChange","msg":"service converged, no change","service":"default/my-release-apache","ts":"2021-09-27T15:42:12.923068808Z"}
{"caller":"main.go:76","event":"endUpdate","msg":"end of service update","service":"default/my-release-apache","ts":"2021-09-27T15:42:12.923079006Z"}
{"caller":"main.go:49","event":"startUpdate","msg":"start of service update","service":"default/kubernetes","ts":"2021-09-27T15:42:12.923085505Z"}
{"caller":"service.go:33","event":"clearAssignment","msg":"not a LoadBalancer","reason":"notLoadBalancer","service":"default/kubernetes","ts":"2021-09-27T15:42:12.923097502Z"}
{"caller":"main.go:75","event":"noChange","msg":"service converged, no change","service":"default/kubernetes","ts":"2021-09-27T15:42:12.923129396Z"}
{"caller":"main.go:76","event":"endUpdate","msg":"end of service update","service":"default/kubernetes","ts":"2021-09-27T15:42:12.923134395Z"}
{"caller":"main.go:49","event":"startUpdate","msg":"start of service update","service":"kube-system/kube-dns","ts":"2021-09-27T15:42:12.923140093Z"}
{"caller":"service.go:33","event":"clearAssignment","msg":"not a LoadBalancer","reason":"notLoadBalancer","service":"kube-system/kube-dns","ts":"2021-09-27T15:42:12.923146892Z"}
{"caller":"main.go:75","event":"noChange","msg":"service converged, no change","service":"kube-system/kube-dns","ts":"2021-09-27T15:42:12.923329953Z"}
{"caller":"main.go:76","event":"endUpdate","msg":"end of service update","service":"kube-system/kube-dns","ts":"2021-09-27T15:42:12.923343351Z"}
{"caller":"main.go:49","event":"startUpdate","msg":"start of service update","service":"kube-system/metrics-server","ts":"2021-09-27T15:42:12.923350449Z"}
{"caller":"service.go:33","event":"clearAssignment","msg":"not a LoadBalancer","reason":"notLoadBalancer","service":"kube-system/metrics-server","ts":"2021-09-27T15:42:12.923358647Z"}
{"caller":"main.go:75","event":"noChange","msg":"service converged, no change","service":"kube-system/metrics-server","ts":"2021-09-27T15:42:12.923409037Z"}
{"caller":"main.go:76","event":"endUpdate","msg":"end of service update","service":"kube-system/metrics-server","ts":"2021-09-27T15:42:12.923415236Z"}
{"caller":"main.go:118","event":"stateSynced","msg":"controller synced, can allocate IPs now","ts":"2021-09-27T15:42:12.923420734Z"}
{"caller":"main.go:49","event":"startUpdate","msg":"start of service update","service":"cert-manager/cert-manager-webhook","ts":"2021-09-27T15:42:12.925518294Z"}
{"caller":"service.go:33","event":"clearAssignment","msg":"not a LoadBalancer","reason":"notLoadBalancer","service":"cert-manager/cert-manager-webhook","ts":"2021-09-27T15:42:12.925564685Z"}
{"caller":"main.go:75","event":"noChange","msg":"service converged, no change","service":"cert-manager/cert-manager-webhook","ts":"2021-09-27T15:42:12.92563447Z"}
{"caller":"main.go:76","event":"endUpdate","msg":"end of service update","service":"cert-manager/cert-manager-webhook","ts":"2021-09-27T15:42:12.925641568Z"}
{"caller":"main.go:49","event":"startUpdate","msg":"start of service update","service":"default/ingress-nginx-plain-controller-admission","ts":"2021-09-27T15:42:12.925649467Z"}
{"caller":"service.go:33","event":"clearAssignment","msg":"not a LoadBalancer","reason":"notLoadBalancer","service":"default/ingress-nginx-plain-controller-admission","ts":"2021-09-27T15:42:12.925657465Z"}
{"caller":"main.go:75","event":"noChange","msg":"service converged, no change","service":"default/ingress-nginx-plain-controller-admission","ts":"2021-09-27T15:42:12.925720552Z"}
{"caller":"main.go:76","event":"endUpdate","msg":"end of service update","service":"default/ingress-nginx-plain-controller-admission","ts":"2021-09-27T15:42:12.925726151Z"}
{"caller":"main.go:49","event":"startUpdate","msg":"start of service update","service":"default/ingress-nginx-plain-controller","ts":"2021-09-27T15:42:12.925731949Z"}
{"caller":"main.go:75","event":"noChange","msg":"service converged, no change","service":"default/ingress-nginx-plain-controller","ts":"2021-09-27T15:42:12.925838127Z"}
{"caller":"main.go:76","event":"endUpdate","msg":"end of service update","service":"default/ingress-nginx-plain-controller","ts":"2021-09-27T15:42:12.925851424Z"}
{"caller":"main.go:49","event":"startUpdate","msg":"start of service update","service":"default/my-release-apache","ts":"2021-09-27T15:42:12.925858023Z"}
{"caller":"service.go:33","event":"clearAssignment","msg":"not a LoadBalancer","reason":"notLoadBalancer","service":"default/my-release-apache","ts":"2021-09-27T15:42:12.925866621Z"}
{"caller":"main.go:75","event":"noChange","msg":"service converged, no change","service":"default/my-release-apache","ts":"2021-09-27T15:42:12.925930708Z"}
{"caller":"main.go:76","event":"endUpdate","msg":"end of service update","service":"default/my-release-apache","ts":"2021-09-27T15:42:12.925936207Z"}
{"caller":"main.go:49","event":"startUpdate","msg":"start of service update","service":"default/kubernetes","ts":"2021-09-27T15:42:12.925942005Z"}
{"caller":"service.go:33","event":"clearAssignment","msg":"not a LoadBalancer","reason":"notLoadBalancer","service":"default/kubernetes","ts":"2021-09-27T15:42:12.925957302Z"}
{"caller":"main.go:75","event":"noChange","msg":"service converged, no change","service":"default/kubernetes","ts":"2021-09-27T15:42:12.926009091Z"}
{"caller":"main.go:76","event":"endUpdate","msg":"end of service update","service":"default/kubernetes","ts":"2021-09-27T15:42:12.926020189Z"}
{"caller":"main.go:49","event":"startUpdate","msg":"start of service update","service":"kube-system/kube-dns","ts":"2021-09-27T15:42:12.926028887Z"}
{"caller":"service.go:33","event":"clearAssignment","msg":"not a LoadBalancer","reason":"notLoadBalancer","service":"kube-system/kube-dns","ts":"2021-09-27T15:42:12.926041884Z"}
{"caller":"main.go:75","event":"noChange","msg":"service converged, no change","service":"kube-system/kube-dns","ts":"2021-09-27T15:42:12.926147262Z"}
{"caller":"main.go:76","event":"endUpdate","msg":"end of service update","service":"kube-system/kube-dns","ts":"2021-09-27T15:42:12.926155661Z"}
{"caller":"main.go:49","event":"startUpdate","msg":"start of service update","service":"kube-system/metrics-server","ts":"2021-09-27T15:42:12.926161859Z"}
{"caller":"service.go:33","event":"clearAssignment","msg":"not a LoadBalancer","reason":"notLoadBalancer","service":"kube-system/metrics-server","ts":"2021-09-27T15:42:12.926168858Z"}
{"caller":"main.go:75","event":"noChange","msg":"service converged, no change","service":"kube-system/metrics-server","ts":"2021-09-27T15:42:12.926241343Z"}
{"caller":"main.go:76","event":"endUpdate","msg":"end of service update","service":"kube-system/metrics-server","ts":"2021-09-27T15:42:12.92625374Z"}
{"caller":"main.go:49","event":"startUpdate","msg":"start of service update","service":"cert-manager/cert-manager","ts":"2021-09-27T15:42:12.926260739Z"}
{"caller":"service.go:33","event":"clearAssignment","msg":"not a LoadBalancer","reason":"notLoadBalancer","service":"cert-manager/cert-manager","ts":"2021-09-27T15:42:12.926267837Z"}
{"caller":"main.go:75","event":"noChange","msg":"service converged, no change","service":"cert-manager/cert-manager","ts":"2021-09-27T15:42:12.926341922Z"}
{"caller":"main.go:76","event":"endUpdate","msg":"end of service update","service":"cert-manager/cert-manager","ts":"2021-09-27T15:42:12.92634992Z"}

These taken straight after a boot of the virtual machine.

Ingresses are:

kubectl get ingress -A
NAMESPACE   NAME                CLASS   HOSTS           ADDRESS        PORTS     AGE
default     my-release-apache   nginx   apache1.local   192.168.1.17   80, 443   11d

I’m very grateful for the assisstance,

7

The version of the ingress controller doesn’t matter as MetalLB is assigning an IP to the service itself, not the workload.

I think I might have an idea of what is stuck… My assumption is when you created the service, the MetalLB controller did not know what inetfa was (either your configmap didn’t have inetfa or the MetalLB controller wasn’t reloaded since it was there).

Since your lan-facing pool is open for MetalLB to do as it pleases, MetalLB came across the service, saw you wanted an IP from an inetfa pool… MetalLB didn’t understand that so just went with one from lan-facing. The service was then converged and so now MetalLB is happy with it’s work it did and moved on.

Then events unfolded, MetalLB is now aware of an inetfa pool; However, your service has already converged so MetalLB takes a single look at it, realizes it already has an IP and it doesn’t like doing extra work, so it logs it looked at it, was happy with it and moved right along.

{"caller":"main.go:49","event":"startUpdate","msg":"start of service update","service":"default/ingress-nginx-plain-controller","ts":"2021-09-27T15:42:12.922812262Z"}
{"caller":"main.go:75","event":"noChange","msg":"service converged, no change","service":"default/ingress-nginx-plain-controller","ts":"2021-09-27T15:42:12.922977128Z"}
{"caller":"main.go:76","event":"endUpdate","msg":"end of service update","service":"default/ingress-nginx-plain-controller","ts":"2021-09-27T15:42:12.922990625Z"}

So, what I propose you do is make sure you have the config available to remake the service. Delete the current service. Re-make the service (make sure to not include anything about the current (wrong) IP). This will now be a new service that has yet to be converged. MetalLB will pick it up, see it has some work to do and with MetalLB already knowing what the inetfa pool is, MetalLB should correctly converge the service with the desired IP.

8

Sorry, do you mean my ingress-nginx service or the apache service ?

9

Your ingress, the one with a type of LoadBalancer