[SOLVED] Unable to use Ingress/Traefik in K3S version v1.23.6+k3s1 (418c3fa8)

Hello, I was wondering if you could help me, as I’m trying to troubleshoot issues with basic K3S setup with (default) Traefik-based ingress. I’m using K3S locally on my laptop running Fedora 35 (Fedora Linux 35 (Workstation Edition) x86_64, Kernel: 5.17.5-200.fc35.x86_64).

So the setup I’m trying to get to run is dead simple: single deployment with default Apache server, single service with ClusterIP and Ingress pointing to the service.

Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apache-hello
spec:
  replicas: 1
  selector:
    matchLabels:
      app: apache-hello
  template:
    metadata:
      labels:
        app: apache-hello
        name: apache-hello
    spec:
      containers:
        - name: apache-hello
          image: httpd
          ports:
            - containerPort: 80

Service:

apiVersion: v1
kind: Service
metadata:
  name: apache-hello-service
  labels:
    run: apache-hello
spec:
  selector:
    app: apache-hello
  ports:
    - protocol: TCP
      port: 80

These two are running just fine:

> kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   local-path-provisioner-6c79684f77-wmtss   1/1     Running     0          39m
kube-system   coredns-d76bd69b-p62sj                    1/1     Running     0          39m
kube-system   helm-install-traefik-crd-djjsb            0/1     Completed   0          39m
kube-system   helm-install-traefik-j95sj                0/1     Completed   1          39m
kube-system   svclb-traefik-4wqgs                       2/2     Running     0          38m
kube-system   metrics-server-7cd5fcb6b7-bcpck           1/1     Running     0          39m
kube-system   traefik-df4ff85d6-zx87n                   1/1     Running     0          38m
default       apache-hello-5876f688c9-8rqrg             1/1     Running     0          33m

Deployment running fine:

> kubectl describe deployments.apps/apache-hello 
Name:                   apache-hello
Namespace:              default
CreationTimestamp:      Mon, 09 May 2022 08:44:19 +0200
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=apache-hello
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=apache-hello
           name=apache-hello
  Containers:
   apache-hello:
    Image:        httpd
    Port:         80/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   apache-hello-5876f688c9 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  33m   deployment-controller  Scaled up replica set apache-hello-5876f688c9 to 1

So is service:

> kubectl describe services/apache-hello-service 
Name:              apache-hello-service
Namespace:         default
Labels:            run=apache-hello
Annotations:       <none>
Selector:          app=apache-hello
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.43.31.41
IPs:               10.43.31.41
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.42.0.9:80
Session Affinity:  None
Events:            <none>

I can connect to the service directly using ClusterIP (running K3S locally, I’m actually on the node itself):

> curl -v http://10.43.31.41:80
*   Trying 10.43.31.41:80...
* Connected to 10.43.31.41 (10.43.31.41) port 80 (#0)
> GET / HTTP/1.1
> Host: 10.43.31.41
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Mon, 09 May 2022 07:20:19 GMT
< Server: Apache/2.4.53 (Unix)
< Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
< ETag: "2d-432a5e4a73a80"
< Accept-Ranges: bytes
< Content-Length: 45
< Content-Type: text/html
< 
<html><body><h1>It works!</h1></body></html>
* Connection #0 to host 10.43.31.41 left intact

I have also tested that port-forward option of kubectl:

> kubectl port-forward services/apache-hello-service 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
Handling connection for 8080
Handling connection for 8080
> curl -v http://localhost:8080
*   Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Mon, 09 May 2022 07:21:27 GMT
< Server: Apache/2.4.53 (Unix)
< Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
< ETag: "2d-432a5e4a73a80"
< Accept-Ranges: bytes
< Content-Length: 45
< Content-Type: text/html
< 
<html><body><h1>It works!</h1></body></html>
* Connection #0 to host localhost left intact

I can also check Apache logs:

> kubectl logs deployments/apache-hello 
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.42.0.9. Set the 'ServerName' directive globally to suppress this message
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.42.0.9. Set the 'ServerName' directive globally to suppress this message
[Mon May 09 06:44:27.522342 2022] [mpm_event:notice] [pid 1:tid 140417909382464] AH00489: Apache/2.4.53 (Unix) configured -- resuming normal operations
[Mon May 09 06:44:27.522458 2022] [core:notice] [pid 1:tid 140417909382464] AH00094: Command line: 'httpd -D FOREGROUND'
10.42.0.1 - - [09/May/2022:06:45:17 +0000] "GET / HTTP/1.1" 200 45
10.42.0.1 - - [09/May/2022:06:45:58 +0000] "GET / HTTP/1.1" 200 45
127.0.0.1 - - [09/May/2022:07:10:59 +0000] "GET / HTTP/1.1" 200 45
127.0.0.1 - - [09/May/2022:07:11:28 +0000] "GET / HTTP/1.1" 200 45
10.42.0.1 - - [09/May/2022:07:20:19 +0000] "GET / HTTP/1.1" 200 45
127.0.0.1 - - [09/May/2022:07:21:27 +0000] "GET / HTTP/1.1" 200 45

So yeah, I’m positive my Service and Deployment are working just fine. Now, when I try to add Ingress to the picture:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: apache-hello-ingress
  annotations:
    kubernetes.io/ingress.class: "traefik"
spec:
  rules:
  - host:
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: apache-hello-service
            port:
              number: 80

Installation seems to succeed:

> kubectl get ingress/apache-hello-ingress 
NAME                   CLASS    HOSTS   ADDRESS         PORTS   AGE
apache-hello-ingress   <none>   *       192.168.1.239   80      37m

> kubectl describe ingress/apache-hello-ingress 
Name:             apache-hello-ingress
Labels:           <none>
Namespace:        default
Address:          192.168.1.239
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /   apache-hello-service:80 (10.42.0.9:80)
Annotations:  kubernetes.io/ingress.class: traefik
Events:       <none>

But the problem is that whenever I try to use the ingress, I get “Bad Gateway” response like so:

> curl -v http://localhost:80
*   Trying 127.0.0.1:80...
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 502 Bad Gateway
< Date: Mon, 09 May 2022 07:25:18 GMT
< Content-Length: 11
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host localhost left intact
Bad Gateway

I can’t see anything in the logs:

> kubectl -n kube-system logs deployments/traefik 
time="2022-05-09T06:39:04Z" level=info msg="Configuration loaded from flags."
kubectl -n kube-system logs daemonsets/svclb-traefik 
Defaulted container "lb-tcp-80" out of: lb-tcp-80, lb-tcp-443
+ trap exit TERM INT
+ echo 10.43.74.176
+ grep -Eq :
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 1 '!=' 1 ]
+ iptables -t nat -I PREROUTING '!' -s 10.43.74.176/32 -p TCP --dport 80 -j DNAT --to 10.43.74.176:80
+ iptables -t nat -I POSTROUTING -d 10.43.74.176/32 -p TCP -j MASQUERADE
+ '[' '!' -e /pause ]
+ mkfifo /pause

This is what I can see in the description of Traefik service:

> kubectl -n kube-system describe services/traefik 
Name:                     traefik
Namespace:                kube-system
Labels:                   app.kubernetes.io/instance=traefik
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=traefik
                          helm.sh/chart=traefik-10.19.300
Annotations:              meta.helm.sh/release-name: traefik
                          meta.helm.sh/release-namespace: kube-system
Selector:                 app.kubernetes.io/instance=traefik,app.kubernetes.io/name=traefik
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.43.74.176
IPs:                      10.43.74.176
LoadBalancer Ingress:     192.168.1.239
Port:                     web  80/TCP
TargetPort:               web/TCP
NodePort:                 web  30976/TCP
Endpoints:                10.42.0.8:8000
Port:                     websecure  443/TCP
TargetPort:               websecure/TCP
NodePort:                 websecure  31975/TCP
Endpoints:                10.42.0.8:8443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason            Age                From           Message
  ----    ------            ----               ----           -------
  Normal  UpdatedIngressIP  48m                svccontroller  LoadBalancer Ingress IP addresses updated: 192.168.1.239
  Normal  AppliedDaemonSet  47m (x6 over 48m)  svccontroller  Applied LoadBalancer DaemonSet kube-system/svclb-traefik

What is weird, I can’t connect using either of the IPs, names or ports (even NodePorts):

curl http://192.168.1.239:80
Bad Gateway
curl http://192.168.1.239:80/
Bad Gateway
curl http://10.43.74.176:80/
Bad Gateway
curl http://192.168.1.239:30976
Bad Gateway
curl http://localhost:30976
Bad Gateway

Do you have any idea what I might be doing wrong? All suggestions are welcome and any help appreciated!

Actually figured it out, I had to disable the firewalld service, as described here:

Restart of the whole system was also necessary - simply stopping service didn’t suffice.