Hello, I was wondering if you could help me, as I’m trying to troubleshoot issues with basic K3S setup with (default) Traefik-based ingress. I’m using K3S locally on my laptop running Fedora 35 (Fedora Linux 35 (Workstation Edition) x86_64, Kernel: 5.17.5-200.fc35.x86_64).
So the setup I’m trying to get to run is dead simple: single deployment with default Apache server, single service with ClusterIP and Ingress pointing to the service.
Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: apache-hello
spec:
replicas: 1
selector:
matchLabels:
app: apache-hello
template:
metadata:
labels:
app: apache-hello
name: apache-hello
spec:
containers:
- name: apache-hello
image: httpd
ports:
- containerPort: 80
Service:
apiVersion: v1
kind: Service
metadata:
name: apache-hello-service
labels:
run: apache-hello
spec:
selector:
app: apache-hello
ports:
- protocol: TCP
port: 80
These two are running just fine:
> kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system local-path-provisioner-6c79684f77-wmtss 1/1 Running 0 39m
kube-system coredns-d76bd69b-p62sj 1/1 Running 0 39m
kube-system helm-install-traefik-crd-djjsb 0/1 Completed 0 39m
kube-system helm-install-traefik-j95sj 0/1 Completed 1 39m
kube-system svclb-traefik-4wqgs 2/2 Running 0 38m
kube-system metrics-server-7cd5fcb6b7-bcpck 1/1 Running 0 39m
kube-system traefik-df4ff85d6-zx87n 1/1 Running 0 38m
default apache-hello-5876f688c9-8rqrg 1/1 Running 0 33m
Deployment running fine:
> kubectl describe deployments.apps/apache-hello
Name: apache-hello
Namespace: default
CreationTimestamp: Mon, 09 May 2022 08:44:19 +0200
Labels: <none>
Annotations: deployment.kubernetes.io/revision: 1
Selector: app=apache-hello
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=apache-hello
name=apache-hello
Containers:
apache-hello:
Image: httpd
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: apache-hello-5876f688c9 (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 33m deployment-controller Scaled up replica set apache-hello-5876f688c9 to 1
So is service:
> kubectl describe services/apache-hello-service
Name: apache-hello-service
Namespace: default
Labels: run=apache-hello
Annotations: <none>
Selector: app=apache-hello
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.43.31.41
IPs: 10.43.31.41
Port: <unset> 80/TCP
TargetPort: 80/TCP
Endpoints: 10.42.0.9:80
Session Affinity: None
Events: <none>
I can connect to the service directly using ClusterIP (running K3S locally, I’m actually on the node itself):
> curl -v http://10.43.31.41:80
* Trying 10.43.31.41:80...
* Connected to 10.43.31.41 (10.43.31.41) port 80 (#0)
> GET / HTTP/1.1
> Host: 10.43.31.41
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Mon, 09 May 2022 07:20:19 GMT
< Server: Apache/2.4.53 (Unix)
< Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
< ETag: "2d-432a5e4a73a80"
< Accept-Ranges: bytes
< Content-Length: 45
< Content-Type: text/html
<
<html><body><h1>It works!</h1></body></html>
* Connection #0 to host 10.43.31.41 left intact
I have also tested that port-forward option of kubectl:
> kubectl port-forward services/apache-hello-service 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
Handling connection for 8080
Handling connection for 8080
> curl -v http://localhost:8080
* Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Mon, 09 May 2022 07:21:27 GMT
< Server: Apache/2.4.53 (Unix)
< Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
< ETag: "2d-432a5e4a73a80"
< Accept-Ranges: bytes
< Content-Length: 45
< Content-Type: text/html
<
<html><body><h1>It works!</h1></body></html>
* Connection #0 to host localhost left intact
I can also check Apache logs:
> kubectl logs deployments/apache-hello
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.42.0.9. Set the 'ServerName' directive globally to suppress this message
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.42.0.9. Set the 'ServerName' directive globally to suppress this message
[Mon May 09 06:44:27.522342 2022] [mpm_event:notice] [pid 1:tid 140417909382464] AH00489: Apache/2.4.53 (Unix) configured -- resuming normal operations
[Mon May 09 06:44:27.522458 2022] [core:notice] [pid 1:tid 140417909382464] AH00094: Command line: 'httpd -D FOREGROUND'
10.42.0.1 - - [09/May/2022:06:45:17 +0000] "GET / HTTP/1.1" 200 45
10.42.0.1 - - [09/May/2022:06:45:58 +0000] "GET / HTTP/1.1" 200 45
127.0.0.1 - - [09/May/2022:07:10:59 +0000] "GET / HTTP/1.1" 200 45
127.0.0.1 - - [09/May/2022:07:11:28 +0000] "GET / HTTP/1.1" 200 45
10.42.0.1 - - [09/May/2022:07:20:19 +0000] "GET / HTTP/1.1" 200 45
127.0.0.1 - - [09/May/2022:07:21:27 +0000] "GET / HTTP/1.1" 200 45
So yeah, I’m positive my Service and Deployment are working just fine. Now, when I try to add Ingress to the picture:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: apache-hello-ingress
annotations:
kubernetes.io/ingress.class: "traefik"
spec:
rules:
- host:
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: apache-hello-service
port:
number: 80
Installation seems to succeed:
> kubectl get ingress/apache-hello-ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
apache-hello-ingress <none> * 192.168.1.239 80 37m
> kubectl describe ingress/apache-hello-ingress
Name: apache-hello-ingress
Labels: <none>
Namespace: default
Address: 192.168.1.239
Ingress Class: <none>
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
*
/ apache-hello-service:80 (10.42.0.9:80)
Annotations: kubernetes.io/ingress.class: traefik
Events: <none>
But the problem is that whenever I try to use the ingress, I get “Bad Gateway” response like so:
> curl -v http://localhost:80
* Trying 127.0.0.1:80...
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 502 Bad Gateway
< Date: Mon, 09 May 2022 07:25:18 GMT
< Content-Length: 11
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host localhost left intact
Bad Gateway
I can’t see anything in the logs:
> kubectl -n kube-system logs deployments/traefik
time="2022-05-09T06:39:04Z" level=info msg="Configuration loaded from flags."
kubectl -n kube-system logs daemonsets/svclb-traefik
Defaulted container "lb-tcp-80" out of: lb-tcp-80, lb-tcp-443
+ trap exit TERM INT
+ echo 10.43.74.176
+ grep -Eq :
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 1 '!=' 1 ]
+ iptables -t nat -I PREROUTING '!' -s 10.43.74.176/32 -p TCP --dport 80 -j DNAT --to 10.43.74.176:80
+ iptables -t nat -I POSTROUTING -d 10.43.74.176/32 -p TCP -j MASQUERADE
+ '[' '!' -e /pause ]
+ mkfifo /pause
This is what I can see in the description of Traefik service:
> kubectl -n kube-system describe services/traefik
Name: traefik
Namespace: kube-system
Labels: app.kubernetes.io/instance=traefik
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=traefik
helm.sh/chart=traefik-10.19.300
Annotations: meta.helm.sh/release-name: traefik
meta.helm.sh/release-namespace: kube-system
Selector: app.kubernetes.io/instance=traefik,app.kubernetes.io/name=traefik
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.43.74.176
IPs: 10.43.74.176
LoadBalancer Ingress: 192.168.1.239
Port: web 80/TCP
TargetPort: web/TCP
NodePort: web 30976/TCP
Endpoints: 10.42.0.8:8000
Port: websecure 443/TCP
TargetPort: websecure/TCP
NodePort: websecure 31975/TCP
Endpoints: 10.42.0.8:8443
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal UpdatedIngressIP 48m svccontroller LoadBalancer Ingress IP addresses updated: 192.168.1.239
Normal AppliedDaemonSet 47m (x6 over 48m) svccontroller Applied LoadBalancer DaemonSet kube-system/svclb-traefik
What is weird, I can’t connect using either of the IPs, names or ports (even NodePorts):
curl http://192.168.1.239:80
Bad Gateway
curl http://192.168.1.239:80/
Bad Gateway
curl http://10.43.74.176:80/
Bad Gateway
curl http://192.168.1.239:30976
Bad Gateway
curl http://localhost:30976
Bad Gateway
Do you have any idea what I might be doing wrong? All suggestions are welcome and any help appreciated!