Hi all.
Summary:
I installed rke2 on a single node running CentOS Stream 9 with all default config options except for selinux: true
. I deployed a whoami service for testing. From the node, I can curl the pod’s IP address and see the correct response. With the default ingress controller, I get 502 bad gateway responses. Looking in the ingress controller logs, the error is
2023/06/28 17:08:06 [error] 445#445: *109803 connect() failed (113: No route to host) while connecting to upstream, client: x.x.x.x, server: mydomain.com, request: "GET / HTTP/1.1", upstream: "http://10.42.39.147:80/", host: "mydomain.com"
If I then create another pod and try to curl the whoami pod from inside of it, it is unable to reach it, which seems to me that perhaps pods in general aren’t able to communicate with each other. Strangely, it seems that the whoami pod is reachable by other pods via its ClusterIP service but not via its IP (see details below). There are no NetworkPolicies. I tried reinstalling with calico instead of canal and had the same problem. I also tried adding /etc/NetworkManager/conf.d/rke2-canal.conf as described in Known Issues and Limitations | RKE 2 and this had no effect. Any ideas? Thanks
Details:
whoami manifest:
kind: Deployment
apiVersion: apps/v1
metadata:
name: whoami
labels:
app: whoami
spec:
replicas: 1
selector:
matchLabels:
app: whoami
template:
metadata:
labels:
app: whoami
spec:
containers:
- name: whoami
image: docker.io/containous/whoami
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: whoami
spec:
ports:
- name: http
targetPort: 80
port: 80
selector:
app: whoami
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: whoami
spec:
rules:
- host: mydomain.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: whoami
port:
number: 80
Manifest for another pod to try to curl the whoami pod
apiVersion: v1
kind: Pod
metadata:
name: curler
spec:
securityContext:
runAsUser: 0
containers:
- name: curler
image: docker.io/curlimages/curl
command: ["sleep", "100000"]
securityContext:
privileged: true
Attempts:
$ kubectl get pods -o wide
> NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
> curler 1/1 Running 1 (49m ago) 172m 10.42.39.150 xxxx <none> <none>
> whoami-54c68c4767-fpvc7 1/1 Running 1 (49m ago) 172m 10.42.39.147 xxxx <none> <none>
$ kubectl exec curler -- curl http://10.42.39.147
> curl: (7) Failed to connect to 10.42.39.147 port 80 after 0 ms: Couldn't connect to server
$ kubectl exec curler -- curl https://forums.rancher.com
> #normal html from this site
$ kubectl get svc
> NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
> kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 3h6m
> whoami ClusterIP 10.43.58.126 <none> 80/TCP 174m
$ kubectl exec curler -- curl http://10.43.58.126
> #normal response from whoami
$ curl http://10.42.39.147
> #normal response from whoami