RKE2 ingress 502, seems like pod connectivity issue

Rancher 2.x
1

Hi all.

Summary:
I installed rke2 on a single node running CentOS Stream 9 with all default config options except for selinux: true. I deployed a whoami service for testing. From the node, I can curl the pod’s IP address and see the correct response. With the default ingress controller, I get 502 bad gateway responses. Looking in the ingress controller logs, the error is
2023/06/28 17:08:06 [error] 445#445: *109803 connect() failed (113: No route to host) while connecting to upstream, client: x.x.x.x, server: mydomain.com, request: "GET / HTTP/1.1", upstream: "http://10.42.39.147:80/", host: "mydomain.com"

If I then create another pod and try to curl the whoami pod from inside of it, it is unable to reach it, which seems to me that perhaps pods in general aren’t able to communicate with each other. Strangely, it seems that the whoami pod is reachable by other pods via its ClusterIP service but not via its IP (see details below). There are no NetworkPolicies. I tried reinstalling with calico instead of canal and had the same problem. I also tried adding /etc/NetworkManager/conf.d/rke2-canal.conf as described in Known Issues and Limitations | RKE 2 and this had no effect. Any ideas? Thanks :slight_smile:

Details:

whoami manifest:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: whoami
  labels:
    app: whoami
spec:
  replicas: 1
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
      - name: whoami
        image: docker.io/containous/whoami
        ports:
        - containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: whoami
spec:
  ports:
  - name: http
    targetPort: 80
    port: 80
  selector:
    app: whoami

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: whoami
spec:
  rules:
  - host: mydomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: whoami
            port: 
              number: 80

Manifest for another pod to try to curl the whoami pod

apiVersion: v1
kind: Pod
metadata:
  name: curler
spec:
  securityContext:
    runAsUser: 0
  containers:
    - name: curler
      image: docker.io/curlimages/curl
      command: ["sleep", "100000"]
      securityContext:
        privileged: true

Attempts:

$ kubectl get pods -o wide
> NAME                      READY   STATUS    RESTARTS      AGE    IP             NODE   NOMINATED NODE   READINESS GATES
> curler                    1/1     Running   1 (49m ago)   172m   10.42.39.150   xxxx   <none>           <none>
> whoami-54c68c4767-fpvc7   1/1     Running   1 (49m ago)   172m   10.42.39.147   xxxx   <none>           <none>
$ kubectl exec curler -- curl http://10.42.39.147
> curl: (7) Failed to connect to 10.42.39.147 port 80 after 0 ms: Couldn't connect to server
$ kubectl exec curler -- curl https://forums.rancher.com
> #normal html from this site
$ kubectl get svc
> NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
> kubernetes   ClusterIP   10.43.0.1      <none>        443/TCP   3h6m
> whoami       ClusterIP   10.43.58.126   <none>        80/TCP    174m
$ kubectl exec curler -- curl http://10.43.58.126
> #normal response from whoami
$ curl http://10.42.39.147
> #normal response from whoami
2

Update:
I get the same behavior if I disable SELinux sudo setenforce 0 and install with selinux: false, as well as if I disable NetworkManager.