L7 Ingress can't reach service/pod

Setup:

  • Dedicated rancher node
  • 1 etcd/controlplane/worker node
  • 1 worker node

Default cluster config

addon_job_timeout: 30
authentication: 
  strategy: "x509"
bastion_host: 
  ssh_agent_auth: false
ignore_docker_version: true
ingress: 
  provider: "nginx"
kubernetes_version: "v1.11.2-rancher1-1"
monitoring: 
  provider: "metrics-server"
network: 
  plugin: "canal" 
services: 
  etcd: 
    extra_args: 
      election-timeout: "5000"
      heartbeat-interval: "500"
    snapshot: false
  kube-api: 
    pod_security_policy: false
  kubelet: 
    fail_swap_on: false
ssh_agent_auth: false

Pods exposing port 3000. Service remapping from 80 to 3000. I can reach the service via port-forward:

 λ kubectl port-forward svc/mastodon-web 8082:80
Forwarding from 127.0.0.1:8082 -> 3000
Forwarding from [::1]:8082 -> 3000
Handling connection for 8082

Ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    field.cattle.io/creatorId: user-955kz
    field.cattle.io/ingressState: '{"azhzLWRlYWRpbnNpLmRl":"mastodon:k8s-deadinsi.de","bWFzdG9kb24vbWFzdG9kb24vazhzLmRlYWRpbnNpLmRlLy8vODA=":""}'
    field.cattle.io/publicEndpoints: '[{"addresses":["95.216.190.83"],"port":80,"protocol":"HTTP","serviceName":"mastodon:mastodon-web","ingressName":"mastodon:mastodon","hostname":"k8s.deadinsi.de","path":"/","allNodes":true}]'
  creationTimestamp: 2018-10-05T12:26:22Z
  generation: 3
  name: mastodon
  namespace: mastodon
  resourceVersion: "7682"
  selfLink: /apis/extensions/v1beta1/namespaces/mastodon/ingresses/mastodon
  uid: deb4db3a-c899-11e8-8009-9600001193e0
spec:
  rules:
  - host: k8s.deadinsi.de
    http:
      paths:
      - backend:
          serviceName: mastodon-web
          servicePort: 80
        path: /
  tls:
  - secretName: k8s-deadinsi.de
status:
  loadBalancer:
    ingress:
    - ip: 95.216.190.83
    - ip: 95.216.194.226

TLS works fine. Nginx logs showing connect errors / 502 returned to user (check it, it’s up right now). Here’s the upstream block the ingress writes. It’s the single pod and not the service, but I assume that’s an upstream optimization to not hit the service iptables all the time.

        upstream mastodon-mastodon-web-80 {
                least_conn;

                keepalive 32;

                server 10.42.0.16:3000 max_fails=0 fail_timeout=0;

        }

From the container:

www-data@node-1:/etc/nginx$ curl 10.42.0.16:3000
curl: (7) Failed to connect to 10.42.0.16 port 3000: Connection refused
www-data@node-1:/etc/nginx$ ping 10.42.0.16
PING 10.42.0.16 (10.42.0.16) 56(84) bytes of data.
64 bytes from 10.42.0.16: icmp_seq=1 ttl=64 time=0.146 ms
64 bytes from 10.42.0.16: icmp_seq=2 ttl=64 time=0.083 ms
^C
--- 10.42.0.16 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1012ms
rtt min/avg/max/mdev = 0.083/0.114/0.146/0.033 ms
www-data@node-1:/etc/nginx$ ip route
default via 172.31.1.1 dev eth0
10.42.0.2 dev cali53e002be814 scope link
10.42.0.3 dev cali2c0c49f47ec scope link
10.42.0.4 dev calic5797bb4fe3 scope link
10.42.0.5 dev cali7ed0d6067ab scope link
10.42.0.6 dev cali97035713169 scope link
10.42.0.7 dev cali48cec6d5d0e scope link
10.42.0.8 dev cali3ed8bd671ff scope link
10.42.0.9 dev cali09770b813dc scope link
10.42.0.11 dev cali396f02a609b scope link
10.42.0.12 dev cali850651bb576 scope link
10.42.0.16 dev calif8dbbfd640b scope link
10.42.0.17 dev caliadd39653534 scope link
10.42.1.0/24 via 10.42.1.0 dev flannel.1 onlink
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.31.1.1 dev eth0 scope link

At this point I’m clueless. I’ve only ever worked with from-scratch clusters in a private network running flannel.

Hi, i’m also having issues with ingress returning 502 for service - i’m hosted on gcp though.

Did you ever find the solution to this?

Yes, I set up my own Layer 3 VPN (Wireguard) and used flannel instead. Any other network provider has never properly worked for me.