I have been trying to add a Network Policy to an application that I’m deploying in it’s own namespace. My network policy is only allowing Ingress from the namespace where the ingress controller is running and from pods with a label matching pods in the SAME namespace.
This mostly works but sometimes the ingress routes don’t work (504 gateway) error and I have narrowed down when they do and don’t work.
If the ingress controller pod selected to do the route is on the SAME NODE as the service I’m routing to, everything is fine (I get a valid response back from curling the ingress endpoint).
However, if the ingress controller pod selected to do the route is on a DIFFERENT NODE than the service I’m routing to, I get the 504 gateway error (instead of a service response back from curl).
If I remove the Network Policy, the routine ALWAYS works. Somehow the network policy is not working with the ingress controller when the controller selected is on a different node than the pod/service being routed to.
All of my network policies are just using label selectors for pods and namespaces and work fine so long as the ingress controller and pod being routed to are on the same node.
How can I overcome this?