Cannot see container logs in Rancher UI

We have a RKE deployed kubernetes cluster. We had a bit of a mishap with the rancher 2.4.8 web server certificate as it expired. We managed to bring back the rancher-ui and also connected to the cluster as the kube-api docker container came back on the master node. That was another issue but long story short we got that part resolved.

Anyway we are facing another issue . We cannot see the logs in the container.
If we go to a pod and click on View logs, the window appears but it stays initializing and then disconnected.

So I did a bit of digging and did a kubectl to the app.

This is an example of the command i run:

kubectl logs redacted-image-6d4b66b659-8jhkf -n temp

and i got this error:

kubectl : Error from server: Get https://<redacted>:10250/
containerLogs/temp/redacted-image: dial tcp redacted:10250: i/o timeout

Both Worker nodes are exhibiting this error.

I did a test on the master node to check to see if there was an issue. I looked at the pods in cattle-system and I can see the logs of the containers sitting on the master server.

Has anyone come across this before? I can provide further information if needed.

Any help is most appreciated.

We managed to fix this by opening up 10250 port through iptables

If that wasn’t already open, you might want to check Rancher Docs: Port Requirements to make sure you have all the ports open you need.