We have a RKE deployed kubernetes cluster. We had a bit of a mishap with the rancher 2.4.8 web server certificate as it expired. We managed to bring back the rancher-ui and also connected to the cluster as the kube-api docker container came back on the master node. That was another issue but long story short we got that part resolved.
Anyway we are facing another issue . We cannot see the logs in the container.
If we go to a pod and click on View logs, the window appears but it stays initializing and then disconnected.
So I did a bit of digging and did a kubectl to the app.
This is an example of the command i run:
kubectl logs redacted-image-6d4b66b659-8jhkf -n temp
and i got this error:
kubectl : Error from server: Get https://<redacted>:10250/
containerLogs/temp/redacted-image: dial tcp redacted:10250: i/o timeout
Both Worker nodes are exhibiting this error.
I did a test on the master node to check to see if there was an issue. I looked at the pods in cattle-system and I can see the logs of the containers sitting on the master server.
Has anyone come across this before? I can provide further information if needed.
Any help is most appreciated.