RKE cluster: kubectl logs 'connect: no route to host' (Resolved)

I have a fresh cluster created by RKE:

 $ kubectl get nodes -o wide
NAME            STATUS   ROLES          AGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME   Ready    etcd           107s   v1.13.5   <none>        CentOS Linux 7 (Core)   3.10.0-957.10.1.el7.x86_64   docker://18.9.2   Ready    controlplane   109s   v1.13.5   <none>        CentOS Linux 7 (Core)   3.10.0-957.10.1.el7.x86_64   docker://18.9.2     Ready    worker         108s   v1.13.5     <none>        CentOS Linux 7 (Core)   3.10.0-957.10.1.el7.x86_64   docker://18.9.2

$ kubectl get pods -o wide --all-namespaces
    NAMESPACE       NAME                                      READY   STATUS      RESTARTS   AGE    IP              NODE            NOMINATED NODE   READINESS GATES
    ingress-nginx   default-http-backend-7f8fbb85db-sj4th     1/1     Running     0          93s     <none>           <none>
    ingress-nginx   nginx-ingress-controller-s9nrt            1/1     Running     0          93s     <none>           <none>
    kube-system     canal-5758r                               2/2     Running     0          112s     <none>           <none>
    kube-system     canal-5j7jx                               2/2     Running     0          112s   <none>           <none>
    kube-system     canal-95r9m                               2/2     Running     0          112s   <none>           <none>
    kube-system     kube-dns-5fd74c7488-rtpbf                 3/3     Running     0          103s     <none>           <none>
    kube-system     kube-dns-autoscaler-c89df977f-jm9rd       1/1     Running     0          102s     <none>           <none>
    kube-system     metrics-server-7fbd549b78-mhww9           1/1     Running     0          98s     <none>           <none>
    kube-system     rke-ingress-controller-deploy-job-7mjhg   0/1     Completed   0          97s   <none>           <none>
    kube-system     rke-kubedns-addon-deploy-job-56vmn        0/1     Completed   0          107s   <none>           <none>
    kube-system     rke-metrics-addon-deploy-job-vql2d        0/1     Completed   0          102s   <none>           <none>
    kube-system     rke-network-plugin-deploy-job-6c2fm       0/1     Completed   0          117s   <none>           <none>

Everything seems to be fine, however:

kubectl logs -f nginx-ingress-controller-s9nrt -n=ingress-nginx
Error from server: Get dial tcp connect: no route to host

- address:
user: rancher
- controlplane
- address:
user: rancher
- etcd
- address:
user: rancher
- worker

ignore_docker_version: true

  mode: rbac

  provider: nginx

      kubelet-preferred-address-types: "InternalIP,ExternalIP,Hostname"
       fail-swap-on: false

I’m puzzled and not sure where to look for a problem, any pointer will be greatly appreciated.

UPDATE: found this in kubelet log:
I0423 21:07:24.846609 7158 log.go:172] http: TLS handshake error from remote error: tls: unknown certificate authority

this didn’t helped:
$ rm -rf /var/lib/kubelet/pki/*

RESOLVED: Disabling os firewall on the worker node (enabled by default by the CentOS installer) solved the issue… duh

1 Like