RKE cluster: kubectl logs 'connect: no route to host' (Resolved)

Rancher
1

I have a fresh cluster created by RKE:

 $ kubectl get nodes -o wide
NAME            STATUS   ROLES          AGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
192.168.2.220   Ready    etcd           107s   v1.13.5   192.168.2.220   <none>        CentOS Linux 7 (Core)   3.10.0-957.10.1.el7.x86_64   docker://18.9.2
192.168.2.227   Ready    controlplane   109s   v1.13.5   192.168.2.227   <none>        CentOS Linux 7 (Core)   3.10.0-957.10.1.el7.x86_64   docker://18.9.2
192.168.2.7     Ready    worker         108s   v1.13.5   192.168.2.7     <none>        CentOS Linux 7 (Core)   3.10.0-957.10.1.el7.x86_64   docker://18.9.2

$ kubectl get pods -o wide --all-namespaces
    NAMESPACE       NAME                                      READY   STATUS      RESTARTS   AGE    IP              NODE            NOMINATED NODE   READINESS GATES
    ingress-nginx   default-http-backend-7f8fbb85db-sj4th     1/1     Running     0          93s    10.42.1.5       192.168.2.7     <none>           <none>
    ingress-nginx   nginx-ingress-controller-s9nrt            1/1     Running     0          93s    192.168.2.7     192.168.2.7     <none>           <none>
    kube-system     canal-5758r                               2/2     Running     0          112s   192.168.2.7     192.168.2.7     <none>           <none>
    kube-system     canal-5j7jx                               2/2     Running     0          112s   192.168.2.227   192.168.2.227   <none>           <none>
    kube-system     canal-95r9m                               2/2     Running     0          112s   192.168.2.220   192.168.2.220   <none>           <none>
    kube-system     kube-dns-5fd74c7488-rtpbf                 3/3     Running     0          103s   10.42.1.3       192.168.2.7     <none>           <none>
    kube-system     kube-dns-autoscaler-c89df977f-jm9rd       1/1     Running     0          102s   10.42.1.2       192.168.2.7     <none>           <none>
    kube-system     metrics-server-7fbd549b78-mhww9           1/1     Running     0          98s    10.42.1.4       192.168.2.7     <none>           <none>
    kube-system     rke-ingress-controller-deploy-job-7mjhg   0/1     Completed   0          97s    192.168.2.227   192.168.2.227   <none>           <none>
    kube-system     rke-kubedns-addon-deploy-job-56vmn        0/1     Completed   0          107s   192.168.2.227   192.168.2.227   <none>           <none>
    kube-system     rke-metrics-addon-deploy-job-vql2d        0/1     Completed   0          102s   192.168.2.227   192.168.2.227   <none>           <none>
    kube-system     rke-network-plugin-deploy-job-6c2fm       0/1     Completed   0          117s   192.168.2.227   192.168.2.227   <none>           <none>

Everything seems to be fine, however:

kubectl logs -f nginx-ingress-controller-s9nrt -n=ingress-nginx
Error from server: Get https://192.168.2.7:10250/containerLogs/ingress-nginx/nginx-ingress-controller-s9nrt/nginx-ingress-controller?follow=true: dial tcp 192.168.2.7:10250: connect: no route to host

cluster.yml:
nodes:
- address: 192.168.2.227
user: rancher
role:
- controlplane
- address: 192.168.2.220
user: rancher
role:
- etcd
- address: 192.168.2.7
user: rancher
role:
- worker

ignore_docker_version: true

authorization:
  mode: rbac

ingress:
  provider: nginx

services:
  kube-api:
    extra_args:
      kubelet-preferred-address-types: "InternalIP,ExternalIP,Hostname"
  kubelet:
    extra_args:
       fail-swap-on: false

I’m puzzled and not sure where to look for a problem, any pointer will be greatly appreciated.

UPDATE: found this in kubelet log:
I0423 21:07:24.846609 7158 log.go:172] http: TLS handshake error from 192.168.2.7:35786: remote error: tls: unknown certificate authority

this didn’t helped:
$ rm -rf /var/lib/kubelet/pki/*

RESOLVED: Disabling os firewall on the worker node (enabled by default by the CentOS installer) solved the issue… duh

1 Like