Kube-flannel fails on multiple nodes

Problems

  • Kube-flannel exist with error Failed to create SubnetManager: error retrieving pod spec for 'kube-system/kube-flannel-mbg4f': Get https://10.43.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-mbg4f: dial tcp 10.43.0.1:443: i/o timeout multiple times on multi special nodes

  • nginx-ingress-controller exit on the same nodes where kube-flannel failed

Every 2.0s: kubectl get pod --all-namespaces -o wide                                                                                                                                                                 Tue Apr 17 09:10:43 2018

NAMESPACE       NAME                                      READY     STATUS             RESTARTS   AGE       IP            NODE
cattle-system   cattle-cluster-agent-5c98479d57-cjttp     1/1       Running            0          16h       10.42.0.5     ceph4
cattle-system   cattle-node-agent-7xtq6                   1/1       Running            0          16h       10.0.1.13     ceph3
cattle-system   cattle-node-agent-8r4rj                   1/1       Running            0          16h       172.18.1.10   labmanager1
cattle-system   cattle-node-agent-8rf6n                   1/1       Running            0          29m       10.0.1.12     ceph2
cattle-system   cattle-node-agent-d8w7d                   1/1       Running            0          7m        10.0.1.11     ceph1
cattle-system   cattle-node-agent-rwd5k                   1/1       Running            0          16h       10.0.1.14     ceph4
ingress-nginx   default-http-backend-565bc99d5b-pxg5t     1/1       Running            0          16h       10.42.0.4     ceph4
ingress-nginx   nginx-ingress-controller-6hxct            0/1       CrashLoopBackOff   6          7m        10.0.1.11     ceph1
ingress-nginx   nginx-ingress-controller-872l2            0/1       CrashLoopBackOff   12         29m       10.0.1.12     ceph2
ingress-nginx   nginx-ingress-controller-8b5s7            0/1       CrashLoopBackOff   338        16h       10.0.1.13     ceph3
ingress-nginx   nginx-ingress-controller-k248p            1/1       Running            0          16h       10.0.1.14     ceph4
ingress-nginx   nginx-ingress-controller-vljbh            1/1       Running            0          16h       172.18.1.10   labmanager1
kube-system     kube-dns-6cc4f65d44-2874s                 3/3       Running            0          7m        10.42.2.2     ceph3
kube-system     kube-dns-6cc4f65d44-4wf5d                 3/3       Running            0          16h       10.42.0.2     ceph4
kube-system     kube-dns-autoscaler-6488788c4c-k5vjr      1/1       Running            0          16h       10.42.0.3     ceph4
kube-system     kube-flannel-4gx5j                        1/2       CrashLoopBackOff   9          29m       10.0.1.12     ceph2
kube-system     kube-flannel-7bc5l                        1/2       CrashLoopBackOff   181        16h       10.0.1.13     ceph3
kube-system     kube-flannel-mbg4f                        1/2       CrashLoopBackOff   5          7m        10.0.1.11     ceph1
kube-system     kube-flannel-rbhh4                        2/2       Running            0          16h       10.0.1.14     ceph4
kube-system     kube-flannel-skn5w                        2/2       Running            0          16h       172.18.1.10   labmanager1
kube-system     rke-ingress-controller-deploy-job-dcn4x   0/1       Completed          0          16h       10.0.1.14     ceph4
kube-system     rke-kubedns-addon-deploy-job-zgl87        0/1       Completed          0          16h       10.0.1.14     ceph4
kube-system     rke-network-plugin-deploy-job-hmtpq       0/1       Completed          0          16h       10.0.1.14     ceph4
# k8s node info 
ubuntu@labmanager1:~$ kubectl get node
NAME          STATUS    ROLES                      AGE       VERSION
ceph1         Ready     worker                     8m        v1.9.5-rancher1
ceph2         Ready     worker                     30m       v1.9.5-rancher1
ceph3         Ready     worker                     16h       v1.9.5-rancher1
ceph4         Ready     controlplane,etcd,worker   16h       v1.9.5-rancher1
labmanager1   Ready     worker                     16h       v1.9.5-rancher1

Useful Info
Versions Rancher v2.0.0-beta3 UI: v2.0.35
Access local admin
Route authenticated.cluster.nodes.index