When i try to create a HA cluster using rke , it fails around the point of creating the kubelet for the nodes.
But at this stage , the docker process on my cluster node hangs and I am unable to run ANY docker commands .
Even reboot/removing docker package from the server and re-installing docker doesnt help.
[campaigner@yowcmp-dply01 ~]$ ./rke_linux-amd64 up
INFO[0000] Running RKE version: v1.1.13
INFO[0000] Initiating Kubernetes cluster
INFO[0000] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates
INFO[0000] [certificates] Generating admin certificates and kubeconfig
INFO[0000] Successfully Deployed state file at [./cluster.rkestate]
INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [10.96.190.101]
INFO[0000] [network] Deploying port listener containers
INFO[0000] [network] Starting stopped container [rke-etcd-port-listener] on host [10.96.190.101]
INFO[0000] Starting container [rke-etcd-port-listener] on host [10.96.190.101], try #1
INFO[0000] [network] Successfully started [rke-etcd-port-listener] container on host [10.96.190.101]
INFO[0000] Image [rancher/rke-tools:v0.1.68] exists on host [10.96.190.101]
INFO[0000] Starting container [rke-cp-port-listener] on host [10.96.190.101], try #1
INFO[0000] [network] Successfully started [rke-cp-port-listener] container on host [10.96.190.101]
INFO[0000] Image [rancher/rke-tools:v0.1.68] exists on host [10.96.190.101]
INFO[0001] Starting container [rke-worker-port-listener] on host [10.96.190.101], try #1
INFO[0001] [network] Successfully started [rke-worker-port-listener] container on host [10.96.190.101]
INFO[0001] [network] Port listener containers deployed successfully
INFO[0001] [network] Running control plane -> etcd port checks
INFO[0001] Image [rancher/rke-tools:v0.1.68] exists on host [10.96.190.101]
INFO[0001] Starting container [rke-port-checker] on host [10.96.190.101], try #1
INFO[0001] [network] Successfully started [rke-port-checker] container on host [10.96.190.101]
INFO[0001] Removing container [rke-port-checker] on host [10.96.190.101], try #1
INFO[0001] [network] Running control plane -> worker port checks
INFO[0001] Image [rancher/rke-tools:v0.1.68] exists on host [10.96.190.101]
INFO[0001] Starting container [rke-port-checker] on host [10.96.190.101], try #1
INFO[0001] [network] Successfully started [rke-port-checker] container on host [10.96.190.101]
INFO[0001] Removing container [rke-port-checker] on host [10.96.190.101], try #1
INFO[0001] [network] Running workers -> control plane port checks
INFO[0001] Image [rancher/rke-tools:v0.1.68] exists on host [10.96.190.101]
INFO[0001] Starting container [rke-port-checker] on host [10.96.190.101], try #1
INFO[0002] [network] Successfully started [rke-port-checker] container on host [10.96.190.101]
INFO[0002] Removing container [rke-port-checker] on host [10.96.190.101], try #1
INFO[0002] [network] Checking KubeAPI port Control Plane hosts
INFO[0002] [network] Removing port listener containers
INFO[0002] Removing container [rke-etcd-port-listener] on host [10.96.190.101], try #1
INFO[0002] [remove/rke-etcd-port-listener] Successfully removed container on host [10.96.190.101]
INFO[0002] Removing container [rke-cp-port-listener] on host [10.96.190.101], try #1
INFO[0002] [remove/rke-cp-port-listener] Successfully removed container on host [10.96.190.101]
INFO[0002] Removing container [rke-worker-port-listener] on host [10.96.190.101], try #1
INFO[0002] [remove/rke-worker-port-listener] Successfully removed container on host [10.96.190.101]
INFO[0002] [network] Port listener containers removed successfully
INFO[0002] [certificates] Deploying kubernetes certificates to Cluster nodes
INFO[0002] Checking if container [cert-deployer] is running on host [10.96.190.101], try #1
INFO[0002] Image [rancher/rke-tools:v0.1.68] exists on host [10.96.190.101]
INFO[0002] Starting container [cert-deployer] on host [10.96.190.101], try #1
INFO[0002] Checking if container [cert-deployer] is running on host [10.96.190.101], try #1
INFO[0007] Checking if container [cert-deployer] is running on host [10.96.190.101], try #1
INFO[0007] Removing container [cert-deployer] on host [10.96.190.101], try #1
INFO[0007] [reconcile] Rebuilding and updating local kube config
INFO[0007] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
INFO[0007] [certificates] Successfully deployed kubernetes certificates to Cluster nodes
INFO[0007] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.96.190.101]
INFO[0007] Image [rancher/rke-tools:v0.1.68] exists on host [10.96.190.101]
INFO[0007] Starting container [file-deployer] on host [10.96.190.101], try #1
INFO[0008] Successfully started [file-deployer] container on host [10.96.190.101]
INFO[0008] Waiting for [file-deployer] container to exit on host [10.96.190.101]
INFO[0008] Waiting for [file-deployer] container to exit on host [10.96.190.101]
INFO[0008] Container [file-deployer] is still running on host [10.96.190.101]: stderr: [], stdout: []
INFO[0009] Waiting for [file-deployer] container to exit on host [10.96.190.101]
INFO[0009] Removing container [file-deployer] on host [10.96.190.101], try #1
INFO[0009] [remove/file-deployer] Successfully removed container on host [10.96.190.101]
INFO[0009] [/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes
INFO[0009] [reconcile] Reconciling cluster state
INFO[0009] [reconcile] This is newly generated cluster
INFO[0009] Pre-pulling kubernetes images
INFO[0009] Pulling image [rancher/hyperkube:v1.18.14-rancher1] on host [10.96.190.101], try #1
INFO[0041] Image [rancher/hyperkube:v1.18.14-rancher1] exists on host [10.96.190.101]
INFO[0041] Kubernetes images pulled successfully
INFO[0041] [etcd] Building up etcd plane…
INFO[0041] Image [rancher/rke-tools:v0.1.68] exists on host [10.96.190.101]
INFO[0042] Starting container [etcd-fix-perm] on host [10.96.190.101], try #1
INFO[0042] Successfully started [etcd-fix-perm] container on host [10.96.190.101]
INFO[0042] Waiting for [etcd-fix-perm] container to exit on host [10.96.190.101]
INFO[0042] Waiting for [etcd-fix-perm] container to exit on host [10.96.190.101]
INFO[0042] Container [etcd-fix-perm] is still running on host [10.96.190.101]: stderr: [], stdout: []
INFO[0043] Waiting for [etcd-fix-perm] container to exit on host [10.96.190.101]
INFO[0043] Removing container [etcd-fix-perm] on host [10.96.190.101], try #1
INFO[0043] [remove/etcd-fix-perm] Successfully removed container on host [10.96.190.101]
INFO[0043] Pulling image [rancher/coreos-etcd:v3.4.3-rancher1] on host [10.96.190.101], try #1
INFO[0046] Image [rancher/coreos-etcd:v3.4.3-rancher1] exists on host [10.96.190.101]
INFO[0047] Starting container [etcd] on host [10.96.190.101], try #1
INFO[0047] [etcd] Successfully started [etcd] container on host [10.96.190.101]
INFO[0047] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.96.190.101]
INFO[0047] Image [rancher/rke-tools:v0.1.68] exists on host [10.96.190.101]
INFO[0047] Starting container [etcd-rolling-snapshots] on host [10.96.190.101], try #1
INFO[0047] [etcd] Successfully started [etcd-rolling-snapshots] container on host [10.96.190.101]
INFO[0052] Image [rancher/rke-tools:v0.1.68] exists on host [10.96.190.101]
INFO[0052] Starting container [rke-bundle-cert] on host [10.96.190.101], try #1
INFO[0052] [certificates] Successfully started [rke-bundle-cert] container on host [10.96.190.101]
INFO[0052] Waiting for [rke-bundle-cert] container to exit on host [10.96.190.101]
INFO[0053] Container [rke-bundle-cert] is still running on host [10.96.190.101]: stderr: [], stdout: []
INFO[0054] Waiting for [rke-bundle-cert] container to exit on host [10.96.190.101]
INFO[0054] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.96.190.101]
INFO[0054] Removing container [rke-bundle-cert] on host [10.96.190.101], try #1
INFO[0054] Image [rancher/rke-tools:v0.1.68] exists on host [10.96.190.101]
INFO[0054] Starting container [rke-log-linker] on host [10.96.190.101], try #1
INFO[0054] [etcd] Successfully started [rke-log-linker] container on host [10.96.190.101]
INFO[0054] Removing container [rke-log-linker] on host [10.96.190.101], try #1
INFO[0054] [remove/rke-log-linker] Successfully removed container on host [10.96.190.101]
INFO[0054] [etcd] Successfully started etcd plane… Checking etcd cluster health
INFO[0054] [etcd] etcd host [10.96.190.101] reported healthy=true
INFO[0054] [controlplane] Building up Controller Plane…
INFO[0054] Checking if container [service-sidekick] is running on host [10.96.190.101], try #1
INFO[0054] Image [rancher/rke-tools:v0.1.68] exists on host [10.96.190.101]
INFO[0054] Image [rancher/hyperkube:v1.18.14-rancher1] exists on host [10.96.190.101]
INFO[0055] Starting container [kube-apiserver] on host [10.96.190.101], try #1
INFO[0055] [controlplane] Successfully started [kube-apiserver] container on host [10.96.190.101]
INFO[0055] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.96.190.101]
INFO[0065] [healthcheck] service [kube-apiserver] on host [10.96.190.101] is healthy
INFO[0065] Image [rancher/rke-tools:v0.1.68] exists on host [10.96.190.101]
INFO[0065] Starting container [rke-log-linker] on host [10.96.190.101], try #1
INFO[0065] [controlplane] Successfully started [rke-log-linker] container on host [10.96.190.101]
INFO[0065] Removing container [rke-log-linker] on host [10.96.190.101], try #1
INFO[0065] [remove/rke-log-linker] Successfully removed container on host [10.96.190.101]
INFO[0066] Image [rancher/hyperkube:v1.18.14-rancher1] exists on host [10.96.190.101]
INFO[0066] Starting container [kube-controller-manager] on host [10.96.190.101], try #1
INFO[0066] [controlplane] Successfully started [kube-controller-manager] container on host [10.96.190.101]
INFO[0066] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.96.190.101]
INFO[0072] [healthcheck] service [kube-controller-manager] on host [10.96.190.101] is healthy
INFO[0072] Image [rancher/rke-tools:v0.1.68] exists on host [10.96.190.101]
INFO[0072] Starting container [rke-log-linker] on host [10.96.190.101], try #1
INFO[0072] [controlplane] Successfully started [rke-log-linker] container on host [10.96.190.101]
INFO[0072] Removing container [rke-log-linker] on host [10.96.190.101], try #1
INFO[0072] [remove/rke-log-linker] Successfully removed container on host [10.96.190.101]
INFO[0072] Image [rancher/hyperkube:v1.18.14-rancher1] exists on host [10.96.190.101]
INFO[0072] Starting container [kube-scheduler] on host [10.96.190.101], try #1
INFO[0072] [controlplane] Successfully started [kube-scheduler] container on host [10.96.190.101]
INFO[0072] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.96.190.101]
INFO[0078] [healthcheck] service [kube-scheduler] on host [10.96.190.101] is healthy
INFO[0078] Image [rancher/rke-tools:v0.1.68] exists on host [10.96.190.101]
INFO[0078] Starting container [rke-log-linker] on host [10.96.190.101], try #1
INFO[0079] [controlplane] Successfully started [rke-log-linker] container on host [10.96.190.101]
INFO[0079] Removing container [rke-log-linker] on host [10.96.190.101], try #1
INFO[0079] [remove/rke-log-linker] Successfully removed container on host [10.96.190.101]
INFO[0079] [controlplane] Successfully started Controller Plane…
INFO[0079] [authz] Creating rke-job-deployer ServiceAccount
INFO[0079] [authz] rke-job-deployer ServiceAccount created successfully
INFO[0079] [authz] Creating system:node ClusterRoleBinding
INFO[0079] [authz] system:node ClusterRoleBinding created successfully
INFO[0079] [authz] Creating kube-apiserver proxy ClusterRole and ClusterRoleBinding
INFO[0079] [authz] kube-apiserver proxy ClusterRole and ClusterRoleBinding created successfully
INFO[0079] Successfully Deployed state file at [./cluster.rkestate]
INFO[0079] [state] Saving full cluster state to Kubernetes
INFO[0079] [state] Successfully Saved full cluster state to Kubernetes ConfigMap: full-cluster-state
INFO[0079] [worker] Building up Worker Plane…
INFO[0079] Checking if container [service-sidekick] is running on host [10.96.190.101], try #1
INFO[0079] [sidekick] Sidekick container already created on host [10.96.190.101]
INFO[0079] Image [rancher/hyperkube:v1.18.14-rancher1] exists on host [10.96.190.101]
INFO[0079] Starting container [kubelet] on host [10.96.190.101], try #1
WARN[0129] Can’t start Docker container [kubelet] on host [10.96.190.101]: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
INFO[0129] Starting container [kubelet] on host [10.96.190.101], try #2
WARN[0179] Can’t start Docker container [kubelet] on host [10.96.190.101]: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
INFO[0179] Starting container [kubelet] on host [10.96.190.101], try #3
WARN[0230] Can’t start Docker container [kubelet] on host [10.96.190.101]: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
FATA[0230] [workerPlane] Failed to bring up Worker Plane: [Failed to start [kubelet] container on host [10.96.190.101]: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?]