RKE version: 0.2.5
Operating system and kernel: centos 7
Type/provider of hosts: openstack
Steps to Reproduce:
I am tyring to run rke -d up command. After some point, it gives health check error for etcd cluster. Error is like following:
INFO[0060] [etcd] Successfully started etcd plane… Checking etcd cluster health
DEBU[0060] [etcd] Check etcd cluster health
DEBU[0071] Failed to get /health for host [10.154.12.89]: Get https://172.16.100.40:2379/health: net/http: TLS handshake timeout
DEBU[0086] Failed to get /health for host [10.154.12.89]: Get https://172.16.100.40:2379/health: net/http: TLS handshake timeout
DEBU[0102] Failed to get /health for host [10.154.12.89]: Get https://172.16.100.40:2379/health: net/http: TLS handshake timeout
DEBU[0107] [etcd] Check etcd cluster health
DEBU[0117] Failed to get /health for host [10.154.12.79]: Get https://172.16.100.15:2379/health: net/http: TLS handshake timeout
DEBU[0133] Failed to get /health for host [10.154.12.79]: Get https://172.16.100.15:2379/health: net/http: TLS handshake timeout
DEBU[0148] Failed to get /health for host [10.154.12.79]: Get https://172.16.100.15:2379/health: net/http: TLS handshake timeout
DEBU[0153] [etcd] Check etcd cluster health
DEBU[0164] Failed to get /health for host [10.154.12.78]: Get https://172.16.100.28:2379/health: net/http: TLS handshake timeout
DEBU[0179] Failed to get /health for host [10.154.12.78]: Get https://172.16.100.28:2379/health: net/http: TLS handshake timeout
DEBU[0195] Failed to get /health for host [10.154.12.78]: Get https://172.16.100.28:2379/health: net/http: TLS handshake timeout
FATA[0200] [etcd] Failed to bring up Etcd Plane: [etcd] Etcd Cluster is not healthy
I also checked control node for etcd container logs and I found following:
2019-07-18 11:25:00.708898 I | etcdmain: rejected connection from “172.16.100.40:43876” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:00.728766 I | etcdmain: rejected connection from “172.16.100.28:44722” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:00.784241 I | etcdmain: rejected connection from “172.16.100.28:44726” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:00.796296 I | etcdmain: rejected connection from “172.16.100.40:43882” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:00.815147 I | etcdmain: rejected connection from “172.16.100.40:43884” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:00.821242 I | raft: d17ee7c963f68089 is starting a new election at term 374
2019-07-18 11:25:00.821261 I | raft: d17ee7c963f68089 became candidate at term 375
2019-07-18 11:25:00.821269 I | raft: d17ee7c963f68089 received MsgVoteResp from d17ee7c963f68089 at term 375
2019-07-18 11:25:00.821276 I | raft: d17ee7c963f68089 [logterm: 1, index: 3] sent MsgVote request to d9c6974366ce1ba at term 375
2019-07-18 11:25:00.821282 I | raft: d17ee7c963f68089 [logterm: 1, index: 3] sent MsgVote request to 9019c714a7a7fe3b at term 375
2019-07-18 11:25:00.833387 I | etcdmain: rejected connection from “172.16.100.28:44730” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:00.888661 I | etcdmain: rejected connection from “172.16.100.28:44734” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:00.900891 I | etcdmain: rejected connection from “172.16.100.40:43890” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:00.919489 I | etcdmain: rejected connection from “172.16.100.40:43892” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:00.939689 I | etcdmain: rejected connection from “172.16.100.28:44738” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:00.993042 I | etcdmain: rejected connection from “172.16.100.28:44742” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.005607 I | etcdmain: rejected connection from “172.16.100.40:43898” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.024015 I | etcdmain: rejected connection from “172.16.100.40:43900” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.045844 I | etcdmain: rejected connection from “172.16.100.28:44746” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.097482 I | etcdmain: rejected connection from “172.16.100.28:44750” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.111659 I | etcdmain: rejected connection from “172.16.100.40:43906” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.130189 I | etcdmain: rejected connection from “172.16.100.40:43908” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.155660 I | etcdmain: rejected connection from “172.16.100.28:44754” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.203604 I | etcdmain: rejected connection from “172.16.100.28:44758” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.217830 I | etcdmain: rejected connection from “172.16.100.40:43914” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.236230 I | etcdmain: rejected connection from “172.16.100.40:43916” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.256765 I | etcdmain: rejected connection from “172.16.100.28:44762” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.309802 I | etcdmain: rejected connection from “172.16.100.28:44766” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.324055 I | etcdmain: rejected connection from “172.16.100.40:43922” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.342295 I | etcdmain: rejected connection from “172.16.100.40:43924” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.361748 I | etcdmain: rejected connection from “172.16.100.28:44770” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.414243 I | etcdmain: rejected connection from “172.16.100.28:44774” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.428648 I | etcdmain: rejected connection from “172.16.100.40:43930” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.446695 I | etcdmain: rejected connection from “172.16.100.40:43932” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.466458 I | etcdmain: rejected connection from “172.16.100.28:44778” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.518936 I | etcdmain: rejected connection from “172.16.100.28:44782” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.533229 I | etcdmain: rejected connection from “172.16.100.40:43938” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.551178 I | etcdmain: rejected connection from “172.16.100.40:43940” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.572610 I | etcdmain: rejected connection from “172.16.100.28:44786” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.623518 I | etcdmain: rejected connection from “172.16.100.28:44790” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.637605 I | etcdmain: rejected connection from “172.16.100.40:43946” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.655754 I | etcdmain: rejected connection from “172.16.100.40:43948” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.677373 I | etcdmain: rejected connection from “172.16.100.28:44794” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.729742 I | etcdmain: rejected connection from “172.16.100.28:44798” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.743776 I | etcdmain: rejected connection from “172.16.100.40:43954” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.761868 I | etcdmain: rejected connection from “172.16.100.40:43956” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.783646 I | etcdmain: rejected connection from “172.16.100.28:44802” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.835898 I | etcdmain: rejected connection from “172.16.100.28:44806” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.849959 I | etcdmain: rejected connection from “172.16.100.40:43962” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.867992 I | etcdmain: rejected connection from “172.16.100.40:43964” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.889989 I | etcdmain: rejected connection from “172.16.100.28:44810” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.940220 I | etcdmain: rejected connection from “172.16.100.28:44816” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.947005 I | etcdmain: rejected connection from “172.16.100.28:44818” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.954343 I | etcdmain: rejected connection from “172.16.100.40:43970” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.972617 I | etcdmain: rejected connection from “172.16.100.40:43972” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:01.999263 I | etcdmain: rejected connection from “172.16.100.28:44822” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.046431 I | etcdmain: rejected connection from “172.16.100.28:44826” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.060888 I | etcdmain: rejected connection from “172.16.100.40:43978” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.079187 I | etcdmain: rejected connection from “172.16.100.40:43980” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.103844 I | etcdmain: rejected connection from “172.16.100.28:44830” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.152498 I | etcdmain: rejected connection from “172.16.100.28:44834” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.166856 I | etcdmain: rejected connection from “172.16.100.40:43986” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.183693 I | etcdmain: rejected connection from “172.16.100.40:43988” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.209979 I | etcdmain: rejected connection from “172.16.100.28:44838” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.258653 I | etcdmain: rejected connection from “172.16.100.28:44842” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.273014 I | etcdmain: rejected connection from “172.16.100.40:43994” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.289726 I | etcdmain: rejected connection from “172.16.100.40:43996” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.317776 I | etcdmain: rejected connection from “172.16.100.28:44846” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.364844 I | etcdmain: rejected connection from “172.16.100.28:44850” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.380673 I | etcdmain: rejected connection from “172.16.100.40:44002” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.395944 I | etcdmain: rejected connection from “172.16.100.40:44004” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.428887 I | etcdmain: rejected connection from “172.16.100.28:44854” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.469235 I | etcdmain: rejected connection from “172.16.100.28:44858” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.485232 I | etcdmain: rejected connection from “172.16.100.40:44010” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.502181 I | etcdmain: rejected connection from “172.16.100.40:44012” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.535342 I | etcdmain: rejected connection from “172.16.100.28:44862” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.573931 I | etcdmain: rejected connection from “172.16.100.28:44866” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.589598 I | etcdmain: rejected connection from “172.16.100.40:44018” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.606645 I | etcdmain: rejected connection from “172.16.100.40:44020” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.639958 I | etcdmain: rejected connection from “172.16.100.28:44870” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.680285 I | etcdmain: rejected connection from “172.16.100.28:44874” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.694005 I | etcdmain: rejected connection from “172.16.100.40:44026” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.711096 I | etcdmain: rejected connection from “172.16.100.40:44028” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.720884 W | rafthttp: health check for peer d9c6974366ce1ba could not connect: x509: certificate has expired or is not yet valid
2019-07-18 11:25:02.724850 W | rafthttp: health check for peer 9019c714a7a7fe3b could not connect: x509: certificate has expired or is not yet valid
2019-07-18 11:25:02.748170 I | etcdmain: rejected connection from “172.16.100.28:44878” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.786614 I | etcdmain: rejected connection from “172.16.100.28:44882” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.800473 I | etcdmain: rejected connection from “172.16.100.40:44034” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.817778 I | etcdmain: rejected connection from “172.16.100.40:44036” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.852748 I | etcdmain: rejected connection from “172.16.100.28:44886” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.891056 I | etcdmain: rejected connection from “172.16.100.28:44890” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.904937 I | etcdmain: rejected connection from “172.16.100.40:44042” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.925306 I | etcdmain: rejected connection from “172.16.100.40:44044” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.960414 I | etcdmain: rejected connection from “172.16.100.28:44894” (error “remote error: tls: bad certificate”, ServerName “”)
2019-07-18 11:25:02.998057 I | etcdmain: rejected connection from “172.16.100.28:44898” (error “remote error: tls: bad certificate”, ServerName “”)
Results:
I was expecting rke to be succesfully deployed. However, there is tls: bad certificate error and health check problem.