Relate issue from GitHub here
Steps to reproduce (least amount of steps as possible):
Install fresh Rancher Server with HA over a 2 node K3s cluster
Create new cluster
Try to register a new control-plane/etcd or worker node
Result:
New node don’t register to kubelet, cluster on provisioning state forever
Expected Result:
New node get registered and cluster deployed with healthy status
Other details that may be helpful:
Log for Traefik ingress on k3s Rancher server:
{"level":"info","msg":"Using TOML configuration file /config/traefik.toml","time":"2021-06-25T04:30:20Z"}
--
Fri, Jun 25 2021 12:30:20 am | {"level":"info","msg":"No tls.defaultCertificate given for https: using the first item in tls.certificates as a fallback.","time":"2021-06-25T04:30:20Z"}
Fri, Jun 25 2021 12:30:20 am | {"level":"info","msg":"Traefik version v1.7.19 built on 2019-10-28_02:07:32PM","time":"2021-06-25T04:30:20Z"}
Fri, Jun 25 2021 12:30:20 am | {"level":"info","msg":"\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/v1.7/basics/#collected-data\n","time":"2021-06-25T04:30:20Z"}
Fri, Jun 25 2021 12:30:20 am | {"level":"info","msg":"Preparing server http \u0026{Address::80 TLS:\u003cnil\u003e Redirect:\u003cnil\u003e Auth:\u003cnil\u003e WhitelistSourceRange:[] WhiteList:\u003cnil\u003e Compress:true ProxyProtocol:\u003cnil\u003e ForwardedHeaders:0xc000473f40} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s","time":"2021-06-25T04:30:20Z"}
Fri, Jun 25 2021 12:30:20 am | {"level":"info","msg":"Preparing server https \u0026{Address::443 TLS:0xc0005cb5f0 Redirect:\u003cnil\u003e Auth:\u003cnil\u003e WhitelistSourceRange:[] WhiteList:\u003cnil\u003e Compress:true ProxyProtocol:\u003cnil\u003e ForwardedHeaders:0xc000473ec0} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s","time":"2021-06-25T04:30:20Z"}
Fri, Jun 25 2021 12:30:20 am | {"level":"info","msg":"Preparing server prometheus \u0026{Address::9100 TLS:\u003cnil\u003e Redirect:\u003cnil\u003e Auth:\u003cnil\u003e WhitelistSourceRange:[] WhiteList:\u003cnil\u003e Compress:false ProxyProtocol:\u003cnil\u003e ForwardedHeaders:0xc000473f00} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s","time":"2021-06-25T04:30:20Z"}
Fri, Jun 25 2021 12:30:20 am | {"level":"info","msg":"Starting provider configuration.ProviderAggregator {}","time":"2021-06-25T04:30:20Z"}
Fri, Jun 25 2021 12:30:20 am | {"level":"info","msg":"Starting server on :80","time":"2021-06-25T04:30:20Z"}
Fri, Jun 25 2021 12:30:20 am | {"level":"info","msg":"Starting server on :443","time":"2021-06-25T04:30:20Z"}
Fri, Jun 25 2021 12:30:20 am | {"level":"info","msg":"Starting server on :9100","time":"2021-06-25T04:30:20Z"}
Fri, Jun 25 2021 12:30:20 am | {"level":"info","msg":"Starting provider *kubernetes.Provider {\"Watch\":true,\"Filename\":\"\",\"Constraints\":[],\"Trace\":false,\"TemplateVersion\":0,\"DebugLogGeneratedTemplate\":false,\"Endpoint\":\"\",\"Token\":\"\",\"CertAuthFilePath\":\"\",\"DisablePassHostHeaders\":false,\"EnablePassTLSCert\":false,\"Namespaces\":null,\"LabelSelector\":\"\",\"IngressClass\":\"\",\"IngressEndpoint\":{\"IP\":\"\",\"Hostname\":\"\",\"PublishedService\":\"kube-system/traefik\"},\"ThrottleDuration\":0}","time":"2021-06-25T04:30:20Z"}
Fri, Jun 25 2021 12:30:20 am | {"level":"info","msg":"ingress label selector is: \"\"","time":"2021-06-25T04:30:20Z"}
Fri, Jun 25 2021 12:30:20 am | {"level":"info","msg":"Creating in-cluster Provider client","time":"2021-06-25T04:30:20Z"}
Fri, Jun 25 2021 12:30:20 am | {"level":"error","msg":"Error configuring TLS for ingress cattle-system/rancher: secret cattle-system/tls-rancher-ingress does not exist","time":"2021-06-25T04:30:20Z"}
Fri, Jun 25 2021 12:30:20 am | {"level":"info","msg":"Server configuration reloaded on :80","time":"2021-06-25T04:30:20Z"}
Fri, Jun 25 2021 12:30:20 am | {"level":"info","msg":"Server configuration reloaded on :443","time":"2021-06-25T04:30:20Z"}
Fri, Jun 25 2021 12:30:20 am | {"level":"info","msg":"Server configuration reloaded on :9100","time":"2021-06-25T04:30:20Z"}
Fri, Jun 25 2021 12:30:22 am | {"level":"info","msg":"Server configuration reloaded on :443","time":"2021-06-25T04:30:22Z"}
Fri, Jun 25 2021 12:30:22 am | {"level":"info","msg":"Server configuration reloaded on :9100","time":"2021-06-25T04:30:22Z"}
Fri, Jun 25 2021 12:30:22 am | {"level":"info","msg":"Server configuration reloaded on :80","time":"2021-06-25T04:30:22Z"}
Fri, Jun 25 2021 12:30:49 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:30:49Z"}
Fri, Jun 25 2021 12:30:59 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:30:59Z"}
Fri, Jun 25 2021 12:31:09 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:31:09Z"}
Fri, Jun 25 2021 12:31:19 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:31:19Z"}
Fri, Jun 25 2021 12:31:29 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:31:29Z"}
Fri, Jun 25 2021 12:31:39 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:31:39Z"}
Fri, Jun 25 2021 12:31:49 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:31:49Z"}
Fri, Jun 25 2021 12:31:59 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:31:59Z"}
Fri, Jun 25 2021 12:32:09 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:32:09Z"}
Fri, Jun 25 2021 12:32:19 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:32:19Z"}
Fri, Jun 25 2021 12:32:29 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:32:29Z"}
Fri, Jun 25 2021 12:32:39 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:32:39Z"}
Fri, Jun 25 2021 12:32:49 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:32:49Z"}
Fri, Jun 25 2021 12:32:59 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:32:59Z"}
Fri, Jun 25 2021 12:33:09 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:33:09Z"}
Fri, Jun 25 2021 12:33:19 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:33:19Z"}
Fri, Jun 25 2021 12:33:29 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:33:29Z"}
Fri, Jun 25 2021 12:33:39 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:33:39Z"}
Fri, Jun 25 2021 12:33:49 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:33:49Z"}
Fri, Jun 25 2021 12:34:00 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:34:00Z"}
Fri, Jun 25 2021 12:34:10 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:34:10Z"}
Fri, Jun 25 2021 12:34:20 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:34:20Z"}
Fri, Jun 25 2021 12:34:30 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:34:30Z"}
Fri, Jun 25 2021 12:34:40 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:34:40Z"}
Fri, Jun 25 2021 12:34:50 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:34:50Z"}
Fri, Jun 25 2021 12:35:00 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:35:00Z"}
Fri, Jun 25 2021 12:35:10 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:35:10Z"}
Fri, Jun 25 2021 12:35:20 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:35:20Z"}
Fri, Jun 25 2021 12:35:30 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:35:30Z"}
Fri, Jun 25 2021 12:35:40 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:35:40Z"}
Fri, Jun 25 2021 12:35:50 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:35:50Z"}
Fri, Jun 25 2021 12:35:58 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error when copying from backend to client: websocket: close 1006 (abnormal closure): unexpected EOF","time":"2021-06-25T04:35:58Z"}
Fri, Jun 25 2021 12:35:58 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error when copying from backend to client: websocket: close 1006 (abnormal closure): unexpected EOF","time":"2021-06-25T04:35:58Z"}
Fri, Jun 25 2021 12:36:00 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:36:00Z"}
Fri, Jun 25 2021 12:36:01 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error when copying from backend to client: websocket: close 1006 (abnormal closure): unexpected EOF","time":"2021-06-25T04:36:01Z"}
Fri, Jun 25 2021 12:36:08 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error when copying from backend to client: websocket: close 1006 (abnormal closure): unexpected EOF","time":"2021-06-25T04:36:08Z"}
Fri, Jun 25 2021 12:36:10 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:36:10Z"}
Fri, Jun 25 2021 12:36:20 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:36:20Z"}
Fri, Jun 25 2021 12:36:30 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:36:30Z"}
Fri, Jun 25 2021 12:36:40 am | {"level":"error","msg":"vulcand/oxy/forward/websocket: Error dialing \"rancher.gaviamarketing.com\": websocket: bad handshake with resp: 400 400 Bad Request","time":"2021-06-25T04:36:40Z"}
Log from rancher-agent on new added node
INFO: Arguments: --server https://rancher.gaviamarketing.com --token REDACTED --address ipify --internal-address 192.168.0.100 --etcd --controlplane
INFO: Environment: CATTLE_ADDRESS=159.69.55.228 CATTLE_INTERNAL_ADDRESS=192.168.0.100 CATTLE_NODE_NAME=GAVIA-HYBRID-1-CP-1 CATTLE_ROLE=,etcd,controlplane CATTLE_SERVER=https://rancher.gaviamarketing.com CATTLE_TOKEN=REDACTED
INFO: Using resolv.conf: nameserver 127.0.0.53 options edns0 trust-ad
WARN: Loopback address found in /etc/resolv.conf, please refer to the documentation how to configure your cluster to resolve DNS properly
INFO: https://rancher.gaviamarketing.com/ping is accessible
INFO: rancher.gaviamarketing.com resolves to 49.12.23.182
time="2021-06-25T03:54:50Z" level=info msg="Listening on /tmp/log.sock"
time="2021-06-25T03:54:50Z" level=info msg="Rancher agent version v2.5.8 is starting"
time="2021-06-25T03:54:50Z" level=info msg="Option customConfig=map[address:159.69.55.228 internalAddress:192.168.0.100 label:map[] roles:[etcd controlplane] taints:[]]"
time="2021-06-25T03:54:50Z" level=info msg="Option etcd=true"
time="2021-06-25T03:54:50Z" level=info msg="Option controlPlane=true"
time="2021-06-25T03:54:50Z" level=info msg="Option worker=false"
time="2021-06-25T03:54:50Z" level=info msg="Option requestedHostname=GAVIA-HYBRID-1-CP-1"
time="2021-06-25T03:54:50Z" level=info msg="Connecting to wss://rancher.gaviamarketing.com/v3/connect/register with token 6c8mtcsbz6xlnm4w45tj5bvl72qmstcqlx88w9c488lmncs2zzp2x2"
time="2021-06-25T03:54:50Z" level=info msg="Connecting to proxy" url="wss://rancher.gaviamarketing.com/v3/connect/register"
time="2021-06-25T03:54:51Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering or etcd and controlplane node have to be registered first"
time="2021-06-25T03:54:53Z" level=info msg="Starting plan monitor, checking every 15 seconds"
time="2021-06-25T04:30:50Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
time="2021-06-25T04:31:00Z" level=info msg="Connecting to wss://rancher.gaviamarketing.com/v3/connect with token 6c8mtcsbz6xlnm4w45tj5bvl72qmstcqlx88w9c488lmncs2zzp2x2"
time="2021-06-25T04:31:00Z" level=info msg="Connecting to proxy" url="wss://rancher.gaviamarketing.com/v3/connect"
time="2021-06-25T04:31:00Z" level=info msg="Starting plan monitor, checking every 15 seconds"
Environment information
- Rancher version (
rancher/rancher/rancher/serverimage tag or shown bottom left in the UI): v2.5.8 - Installation option (single install/HA): HA over K3s v1.20.4
Cluster information
- Cluster type (Hosted/Infrastructure Provider/Custom/Imported): Custom
- Machine type (cloud/VM/metal) and specifications (CPU/memory): Hetzner Cloud / 4vCPU / 8G
- Kubernetes version (use
kubectl version): v1.20 (default) - Docker version (use
docker version): v20.10
Can someone help us here?
Best