How to add a EC2 instance as a worker node to a local vm based rancher 2.4 custom k8s cluster

Rancher
1

Hi,

Has anyone figure out / have documentation on adding a EC2 instance as worker node to an existing rancher 2.4.12 created custom k8s cluster whose control plane is hosted in my internal datacenter?

I have a 1.18 vm based, rancher created k8s cluster hosted in my datacenter. I want to add a worker node hosted in EC2 to do some testing.

Steps I took:

  1. create an EC2 instance using rancheros-v1.5.8-hvm-1 (ami-02fe87f853d560d52) and a security group with all ports open inbound and outbound
  2. ssh into it using rancher user
  3. test dns resolution of the rancher server hosted in my network
  4. test dns resolution of the control plan servers hosted in my network
  5. ran the following command: docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.4.12 --server https://my.rancher.server --token mytoken --ca-checksum mycacheck

This gets the ec2 worker node to show up in my rancher cluster with a state of “Registering” and a message “Waiting to register with Kubernetes”. I can’t seem to get past this point.

These are the kubelet relevant logs from the kubelet docker container on the EC2 worker node:
I0325 18:15:12.296651 2577 csi_plugin.go:945] Failed to contact API server when waiting for CSINode publishing: Get https://127.0.0.1:6443/apis/storage.k8s.io/v1/csinodes/ip-10-19-231-52: dial tcp 127.0.0.1:6443: connect: connection refused
I0325 18:15:12.296684 2577 server.go:145] Starting to listen on 0.0.0.0:10250
E0325 18:15:12.297563 2577 event.go:269] Unable to write event: ‘Post https://127.0.0.1:6443/api/v1/namespaces/default/events: dial tcp 127.0.0.1:6443: connect: connection refused’ (may retry after sleeping)
I0325 18:15:12.299841 2577 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
I0325 18:15:12.301661 2577 volume_manager.go:263] The desired_state_of_world populator starts
I0325 18:15:12.301800 2577 volume_manager.go:265] Starting Kubelet Volume Manager
I0325 18:15:12.302197 2577 reflector.go:175] Starting reflector *v1.CSIDriver (0s) from k8s.io/client-go/informers/factory.go:135
I0325 18:15:12.302780 2577 desired_state_of_world_populator.go:139] Desired state populator starts to run
E0325 18:15:12.297678 2577 kubelet.go:1308] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
I0325 18:15:12.303294 2577 server.go:393] Adding debug handlers to kubelet server.
E0325 18:15:12.311235 2577 controller.go:136] failed to ensure node lease exists, will retry in 200ms, error: Get https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-19-231-52?timeout=10s: dial tcp 127.0.0.1:6443: connect: connection refused
E0325 18:15:12.311510 2577 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.CSIDriver: Get https://127.0.0.1:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused
E0325 18:15:12.334384 2577 kubelet.go:2190] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

any ideas?

Thanks,
Mark