Stuck at "Waiting for API to be available"

I am trying to import an existing Kubernetes cluster (which was set up by using this installation instructions).

Basic overview:

  • CentOS 8
  • Kubernetes 1.22
  • CRI-O 1.22
  • Rancher 2.5.11

Unfortunately, Rancher is stuck with the message “Waiting for API to be available”. By taking a closer look at the API configuration (which, I presume, was created from information reported back by the cattle cluster agent), I kind of understand why. The API endpoint is set to “https://10.96.0.1:443”. This is the internal IP of the “kubernetes” service, so of course the Rancher server can’t connect to that.

$ kubectl get service --all-namespaces
NAMESPACE          NAME                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
calico-apiserver   calico-api                        ClusterIP   10.98.207.137    <none>        443/TCP                  112m
calico-system      calico-kube-controllers-metrics   ClusterIP   10.97.0.222      <none>        9094/TCP                 114m
calico-system      calico-typha                      ClusterIP   10.102.202.143   <none>        5473/TCP                 115m
default            kubernetes                        ClusterIP   10.96.0.1        <none>        443/TCP                  59d
kube-system        kube-dns                          ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP   59d

What I don’t understand is, why that IP was reported back to Rancher as API endpoint. Shouldn’t this be the same address, which is reported by the kubectl config view command and which is the actual physical IP adress of my kubernetes CTL node?

$ kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://172.20.37.107:6443
  name: kubernetes
[....]
$ kubectl cluster-info
Kubernetes control plane is running at https://172.20.37.107:6443
CoreDNS is running at https://172.20.37.107:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
[...]
$ ifconfig
[...]
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.20.37.107  netmask 255.255.255.0  broadcast 172.20.37.255
[...]

What am I missing? How can successfully “connect” my Kubernetes cluster with Rancher?

We saw this with k8s 1.21. Dropped back to k8s 1.20 and that resolved “Waiting for API to be available”

Found this in the Rancher release notes:
Importing a Kubernetes v1.21 cluster might not work properly. We are planning to add support for Kubernetes v1.21 in the future.