Installing rancher [CentOS8, RKE2] - Problem

Hi, I’m fairly new to Kubernetes. I am trying to learn and am trying to create set up to install Rancher.

I am using DigitalOcean CentOS8 VM. I tried many times, I did these steps. (After VM is provisioned)

  • CentOS8
  • RKE2 v1.27.11+rke2r1
  • Rancher stable

I do this simple steps and result is always the same. I checked that ports are OK, nginx is OK, ingress is OK. And afaik this is just simple set up out of the box. What am I missing here? Any help would be greatly appreciated.

Configure NetworkManager to ignore calico/flannel related network interfaces

tee /etc/NetworkManager/conf.d/cni.conf <<EOF
[keyfile]
unmanaged-devices=interface-name:cni0;interface-name:flannel.1;interface-name:cali*
EOF
sudo systemctl restart NetworkManager 

Disable swap

swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

Ensure your system is up to date

dnf update -y
dnf install curl tar nano -y
localectl set-locale LANG=en_US.UTF-8
timedatectl set-timezone Europe/Ljubljana
dnf install chrony -y
systemctl enable --now chronyd 

nfs-utils cryptsetup iscsi-initiator-utils

dnf install nfs-utils cryptsetup iscsi-initiator-utils -y
systemctl enable --now iscsid.service 
dnf update -y
dnf clean all

RKE2

curl -sfL https://get.rke2.io | INSTALL_RKE2_VERSION=v1.27.11+rke2r1 sh -
systemctl enable --now rke2-server.service

Helm

curl -#L https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash`

Rancher, Jetstack Repo

helm repo add rancher-stable https://releases.rancher.com/server-charts/stable
helm repo add jetstack https://charts.jetstack.io 

Cert manager

helm upgrade -i cert-manager jetstack/cert-manager -n cert-manager --create-namespace --set installCRDs=true

Install rancher

kubectl create namespace cattle-system
helm install rancher rancher-stable/rancher \
--namespace cattle-system \
--set hostname=droplet1.inharmonyapp.com \
--set bootstrapPassword=admin \
--set replicas=3 

All pods starts, I can access my rancher dashboard, but my rancher pods are telling me:

2024/03/21 13:02:50 [ERROR] Failed to handle tunnel request from remote address 10.42.0.19:41130: response 400: cluster not found
2024/03/21 13:02:55 [ERROR] Failed to handle tunnel request from remote address 10.42.0.19:41138: response 400: cluster not found
2024/03/21 13:02:55 [ERROR] Failed to connect to peer wss://10.42.0.19/v3/connect [local ID=10.42.0.20]: websocket: bad handshake
2024/03/21 13:03:00 [ERROR] Failed to handle tunnel request from remote address 10.42.0.19:49782: response 400: cluster not found
2024/03/21 13:03:00 [ERROR] Failed to connect to peer wss://10.42.0.19/v3/connect [local ID=10.42.0.20]: websocket: bad handshake
2024/03/21 13:03:05 [ERROR] Failed to handle tunnel request from remote address 10.42.0.19:49794: response 400: cluster not found
2024/03/21 13:03:05 [ERROR] Failed to connect to peer wss://10.42.0.19/v3/connect [local ID=10.42.0.20]: websocket: bad handshake
2024/03/21 13:03:10 [ERROR] Failed to handle tunnel request from remote address 10.42.0.19:49842: response 400: cluster not found
2024/03/21 13:03:10 [ERROR] Failed to connect to peer wss://10.42.0.19/v3/connect [local ID=10.42.0.20]: websocket: bad handshake
2024/03/21 13:03:15 [ERROR] Failed to handle tunnel request from remote address 10.42.0.19:49856: response 400: cluster not found
2024/03/21 13:03:15 [ERROR] Failed to connect to peer wss://10.42.0.19/v3/connect [local ID=10.42.0.20]: websocket: bad handshake
2024/03/21 13:03:20 [ERROR] Failed to connect to peer wss://10.42.0.19/v3/connect [local ID=10.42.0.20]: websocket: bad handshake
2024/03/21 13:03:20 [ERROR] Failed to handle tunnel request from remote address 10.42.0.19:60958: response 400: cluster not found

Firewalld is not installed.

@Matic Hi and welcome to the Forum :smile:
There is this post about the same error https://forums.rancher.com/t/failed-to-connect-to-peer-wss-10-42-0-22-v3-connect-local-id-10-42-1-15-websocket-bad-handshake/20529/7, however I have RKE2 running on one node with one replica (bare-metal and openSUSE MicroOS), sure you want three?

I also set the following option to the rancher install as well as set the version (this also applies to cert-manager version).

--set global.cattle.psp.enabled=false

@Matic Oh, I also have a /etc/sysctl.d/kubernetes.conf containing;

net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.docker0.forwarding = 1
net.ipv4.conf.enp1s0.forwarding = 1
net.ipv4.conf.lo.forwarding = 1