Errors during air gap install

k3s, k3OS, and k3d
1

I have followed the directions on the k3s official site, but cannot seem to get rid of the errors around selinux.

[ec2-user]$ INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_EXEC="--write-kubeconfig-mode 644"  INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_SELINUX_WARN=true INSTALL_K3S_EXEC="--disable traefik" ./install.sh
[INFO]  Skipping k3s download and verify
[INFO]  Skipping installation of SELinux RPM
[WARN]  Failed to find the k3s-selinux policy, please install:
    dnf install -y container-selinux
    dnf install -y https://rpm.rancher.io/k3s/stable/common/centos/8/noarch/

[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s
Job for k3s.service failed because the control process exited with error code.
See "systemctl status k3s.service" and "journalctl -xe" for details.

looking at the logs:

[ec2-user@]$ systemctl status k3s.service

â—Ź k3s.service - Lightweight Kubernetes

Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)

Active: activating (auto-restart) (Result: exit-code) since Wed 2023-10-04 19:53:46 UTC; 2s ago

Docs: https://k3s.io

Process: 3292 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=1/FAILURE)

[ec2-user@]$ journalctl -xe
Oct 04 19:55:15[1]: Starting Lightweight Kubernetes...

-- Subject: Unit k3s.service has begun start-up

-- Defined-By: systemd

-- Support: https://access.redhat.com/support

--

-- Unit k3s.service has begun starting up.

Oct 04 19:55:15  sh[3328]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service

Oct 04 19:55:15 [1]: k3s.service: Control process exited, code=exited status=1

Oct 04 19:55:15 [1]: k3s.service: Failed with result 'exit-code'.

-- Subject: Unit failed

-- Defined-By: systemd

-- Support: https://access.redhat.com/support

--

-- The unit k3s.service has entered the 'failed' state with result 'exit-code'.

Oct 04 19:55:15 [1]: Failed to start Lightweight Kubernetes.

-- Subject: Unit k3s.service has failed

-- Defined-By: systemd

-- Support: https://access.redhat.com/support

--

-- Unit k3s.service has failed.

--

-- The result is failed.

any suggestions, or hints towards what I’m missing?
TIA

2

I’ve even tried this walk through, exact same errors as before:

3

@acatkrom Hi, just tried your notes/links on openSUSE MicroOS, with some of the leg work done via combustion…

All good here… but I did install the k3s-selinux rpm and a seed-token during the combustion run, and added node ip address to the EXEC options;

INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_SELINUX_WARN=true \
>             INSTALL_K3S_EXEC="--write-kubeconfig-mode 644 --disable traefik --node-ip=192.xxx.xxx.141 --node-external-ip=192.xxx.xxx.141" \
>             /root/bin/install.sh
[INFO]  Skipping k3s download and verify
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

airgap-k3s:~ # cp /etc/rancher/k3s/k3s.yaml .
airgap-k3s:~ # sed -i 's/127.0.0.1/192.xxx.xxx.141/g' k3s.yaml
airgap-k3s:~ # export KUBECONFIG=$(pwd)/k3s.yaml

kubectl get nodes -o wide
NAME         STATUS   ROLES                  AGE     VERSION         INTERNAL-IP      EXTERNAL-IP      OS-IMAGE           KERNEL-VERSION    CONTAINER-RUNTIME
airgap-k3s   Ready    control-plane,master   2m17s   v1.25.12+k3s1   192.xxx.xxx.141   192.xxx.xxx.141   openSUSE MicroOS   6.5.4-1-default   containerd://1.7.1-k3s1

kubectl get pods -o wide -A
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE    IP          NODE         NOMINATED NODE   READINESS GATES
kube-system   coredns-8b9777675-rgp57                   1/1     Running   0          2m3s   10.42.0.4   airgap-k3s   <none>           <none>
kube-system   local-path-provisioner-69dff9496c-slfkc   1/1     Running   0          2m3s   10.42.0.2   airgap-k3s   <none>           <none>
kube-system   metrics-server-854c559bd-kkqwm            1/1     Running   0          2m3s   10.42.0.3   airgap-k3s   <none>           <none>

So, do you have a network connection or an interface with a default route on the airgapped system, if not I suspect you need to install k3s-selinux…

4

Thanks!!
A little more background the machine is running RHEL 8.8 on aws
I did install k3s-selinux as shown in the link, before trying to install k3s

Going through the logs, and researching it seemed to be erroring out because nm-cloud-setup.service was up, so I disabled that service with sudo systemctl disable nm-cloud-setup.service

I restarted the k3s service with sudo systemctl restart k3s.service

[ec2-user@ip-~]$ kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP   16h

Next big question is “what is the nm-cloud-setup service” and “do we need it”

5

@acatkrom part of Network Manager https://developer-old.gnome.org/NetworkManager/stable/nm-cloud-setup.html

1 Like