K3s Deployment issue on VirtualBox VMs

k3s, k3OS, and k3d
1

Hello,

I was recently trying to deploy K3s for the first time, so I chose a simple environment: 3 Rocky 8.9 VMs, with the plan being one master node and 2 worker/agent nodes.

I followed the quick-start guide here Quick-Start Guide | K3s and after fixing cert errors and simple stuff like that, I was able to get the installation script to run, and the master/control-plane node was ready. I went on to follow the steps to deploy the agents, but I was having a lot of problems. I won’t list all the specific errors, because they seemed to boil down to one common issue:

  • In each my VMs, I have a NAT nic that allows me to access the internet, but not the other VMs. They also have a ‘host-only’ adapter which allows my host machine to talk to my VMs and allows the VMs to talk to each other. This is what I wanted to use for my k8s networking, but it seemed like it was using the NAT IP that I was using to pull down the installation scripts for everything, which was a different IP and can not be used for communication between the nodes. I know that it was using this because I saw it everywhere in the journalctl logs and in various configuration files. I tried changing several configuration files in /var/lib/ranches/k3s but they seemed to get reset back to the NAT/non-working IP address.

The only way I was able to get the cluster fully up and functional was to stop the k3s and k3s-agent systemd services, and run the server/control plane with:

k3s server --node-ip control-plane-ip

And start the agents manually with

k3s agent --server https://control-plane-ip:6443 --token=$K3S_TOKEN

I feel like I’m missing some dumb configuration flag or file somewhere that would allow me to configure and run the control-plane/master node and the worker nodes/agents from the systemd services, without having to use the commands above to make it work (additionally, since I’m a relative beginner, I don’t even know if what I’m doing is bad practice or not…it just feels a little hacky since no website I saw recommended it).

Can someone please provide some guidance on why the quick-start installations where you just curl down an installation script and run it doesn’t seem to respect the server environment variables I had in place like K3S_URL or various other ones (or if I was using the wrong ones?)?

Last question, I’m sure “it depends” is the answer to this question, but is this method (curl’ing down the installation scripts) and passing them straight to bash the “recommended” way to deploy k3s control planes and workers/agents?

Thanks a lot in advance! If you have any questions for clarification, I’d be glad to give it.

2

@prelic Hi and welcome to the Forum :smile:
You have options --node-ip and --node-external-ip and --flannel-iface

I would suggest a visit to https://www.rancher.academy/ and get some free basic training :wink:

I run MicroOS or Leap libvirt machines, create my own seed token and use for example;

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION={v1.27.11+k3s1} \
   INSTALL_K3S_EXEC="server --cluster-init \
   --node-ip xxx.xxx.xxx.xxx --node-external-ip xxx.xxx.xxx.xxx" \
   K3S_TOKEN_FILE=/etc/rancher/k3s/node-token-seed \
   K3S_RESOLV_CONF=/etc/resolv.conf \
   K3S_NODE_NAME="myhostname" sh -s -
3

Thanks for your tips! I will be sure to try that!

4

@malcolmlewis1 I was actually able to use your example and tweak it a bit and got my control plane and all my worker nodes up and running with the install scripts from get.k3s.io! Even got some of the application deployed to the cluster for my first time! Thanks so much!

1 Like