Upgrading live cluster node OS

k3s, k3OS, and k3d
1

Hi,

I have a small cluster running Armbian and K3S. I am planning to upgrade the release of Armbian and I know it does not officially support the regular userspace upgrades. So my plan so far is as follows:

Before: backup the application data - databases, PVs, Longhorn snapshots.

Then, for each node:

  1. Cordon / drain the node
  2. Stop K3S
  3. Backup K3S data (/var/lib/rancher + /etc/rancher). I do not really have any persistent node volumes, I use Longhorn + mounted external drives. These should survive, I suppose, as long as the cluster metadata survives.
  4. Reimage the node - same way as before, configure it as it should be etc. I have Ansible scripts for that.
  5. Install exactly the same version of K3S, shut it down after the installation.
  6. Restore the previously backed /var/lib/rancher and /etc/rancher directories, start the node, uncordon it

Any flaws in this plan? Node identities do not change, e.g. they will get the same IP address, same host names etc.

Thanks!