I stood up a single node rancher “cluster” on RHEL and would like to add a node to make it HA. Can this be done, or do I need to start from scratch? Thx!!!
Ps. Running k3s
Are you talking about the (a)cluster running rancher or a (b)downstream cluster?
If (a):
How did you set up k3s/rancher exactly? You probably should follow a similar approach as for a migration from docker to helm install (rancher backup, cluster recreate, restore backup). If did run also other workloads (which you should not do), you need to restore those too. Check the k3s guides for how to set up a ha cluster (vs. single node k8s). And of course, if using etcd, you always should have an odd number (1,3,5) of etcd nodes, so a 2 node cluster is never HA.
if (b): Just add nodes from cluster management…
I created a single k3s instance and installed rancher on that. I was hoping there was a way to extend that instance to another node and create an HA environment, but it appears the only way to do this is to scratch the system and start over. Is there not a way to add additional servers to a cluster that has already been created? The register cluster option doesn’t seem to work for us either. We get untrusted certificate chains, even when we do the “insecure” download for the yaml. I will go back and punt with my initial non-prod solution and start over from scratch…
You are still quite unspecific, giving the exact commads you used to setup the k3s instance/cluster would be essential for help. Check the k3s docs on the difference between single node and cluster setup.
What @bpedersen2 is asking about is how you have set up your clusters. For example, if you are looking for a full HA architecture then k8s itself is split into the CONTROL PLANE and the DATA PLANE. The control plane is where you install the primary constructs that manage the cluster, such as Etcd, controller manager, API server, scheduler, etc. In a HA solution you would typically provision at least 3 node that are dedicated to the control plane and NOT run any application workload on them. The data plane is essentially the group of nodes where you deploy your application workloads. Again for HA you would start off with at least 3 of those.
Rancher also has the same concept. The cluster management capabilities (for HA) should be installed on their own dedicated nodes, note, these are separate from the k8s control plane nodes (you can mix, but it just makes management and upgrading harder). The application workload nodes that you add, either through the Rancher management plane or directly in your k8s setup are the SAME. To more easily bring them under the management of Rancher you can add then directly from within Rancher. If you are in a Cloud environment (or others that are supported by Rancher) this can also provision the nodes for you, otherwise just update your k8s DATA PLANE config to add more node and then use Rancher to register them so that workloads can be managed by Rancher (it will install an agent that communicates with the Rancher MANAGEMENT nodes which it turn talks to the k8s CONTROL PLANE).
So, that is a rough outline of an HA architecture using Rancher. I suspect what you have done is combine some of these together (i.e. install Rancher on your k8s control plane nodes, and maybe even run application workloads on the same nodes). This not uncommon but doesn’t represent a HA setup and will bring its own set of problems when you come to upgrade the various components or nodes fail. To make you life a lot easier, I would maintain a strict separation of concerns. For example, in the full HA architecture your Rancher nodes may fail (or you may purposely take them down during an upgrade), but your k8s workloads will keep on running and reconnect when Rancher is back up. You may lose the ability to deploy and manage workloads from Rancher during that time, but at least you don’t have an outage that impacts business facing apps. You also still have access to your primary k8s cluster via the k8s API, typically via the kubectl client or others such as helm.
So in order to help with your question, please describe what nodes you have set up and what they are used for.
Okay, let’s start again.
Firstly, I am using k3s for all the nodes. There is no K8 involved or dependencies related to quorum (that I am aware of). I started with a single server and installed K3s and Rancher on it. I then created a k3s cluster with 2 servers. When I tried to do the registration command I received an error on the to-be-added server that it didn’t trust the certificate path, so I tried the insecure method and got the same error.
@LittleJoe So you have a single node running k3s with Rancher installed, you then created (outside) of rancher a k3s cluster with two servers. Then using rancher you tried to import the generic cluster you created, or did you just create one using the two systems created?
The key question is as per @bpedersen2 how did you create the Rancher k3s system, as in what commands did you run, for example the exact commands for;
- Installing k3s
- Installing helm
- Adding jetstack and rancher repos and update
- Using helm to install cert-manager
- Using helm to install Rancher
It was from your opening post that you mentioned you wanted to make your cluster HA, which means you need to establish a quorum (2 nodes doesn’t really give you that level of resilience). Also mixing your Rancher management plane capabilities with the cluster control plane and application workloads is not a great approach.
K8s is a general term that relates to Kubernetes. K3s is just one distribution (its still k8s and the concepts are the same even if the implementation differs).
That said, if you are just asking about the fact that the registration of a node failed then just show here the exact registration command (if you did that yourself from the terminal, or from the Rancher logs if not) and the exact error that was output.
Keeping the question simple and targeted, and showing the exact inputs and outputs (and/or behaviours) is the key to getting help.
Agreed. I will review the documentation again to make sure I’m not missing anything. The systems I have set up I have done in pairs, because I was under the impression that k3s did not require the odd number of nodes for the cluster, so I started with two. I will look into adding a third.