How to provide additional disk for Longhorn in Rancher

Rancher
1

Hi everyone,
We have a Rancher 2.7 cluster installed with a connected vSphere cloud provider.
In vSphere, we build a vm template for this cluster, that is used in Rancher Node templates.
Now we want to deploy Longhorn, but in the documentation there is a recommendation to use a disk separate from the root partition for Longhorn.
The question is: how can we provision new additional disk to the workers during vm/worker provisioning?
Thanks a lot!

2

Hi,
You can define additional node pool specially itended for storage node and indicate another dedicated template to create nodes in this node pool.
This dedicated template you can prepare by adding additional disk or disks for longhorn.
R2

3

Thanks a lot! I’ll try to do this

4

Hi,
I am exactly in the same situation. I would like to know how to add that second disk for longhorn.

Did you create a new template with two disks? what happens with the “disk” size value of machine pool, which of the two disk the template has is going to grow?

Thanks!

5

Hi,

Yes exactly, at the infrastructure provider level we have a second template with already added disks for dedicated storage node pools in our downstream clusters.

So the topic of resizing is tricky in some ways.
Expanding the available space within the existing pool of storage nodes can be implemented either by adding a node to the pool (no topic here) or by expanding the node disks - here you can, for example, define a template with a target maximum disk size that we can allocate for storage within a node in our infrastructure, and then for specific clusters configuration change (reduce) the size of the logical volume on the disk dedicated to storage (I assume here use of lvm functionality at the OS level).
Disk overprovisioning in the era of current disk array and virtualizers functionalities (deduplication, compression, thin or thick provision) does not represent a significant increase in costs and solves the difficult operation of reprovisioning such node (however, of course if needed, it requires caution in reducing the storage size available to Longhorn (for newly provisoned nodes whent it happens),(all these activities can be performed online)).

To sum up, you need to properly design the configuration, planning the architecture and target storage needs, specifying the maximum resource size of the node and then prepare a dedicated template, remembering that depending on the choice you make, you need to prepare appropriately for day two operation :slight_smile:

Cheers,
R2

6

My concern is more related to the provision facilities rancher offers and the integration with longhorn storage.

Lets suppose I have 1 pool of 3 workers based on one template of 2 virtual disks (one for OS and another for storage). But when the pool is defined, apart of selecting the template, in the rancher vsphere provision tool for RKE2 I can set the CPU, RAM and “disk” size of the nodes of the pool. I do not know for which disk is this size: for the first or second disk of the node?.

Furthermore, if the 3 worker nodes are of the same pool, sometimes, when performing updates or changing configurations, Rancher delete old nodes and create new ones to replace them. I fear this can be risky because Rancher is going to delete nodes with longhorn disks and valid data and I do not know if it is going to wait to new nodes to sync data.

Did you have to face to these problems?

7

Disk size defined in the node pool applies to the first disk on the node.

In the second case appropriate configuration must be used on the cluster and in Longhorn:

  1. Upgrade strategy for cluster (at least):
    workerDrainOptions:
    disableEviction: false
    enabled: true
    force: true
    So we have to drain nodes before upgrade.
  2. Longhorn must have set (at least):
    Node Drain Policy: block-if-contains-last-replica
    Longhorn will block the drain when the node contains the last healthy replica of a volume.
    This is done through pod disruption budget object on the K8s level.

R2

1 Like
8

I found one issue with this setup of one template with multiple disks.

As you said, the “disk size” field of the pool form applies only to the first disk of the virtual machine template. This is great because I can have additional disks with a pre-defined size in the VM template for cloud storage. However, I would like the provisioner to respect the datastore location of those additional disks, because, for example, one could be located in a fast datastore and the second one in a slow datastore. Nevertheless, the vsphere provisioner seems to locate the virtual machine configuration and all the virtual disks in the Datastore specified in the “Data store” field of the pool form.

This is a pity because then I have to manually migrate virtual disks to the desired datastores.

9

Consider building a node pool per site/resource_group and indicate the datastore cluster as the target for a given node pool. Then you land the storage nodes in the right location and the datastore cluster provides you with the logic of balancing VMs disks on the datastores.