Help troubleshooting default-disk-config +ansible

Longhorn
1

I’ve having trouble getting default disks setup in Longhorn for a new RKE2 cluster. I was handed manual commands and setup steps for a full cluster and am attempting to fully automate with Ansible. As with any task like this the fully manual steps aren’t all easily automated. To raise the difficulty level a bit I’m learning Kubernetes as I do this and my Ansible experience is limited. I’m handling that by learning the way to perform a task manually in Kubernetes/RKE2 then learning how it can be automated with a module in Ansible.

So far I’ve got it all automated except the node/disk setup in Longhorn. I read the documentation for default-disk-config from the following two pages:

I’ve got 3 nodes and they’re getting the labels and annotations and I’ve verified in the Longhorn UI that create-default-disk-labeled-nodes: true is set but I don’t get disks created.

I’m not sure how to troubleshoot from here. Is there any logging from Longhorn or in RKE2 that I can look through? I verified the disks are mounted where the disk config specifies and there’s no typos. I’ve spent hours rebuilding the lab and re-running the setup but I can’t figure out what is preventing the disks from being configured for each node.

Here’s the relevant bits of the Ansible-playbook I created:

    - name: Get Longhorn manifest
      ansible.builtin.get_url:
        url: https://raw.githubusercontent.com/longhorn/longhorn/v1.4.2/deploy/longhorn.yaml
        dest: /home/kube/manifests/longhorn.yaml
      register: longhorn_yaml

    - name: Update image paths for cluster registry in longhorn.yaml
      ansible.builtin.replace:
        path: "{{ longhorn_yaml.dest }}"
        regexp: '^(.*image: ).*[^longhornio]*(longhornio.*)'
        replace: '\1{{ registry_fqdn }}:5000/\2'

    - name: enable creation of default disk on labeled nodes
      ansible.builtin.lineinfile:
        path: "{{ longhorn_yaml.dest }}"
        insertafter: '\sdefault-setting.yaml:'
        line: '    create-default-disk-labeled-nodes: True'

    - name: Add default disk config notations to all 3 nodes
      kubernetes.core.k8s:
        state: present
        definition:
          apiVersion: v1
          kind: Node
          metadata:
            name: "{{ item }}"
            annotations:
              node.longhorn.io/default-disk-config: '[{ "name":"xvdb1_pod_pv", "path":"/mnt/pod_pv", "allowScheduling":true, "tags":"pod_pv"}, {"name":"xbdc1_dicom_stor", "path":"/mnt/dicom", "allowScheduling":true, "tags":"dicom_stor" }]'
            labels:
              node.longhorn.io/create-default-disk: 'config'
      with_items:
        - "{{ node1_fqdn }}"
        - "{{ node2_fqdn }}"
        - "{{ node3_fqdn }}"

    - name: Install Longhorn with longhorn.yaml
      kubernetes.core.k8s:
        state: present
        force: true
        src: "{{ longhorn_yaml.dest }}"

Edited to add some version info.
I’m using:
RKE2 1.25.12
Rancher 2.7.6
Longhorn 1.4.2

2

I may have found my error. Missing “s” in default-disks-config. I’m rebuilding now to see if that was it.

*Edit… Nope. Still not adding disks from default-disks-config

3

The error was with the tags. I mistakenly thought I could remove the brackets from examples since I was only using one tag. I’m not 100% familiar with formatting of the json/go struct so I didn’t realize that the square brackets are required whether it’s a list of tags or a single tag.
I used the following to get logs from one of the Longhorn manager pods:

kubectl get pods -n longhorn-system
kubectl logs longhorn-manager-4d956

Found this log entry:

time="2023-11-14T17:35:26Z" level=warning msg="Kubernetes node: invalid annotation node.longhorn.io/default-disks-config: config: failed to unmarshal the default disks annotation: json: cannot unmarshal string into Go struct field DiskSpecWithName.tags of type []string"

Revised annotation that works:

node.longhorn.io/default-disks-config: '[{ "name":"xvdb1_pod_pv", "path":"/mnt/pod_pv", "allowScheduling":true, "tags":[ "pod_pv" ]}, {"name":"xbdc1_dicom_stor", "path":"/mnt/dicom", "allowScheduling":true, "tags":[ "dicom_stor" ]}]'