Rancher Volume not mounting in container OS

Longhorn
1

My storage class is auto provisioning volumes, the volumes get bound, and the mounts succeed in the Events for the pod, but the volume never actually mounts in the container OS. All the volumes show up in /proc/partitions but not in the “mount” listing. Instead all the persistent volume mounts appear to try to remount /dev/sda.

volumeMounts:

  • mountPath: /mssql-data/data
    name: data
  • mountPath: /mssql-translog/translog
    name: transactionlog
  • mountPath: /mssql-backup/backup
    name: backup
  • mountPath: /mssql-data/master
    name: master
    dnsPolicy: ClusterFirst
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext: {}
    terminationGracePeriodSeconds: 30
    volumes:
  • name: data
    persistentVolumeClaim:
    claimName: mssql-linux-mssql-linux-data
  • name: transactionlog
    persistentVolumeClaim:
    claimName: mssql-linux-mssql-linux-translog
  • name: backup
    persistentVolumeClaim:
    claimName: mssql-linux-mssql-linux-backup
  • name: master
    persistentVolumeClaim:
    claimName: mssql-linux-mssql-linux-master

root@mssql-linux-mssql-linux-95d987f66-6rz4d:/# mount | grep mssql
/dev/sda on /mssql-data/data type ext4 (rw,relatime,data=ordered)
/dev/sda on /mssql-translog/translog type ext4 (rw,relatime,data=ordered)
/dev/sda on /mssql-backup/backup type ext4 (rw,relatime,data=ordered)
/dev/sda on /mssql-data/master type ext4 (rw,relatime,data=ordered)
root@mssql-linux-mssql-linux-95d987f66-6rz4d:/#
root@mssql-linux-mssql-linux-95d987f66-6rz4d:/# cat /proc/partitions
major minor #blocks name

11 0 102400 sr0
8 0 102400000 sda
8 64 10485760 sde
8 16 1048576 sdb
8 32 1048576 sdc
8 48 1048576 sdd
8 96 1048576 sdg
root@mssql-linux-mssql-linux-95d987f66-6rz4d:/#

2

ContainerOS? You meant CoreOS or Google Container optimized OS? The latter lacks of iscsiadm so it cannot support iSCSI, thus cannot be supported by Longhorn.

You can check the log of Longhorn Manager to see if anything wrong.

3

When I refer to “container OS” I am referring to the actual container itself, not the node OS.

I am running RKE-based nodes provisioned directly through Rancher 2.0. I am seeing no errors in the Longhorn Manager or in the K8s event log. The devices show up in the container but do not mount on container start.

4

Volume won’t be mounted inside container directly, it’s mounted on the host then bind-mounted to container.

Do you see lost_and_found inside your directories? That’s created by mkfs.ext4, which is a easy way to verify that if the volume was mounted correctly. You can check the host’s mount table and inspect the container information for more details.

5

Nope, there is no lost+found in any of the persistent volume directories. The volumes show in /proc/partitions but the mounts all show /dev/sda being remounted.

bclark@skyfire:~$ kubectl -n mssql-linux exec -it mssql-linux-mssql-linux-95d987f66-6rz4d /bin/bash

root@mssql-linux-mssql-linux-95d987f66-6rz4d:/# mount | grep mssql
/dev/sda on /mssql-data/data type ext4 (rw,relatime,data=ordered)
/dev/sda on /mssql-translog/translog type ext4 (rw,relatime,data=ordered)
/dev/sda on /mssql-backup/backup type ext4 (rw,relatime,data=ordered)
/dev/sda on /mssql-data/master type ext4 (rw,relatime,data=ordered)
root@mssql-linux-mssql-linux-95d987f66-6rz4d:/#

root@mssql-linux-mssql-linux-95d987f66-6rz4d:/# cd /mssql-data/data
root@mssql-linux-mssql-linux-95d987f66-6rz4d:/mssql-data/data# ls -la
total 8
drwxr-x— 2 root root 4096 Aug 30 05:25 .
drwxr-xr-x 4 root root 4096 Aug 30 05:27 …

root@mssql-linux-mssql-linux-95d987f66-6rz4d:/mssql-data# cd /mssql-backup/backup
root@mssql-linux-mssql-linux-95d987f66-6rz4d:/mssql-backup/backup# ls -la
total 8
drwxr-x— 2 root root 4096 Aug 30 05:25 .
drwxr-xr-x 3 root root 4096 Aug 30 05:27 …
root@mssql-linux-mssql-linux-95d987f66-6rz4d:/mssql-backup/backup

root@mssql-linux-mssql-linux-95d987f66-6rz4d:/# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 100267080 59320872 35809824 63% /
tmpfs 4085400 0 4085400 0% /dev
tmpfs 4085400 0 4085400 0% /sys/fs/cgroup
/dev/sda 100267080 59321160 35809536 63% /etc/hosts
shm 65536 0 65536 0% /dev/shm
tmpfs 4085400 12 4085388 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 4085400 0 4085400 0% /sys/firmware
root@mssql-linux-mssql-linux-95d987f66-6rz4d:/#
root@mssql-linux-mssql-linux-95d987f66-6rz4d:/#
root@mssql-linux-mssql-linux-95d987f66-6rz4d:/# cat /proc/partitions
major minor #blocks name

11 0 102400 sr0
8 0 102400000 sda
8 64 10485760 sde
8 16 1048576 sdb
8 32 1048576 sdc
8 48 1048576 sdd
8 96 1048576 sdg

6

I will check the host/node mount to see if I am seeing it mounted there.