Storageos, rancher and Kubelet

Rancher 2.0 Tech Preview
1

Hi,
I’ve got a storageos deployed on rancher-kubernetes (rancher 2.0.0-beta3/kubernetes v1.10.0-rancher1).
After deploying an PersistenceVolumeClaim. I get an error similar to this one:

 MountVolume.SetUp failed for volume "pvc-add90dd2-4474-11e8-a1f6-96000009935a" : stat /var/lib/storageos/volumes/52f21c44-9175-e018-b2c7-96c2a8a58560: no such file or directory

The volume is existing as such it’s related to kubelet being in a container.
I’ve found a possible solution to that in google-cache which was on the storageos site. It mentioned that I have to add --volume=/var/lib/storageos:/var/lib/storageos:rshared to the running kubelet. But I’ve got no idea how to manage it properly.

Can you give me a hint?

Thanks
marc

1 Like
2

Hi,
I’m having the same issue with StorageOS on Rancher v2.0.0 release (installed yesterday). Did you have any luck getting this fixed? I think we’re very close as the volumes are provisioning, just not available to the kubelet check.

Per storageos documentation:

  1. Enable the MountPropagation flag by appending --feature-gates MountPropagation=true to the kube-apiserver and kubelet services.
  2. For deployments where the kubelet runs in a container (eg. OpenShift, CoreOS, Rancher), add --volume=/var/lib/storageos:/var/lib/storageos:rshared to each of the kubelets.

This seems to be hard to do with RKE provisioning the kubelet… perhaps there is an easy solution that I’m missing as well?

My exact error:

Events:
  Type     Reason       Age                From               Message
  ----     ------       ----               ----               -------
  Warning  FailedMount  19m (x27 over 1h)  kubelet, rancher2  Unable to mount volumes for pod "test7777-7dbdf8b8c4-rr6nr_default(eaffa1ae-4e3e-11e8-a1e8-000c2929c6a3)": timeout expired waiting for volumes to attach or mount for pod "default"/"test7777-7dbdf8b8c4-rr6nr". list of unmounted volumes=[test]. list of unattached volumes=[test default-token-fwslf]
  Warning  FailedMount  4m (x45 over 1h)   kubelet, rancher2  MountVolume.SetUp failed for volume "pvc-eaf8533e-4e3e-11e8-a1e8-000c2929c6a3" : stat /var/lib/storageos/volumes/d8470673-a470-6167-a25b-e0c109c5b2bf: no such file or directory
3

Hey, this is Cheryl from StorageOS.

If you have the Rancher UI up, go into the infrastructure stack and click to upgrade the kubelet. There you can add the /var/lib/storageos volume in and upgrade, and any future kubelet will inherit the setting too.

Hope that helps!

4

Hey,

thanks for the reply. But this does not help if u are using Rancher 2.0, since there no stacks or kublets to upgrade.

Did anyone got StorageOS working in Rancher 2.0? Im stuck with the same problems.

5

Same, Trying hard to find a way to get it to work.

6

Any updates on whether anyone got it to work for Rancher 2.0?

7

It looks like most StorageClass plugins except vSphere are broken in the current Rancher 2.x, but the fix has already been merged. I’m looking forward to trying Rancher - StorageOS after the next RKE release.

8

After I have added the volume mapping on the kublet

kubelet:
  extra_args:
     feature-gates: "PersistentLocalVolumes=true, VolumeScheduling=true,MountPropagation=true"
  fail_swap_on: false
  extra_binds:
     - "/var/lib/storageos:/var/lib/storageos:rshared"

I get the following error when the containers start up

MountVolume.SetUp failed for volume "pvc-a8b3bec5-01eb-11e9-97c1-00505691ceae" : API error (Unauthenticated access of secure endpoint, please retry after authentication): Failed to find account

I am very new to the topic at all (k8s, rancher, storageos) so any help/hint is appreciated