Rancher 2.x Rook.io problem

Rancher 2.x
1

Hi,

I have a rancher 2.x managed cluster deployed over digitaloceal. It has three nodes based on ubuntu 16.04.

I have followed this guide
https://rook.io/docs/rook/master/quickstart.html

It seems everythin is OK. The PVC are bound but the pods fail to deploy. All rook related pod are running.

This is the output of get pvc
wp-pv-claim Bound pvc-0875428f-5d0a-11e8-9223-0ab22348a3dd 1Gi RWO rook-block 18h

This is the output of the pod description
Warning FailedMount 34s (x479 over 18h) kubelet, node3 Unable to mount volumes for pod “wordpress-mysql-5bbddff49c-xr786_default(00275a1b-5d0a-11e8-9223-0ab22348a3dd)”: timeout expired waiting for volumes to attach or mount for pod “default”/“wordpress-mysql-5bbddff49c-xr786”. list of unmounted volumes=[mysql-persistent-storage]. list of unattached volumes=[mysql-persistent-storage default-token-bqjlr]

2

Hi there,

I have the exact same problem here, have you managed to get any further with this?

Regards,

3

Hi,

I think I found the solution for my problem. First, I access everynode and make /var/lib/kubelet/volumeplugins chmod 777. Then on the rook-operator.yaml I uncommented the two lines regarding the volumeplugin folder location. This has worked for me on RancherOS and Ubuntu 16.04 nodes

I hope you can solve your problem also

4

Thanks very much, will check it out. I did get this working on centos the other day just by adding the two path bits. An oversight I missed originally. Ideally you never want to use 777 and I’m 99% sure it should not be needed. I haven’t yet retested this with rancher, I switched to kubespey via ansible for now due to some other rancher bugs which was annoying me no end.

5

I have 3 nodes with RancherOS 1.3.0 (and testing with 3 nodes ros 1.4.0)

mounts:
- - /dev/sdb
  - /data
  - ext4
  - ""
rancher:
   autologin: ttyS0
   console: ubuntu
   docker:
     engine: docker-17.03.2-ce
     graph: /data
     storage_driver: overlay2

with the docker installed in the /data directory.

set installed rook with /var/lib/kubelet/volumeplugins (data on the host appeared)

rancher@docker-vm13:~$ ls /var/lib/kubelet/volumeplugins 
ceph.rook.io~rook  ceph.rook.io~rook-ceph-system  rook.io~rook	rook.io~rook-ceph-system

create rook with mysql/wordpress example

kubectl create -f  operator.yaml
kubectl create -f cluster.yaml
kubectl create -f https://raw.githubusercontent.com/rook/rook/master/cluster/examples/kubernetes/mysql.yaml
kubectl create -f https://raw.githubusercontent.com/rook/rook/master/cluster/examples/kubernetes/wordpress.yaml

creates pvm

Screenshot%20(1)
Screenshot%20(1)1383Ă—335 29.5 KB

but the containers do not start
Screenshot
Screenshot1403Ă—451 38.1 KB

kubectl -n rook-ceph-system logs -f rook-discover-5lzkv (infinity write this every 30 seconds)

2018-06-06 08:05:53.282729 I | rook-discover: updating device configmap
2018-06-06 08:05:53.282796 I | exec: Running command: lsblk --all --noheadings --list --output KNAME
2018-06-06 08:05:53.288272 I | exec: Running command: lsblk /dev/loop1 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2018-06-06 08:05:53.291832 W | inventory: skipping device loop1: Failed to complete 'lsblk /dev/loop1': exit status 1. 
2018-06-06 08:05:53.291865 I | exec: Running command: lsblk /dev/sdb --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2018-06-06 08:05:53.294755 I | exec: Running command: udevadm info --query=property /dev/sdb
2018-06-06 08:05:53.297263 I | exec: Running command: udevadm info --query=property /dev/sdb
2018-06-06 08:05:53.299891 I | exec: Running command: lsblk /dev/loop6 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2018-06-06 08:05:53.302668 W | inventory: skipping device loop6: Failed to complete 'lsblk /dev/loop6': exit status 1. 
2018-06-06 08:05:53.302707 I | exec: Running command: lsblk /dev/loop4 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2018-06-06 08:05:53.305731 W | inventory: skipping device loop4: Failed to complete 'lsblk /dev/loop4': exit status 1. 
2018-06-06 08:05:53.305762 I | exec: Running command: lsblk /dev/sr0 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2018-06-06 08:05:53.308949 I | exec: Running command: lsblk /dev/loop2 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2018-06-06 08:05:53.311852 W | inventory: skipping device loop2: Failed to complete 'lsblk /dev/loop2': exit status 1. 
2018-06-06 08:05:53.311896 I | exec: Running command: lsblk /dev/loop0 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2018-06-06 08:05:53.314707 W | inventory: skipping device loop0: Failed to complete 'lsblk /dev/loop0': exit status 1. 
2018-06-06 08:05:53.314748 I | exec: Running command: lsblk /dev/sda --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2018-06-06 08:05:53.318033 I | exec: Running command: udevadm info --query=property /dev/sda
2018-06-06 08:05:53.321099 I | exec: Running command: udevadm info --query=property /dev/sda
2018-06-06 08:05:53.324030 I | exec: Running command: lsblk /dev/sda1 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2018-06-06 08:05:53.328649 I | exec: Running command: udevadm info --query=property /dev/sda1
2018-06-06 08:05:53.331948 I | exec: Running command: lsblk /dev/loop7 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2018-06-06 08:05:53.335981 W | inventory: skipping device loop7: Failed to complete 'lsblk /dev/loop7': exit status 1. 
2018-06-06 08:05:53.336022 I | exec: Running command: lsblk /dev/loop5 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2018-06-06 08:05:53.338982 W | inventory: skipping device loop5: Failed to complete 'lsblk /dev/loop5': exit status 1. 
2018-06-06 08:05:53.339013 I | exec: Running command: lsblk /dev/loop3 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2018-06-06 08:05:53.341732 W | inventory: skipping device loop3: Failed to complete 'lsblk /dev/loop3': exit status 1. 
2018-06-06 08:05:53.341768 I | exec: Running command: lsblk /dev/sdb --bytes --pairs --output NAME,SIZE,TYPE,PKNAME
2018-06-06 08:05:53.344809 I | exec: Running command: udevadm info --query=property /dev/sdb
2018-06-06 08:05:53.347293 I | exec: Running command: lsblk /dev/sda --bytes --pairs --output NAME,SIZE,TYPE,PKNAME
2018-06-06 08:05:53.350987 I | exec: Running command: udevadm info --query=property /dev/sda1
2018-06-06 08:05:53.354279 I | exec: Running command: udevadm info --query=property /dev/sda1
2018-06-06 08:05:53.357668 I | exec: Running command: udevadm info --query=property /dev/sda
2018-06-06 08:05:53.360444 I | rook-discover: available devices: [{Name:sdb Parent: HasChildren:false DevLinks:/dev/disk/by-id/ata-QEMU_HARDDISK_QM00007 /dev/disk/by-uuid/a659a83d-4269-4816-92f5-f118da828da6 Size:107374182400 UUID:a659a83d-4269-4816-92f5-f118da828da6 Serial:QEMU_HARDDISK_QM00007 Type:disk Rotational:true Readonly:false Partitions:[] Filesystem:ext4 Vendor: Model:QEMU_HARDDISK WWN: WWNVendorExtension: Empty:true} {Name:sda Parent: HasChildren:false DevLinks:/dev/disk/by-id/ata-QEMU_HARDDISK_QM00005 Size:10737418240 UUID:4003f5e1 Serial:QEMU_HARDDISK_QM00005 Type:disk Rotational:true Readonly:false Partitions:[{Name:sda1 Size:10735321088 Label: Filesystem:ext4}] Filesystem: Vendor: Model:QEMU_HARDDISK WWN: WWNVendorExtension: Empty:true}]
6

I tried the chmod 777 but it did not work. Any other solutions to this one?

8

@neo1 - You do NOT need to chmod 777 /var/lib/kubelet/volumeplugins to make rook work. You need to configure the operator to use that folder for it’s flexvolume as documented here: https://rook.io/docs/rook/v0.8/flexvolume.html

@Victor_H_Saez - The reason your getting that error is due to the OSD’s not properly being created. Check your cluster config and operator logs.

1 Like
9

@Erik_Van_Norstrand Thanks! that solved the problem!