I am searching for some help to get OpenEBS set up on my Rancher Cluster. My goal is to get automatic provisioning with the OpenEBS Dynamic Local PV provisioner with hostpath running.
My setup is a single node cluster on a root server from Netcup. I use a single node Rancher Server installation on the same server with non-default ports mapped for http/https.
In my tries, the Kubernetes clusters which I am trying to install OpenEBS on were freshly provisioned with the respective Rancher Server version of my installation.
First try with Rancher 2.3.0:
- Here installation of OpenEBS with default configuration options already failed. As 2.2.8 is currently mentioned as stable and I am a newbie regarding Kubernetes and Rancher I switched to 2.28 because of this.
Second try with Rancher 2.2.8
- Installation of OpenEBS seems to work. The pods are all up and running according to the description on the OpenEBS website. And the default storage classes for openebs were also installed.
I now have tried two things:
Enable Grafana/Promotheus Monitoring on my Rancher cluster by “Enable Persistent Storage for Grafana/Promotheus” using “openebs-hostpath” the pods simply do not come up and I cannot see any output in the logs. I can see the relevant pvc’s but the pv’s are not created.
In order to debug the stuff I followed the example from the openebs website
and used the chart on this page to spin up a percona database with the “openebs-hostpath” storage class. The pod is also not coming up. When using
kc describe pod percona-5f878cbbd5-c4dnj
I get the output attached below. One further note, I am using RancherOS as operating system and before installing things I activated iscsi according to the description on the openebs page:
sudo ros s enable open-iscsi
sudo ros s up open-iscsi
ros config set rancher.services.user-volumes.volumes [/home:/home,/opt:/opt,/var/lib/kubelet:/var/lib/kubelet,/etc/kubernetes:/etc/kubernetes,/var/openebs]
system-docker rm all-volumes
One further note: When starting the deployment according to the log below, I can see that openebs successfully create a folder for the host volume given with this path:
even so the log below mentions that it cannot create the path.
MountVolume.NewMounter initialization failed for volume “pvc-caec8f21-edd6-11e9-89d7-56ade17ea74b” : path “/var/openebs/local/pvc-caec8f21-edd6-11e9-89d7-56ade17ea74b” does not exist
However, the folder is empty, I would assume to have some file representing the volume in there. Can someone help me to get this openebs stuff running? I have no idea how to narrow down the issue further
Thanks a lot for your help!
Start Time: Sun, 13 Oct 2019 18:30:45 +0200
Controlled By: ReplicaSet/percona-5f878cbbd5
Host Port: 0/TCP
Restart Count: 0
/var/lib/mysql from demo-vol1 (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-f9789 (ro)
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
Type: Secret (a volume populated by a Secret)
QoS Class: Burstable
node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Type Reason Age From Message
Warning FailedScheduling 4m47s default-scheduler persistentvolumeclaim “demo-vol1-claim” not found
Normal Scheduled 4m41s default-scheduler Successfully assigned default/percona-5f878cbbd5-c4dnj to v220191010512198810
Warning FailedMount 31s (x10 over 4m41s) kubelet, v220191010512198810 MountVolume.NewMounter initialization failed for volume “pvc-caec8f21-edd6-11e9-89d7-56ade17ea74b” : path “/var/openebs/local/pvc-caec8f21-edd6-11e9-89d7-56ade17ea74b” does not exist
Warning FailedMount 22s (x2 over 2m38s) kubelet, v220191010512198810 Unable to mount volumes for pod “percona-5f878cbbd5-c4dnj_default(cadfbaad-edd6-11e9-89d7-56ade17ea74b)”: timeout expired waiting for volumes to attach or mount for pod “default”/“percona-5f878cbbd5-c4dnj”. list of unmounted volumes=[demo-vol1]. list of unattached volumes=[demo-vol1 default-token-f9789]