rhugga
December 17, 2018, 3:30pm
1
I’ve nearly pulled out all my hair. None of the work-arounds are working for us.
Show stopper for us… at a pivotal point where we have to decide to roll k8 clusters from scratch or forge forward with rancher. We’d love to move forward with rancher, we like the BUI and how easy it is for our dba’s and developers.
Here is my submitted issue with details on our current roadblock:
opened 03:12PM - 17 Dec 18 UTC
closed 12:20AM - 15 Nov 19 UTC
area/volume
area/volume/provisioning
## Description
Container failing to mount iscsi volume:
```
Events:
Type… Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m23s (x3 over 2m23s) default-scheduler persistentvolumeclaim "demo-vol1-claim" not found
Warning FailedScheduling 2m18s (x22 over 2m23s) default-scheduler pod has unbound PersistentVolumeClaims (repeated 29 times)
Normal SuccessfulAttachVolume 2m16s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-40d40931-020b-11e9-9f7a-008cfae6ae7c"
Warning FailedMount 14s kubelet, st14p01if-ztds09354301 Unable to mount volumes for pod "percona-66b4fd4ddf-r6rsg_openebs(40b10bcb-020b-11e9-a8b7-008cfae69fd6)": timeout expired waiting for volumes to attach or mount for pod "openebs"/"percona-66b4fd4ddf-r6rsg". list of unmounted volumes=[demo-vol1]. list of unattached volumes=[demo-vol1 default-token-n8pq2]
Warning FailedMount 7s kubelet, st14p01if-ztds09354301 MountVolume.WaitForAttach failed for volume "pvc-40d40931-020b-11e9-9f7a-008cfae6ae7c" : failed to get any path for iscsi disk, last err seen:
iscsi: failed to attach disk: Error: iscsiadm: Could not login to [iface: default, target: iqn.2016-09.com.openebs.cstor:pvc-40d40931-020b-11e9-9f7a-008cfae6ae7c, portal: 172.31.29.21,3260].
iscsiadm: initiator reported error (8 - connection timed out)
iscsiadm: Could not log into all portals
Logging in to [iface: default, target: iqn.2016-09.com.openebs.cstor:pvc-40d40931-020b-11e9-9f7a-008cfae6ae7c, portal: 172.31.29.21,3260] (multiple)
(exit status 8)
```
## Current Behavior
iscsiadm fails to login.
## Your Environment
```
k8s: 1.11.5
rke: 2.1.3
kernel: 4.14.35-1818.3.3.el7uek.x86_64
docker: 18.6.1
openebs: 0.8
os: OracleLinux 7.5
```
Kubelet is running with this config:
```
kubelet:
extra_binds:
- "/etc/iscsi:/etc/iscsi"
- "/sbin/iscsiadm:/sbin/iscsiadm"
- "/lib/modules"
```
The iscsi_tcp module is loaded on the bare metal nodes:
```
# lsmod | grep iscsi
iscsi_tcp 20480 0
libiscsi_tcp 24576 1 iscsi_tcp
libiscsi 57344 2 libiscsi_tcp,iscsi_tcp
scsi_transport_iscsi 98304 3 libiscsi,iscsi_tcp
```
Container using iscsiadm from bare metal node:
```
-126-root@st14p01if-ztds09351001:~ # iscsiadm -V
iscsiadm version 6.2.0.874-7
-0-root@st14p01if-ztds09351001:~ # docker exec -it kubelet /sbin/iscsiadm -V
iscsiadm version 6.2.0.874-7
```
Thanks for any help,
Hi Ben,
Which OS are you planning to use? If you are using rancher os, then the setting up of iSCSI initiator is pretty easy. Here are the steps that one of the user followed:
opened 04:44AM - 03 Jan 19 UTC
closed 02:28PM - 08 Mar 19 UTC
@chtardif tried this and found the following steps work:
1) Install rancheros…
2) Install open-iscsi support
- ros s enable open-iscsi
- ros s up open-iscsi
- These two commands will basically bring iscsi binaries + kernel modules to the host, but on a separate system container, as everything runs in containers on RancherOS
3) Install OpenEBS (+ SPC's, SC's, you know the job)
4) Create your PVC and your container, and enjoy the thing !!!
Thanks