Issues creating volumes with rancher-ebs

Rancher 1.x
1

Hi,

I have an environment configured with rancher-ebs (v0.8.1). I’m trying to automatically create and mount an EBS volume using the following compose file:

version: '2'
volumes:
  elasticsearch:
    driver_opts:
      name: elasticsearch
      size: '20'
      volumeType: gp2
    driver: rancher-ebs
services:
  elasticsearch:
    mem_limit: 2147483648
    image: rebs/elasticsearch:latest
    volumes:
    - elasticsearch:/usr/share/elasticsearch/data
    labels:
      io.rancher.scheduler.affinity:host_label_soft: lifecycle=on-demand

However, Rancher keeps creating new volumes, but does not mount and attach them to the container. The volumes created are attached to the host, but it seems to fail to create a filesystem. The container is stuck in “Starting”.

Here are the logs from the ebs-driver service:

time="2017-12-13T09:57:42Z" level=info msg=create.request name="megatest_elasticsearch_0b545" options=map[name:elasticsearch size:20 volumeType:gp2]
time="2017-12-13T09:57:52Z" level=error msg=create.response error="mke2fs 1.42.13 (17-May-2015) The file /dev/xvdl does not exist and no size was specified."
time="2017-12-13T09:57:52Z" level=info msg=create.request name="megatest_elasticsearch_0b545" options=map[name:elasticsearch size:20 volumeType:gp2]
time="2017-12-13T09:58:02Z" level=error msg=create.response error="mke2fs 1.42.13 (17-May-2015) The file /dev/xvdm does not exist and no size was specified."
time="2017-12-13T09:58:02Z" level=info msg=create.request name="megatest_elasticsearch_0b545" options=map[name:elasticsearch size:20 volumeType:gp2]
time="2017-12-13T09:58:12Z" level=error msg=create.response error="mke2fs 1.42.13 (17-May-2015) The file /dev/xvdn does not exist and no size was specified."
time="2017-12-13T09:58:12Z" level=info msg=create.request name="megatest_elasticsearch_0b545" options=map[name:elasticsearch size:20 volumeType:gp2]
time="2017-12-13T09:58:22Z" level=error msg=create.response error="mke2fs 1.42.13 (17-May-2015) The file /dev/xvdo does not exist and no size was specified."
time="2017-12-13T09:58:22Z" level=info msg=create.request name="megatest_elasticsearch_0b545" options=map[name:elasticsearch size:20 volumeType:gp2]
time="2017-12-13T09:58:32Z" level=error msg=create.response error="mke2fs 1.42.13 (17-May-2015) The file /dev/xvdp does not exist and no size was specified."

This exact setup used to work a few days ago, and works properly in other environments. As far as I know, nothing has changed in the environment or the Rancher stack. I have even tried recreating the environment from scratch and bringing up fresh hosts, with the same effects.

Does anyone know what could be going on ?

Thanks!