Rancher catalog, GlusterFS, how to choose where bricks get created?

Archive Convoy
1

Hi.

If I create a glusterfs cluster using GlusterFS stack from rancher’s catalog, I can’t find a way to specify that I want to use a specific directory for gluster’s bricks. For example, I’d like to use /data/brick1. Is there any way to do this?

I installed rancher on Ubuntu servers with 2 HDs. I want to use the second disk for my replicated storage volume, create a glusterfs cluster (using rancher’s catalog) and then use convoy stack for glusterfs.

2

Since the gluster bricks are in volume containers, I could be wrong, but I don’t think that you can move them to a specific device. They just end up where docker puts its containers (/var/lib/docker in my case).

What I ended up doing was moving /var/lib/docker to an lvm volume so I could expand it as needed.

3

Well, I did something like this:

Reviewing https://github.com/rancher/catalog-dockerfiles/blob/master/glusterfs/0.2.0/docker-compose.yml, I realized that glusterfs uses a data volume container:

glusterfs-data:
  image: rancher/glusterfs:v0.1.3
  command: /bin/true
  volumes:
    - /var/run
  labels:
    io.rancher.container.hostname_override: container_name
    io.rancher.container.start_once: true

So, I check the image’s Dockerfile (https://github.com/rancher/catalog-dockerfiles/blob/master/glusterfs/containers/0.1.3/glusterfs/Dockerfile). It has the line:

VOLUME ["/data/glusterfs/brick1"]

As I didn’t find the way to parameterize this volume, I ended up creating a private catalog cloning rancher’s glusterfs catalog and adding the following “volume” line in docker-compose.yml:

Before:

glusterfs-data:
  image: rancher/glusterfs:v0.1.3
  command: /bin/true
  volumes:
    - /var/run
  labels:
    io.rancher.container.hostname_override: container_name
    io.rancher.container.start_once: true

After:

glusterfs-data:
  image: rancher/glusterfs:v0.1.3
  command: /bin/true
  volumes:
    - /var/run
    - /storage/glusterfs/brick1:/data/glusterfs/brick1
  labels:
    io.rancher.container.hostname_override: container_name
    io.rancher.container.start_once: true

/storage/glusterfs/brick1 is the host’s directory where I mounted my /dev/sdb1 partition.

That worked but,… I’d prefer that I could specify the directory I want to use as gluster’s brick.

There’s a new version of glusterfs catalog. I’ll try it.

4

This is not related to convoy but you shouldn’t keep /var/lib/docker for production deployments. Should create 2 LVM volumes and then assign them to docker with the devicemapper storage driver.

5

If you are going through the effort of creating a private custom catalog, you can parameterize the volume location by adding another question to the rancher-compose.yml

.catalog:
  name: "GlusterFS"
  version: "3.7.9-rancher1"
  description: "Gluster FS (3x) replicated volume"
  questions:
    - variable: "volume_path"
      description: "Path to volume on host system"
      label: "Volume Name"
      required: true
      default: "/storage/glusterfs/brick1"
      type: "string"

And add the variable to the docker-composer.yml

glusterfs-data:
  image: rancher/glusterfs:v0.1.3
  command: /bin/true
  volumes:
    - /var/run
    - /storage/glusterfs/brick1:${volume_path}
  labels:
    io.rancher.container.hostname_override: container_name
    io.rancher.container.start_once: true

See http://docs.rancher.com/rancher/catalog/#creating-private-catalogs for more details.

6

Great idea, Jason. I will try it on Monday. Thank you!

7

@marceloaguero Did it work?!