Convoy-glusterfs performance help

I’ve deployed convoy-glusterfs on four ec2 m4.large hosts with SSD storage.

I’ve mounted these on a test container running on the same hosts with the gluster-server

From my test container I see:

root@068297389b78:/# df
Filesystem             1K-blocks     Used Available Use% Mounted on
none                   264084624  8914220 244333832   4% /
tmpfs                    4087120        0   4087120   0% /dev
tmpfs                    4087120        0   4087120   0% /sys/fs/cgroup
glusterfs:/sharedspace 264084608 20829824 232418176   9% /shared
/dev/xvda1             264084624  8914220 244333832   4% /etc/hosts
shm                        65536        0     65536   0% /dev/shm

Reasonable performance when access the EBS directory from the container.

root@068297389b78:/# dd of=/tmp/BIG if=/dev/zero bs=1024 count=1000
1000+0 records in
1000+0 records out
1024000 bytes (1.0 MB) copied, 0.00302365 s, 339 MB/s

But pretty awful performance (and consistent) while utilizing the shared space.

root@068297389b78:/# dd of=/shared/BIG if=/dev/zero bs=1024 count=1000
1000+0 records in
1000+0 records out
1024000 bytes (1.0 MB) copied, 0.200207 s, 5.1 MB/s
root@068297389b78:/#rancher

From a glusterfs-server I see:

[root@glusterfs_glusterfs-server_1 /]# gluster volume status
Status of volume: sharedspace
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.42.29.186:/data/glusterfs/brick1/s
haredspace                                  49152     0          Y       15
Brick 10.42.65.165:/data/glusterfs/brick1/s
haredspace                                  49152     0          Y       16
Brick 10.42.222.19:/data/glusterfs/brick1/s
haredspace                                  49152     0          Y       15
NFS Server on localhost                     N/A       N/A        N       N/A
Self-heal Daemon on localhost               N/A       N/A        Y       38
NFS Server on 10.42.65.165                  N/A       N/A        N       N/A
Self-heal Daemon on 10.42.65.165            N/A       N/A        Y       43
NFS Server on 10.42.222.19                  N/A       N/A        N       N/A
Self-heal Daemon on 10.42.222.19            N/A       N/A        Y       42

Task Status of Volume sharedspace
------------------------------------------------------------------------------
There are no active volume tasks

Are there some basic configuration or tuning tips I can use to increase the performance of convoy-glusterfs?

I have seen some posts suggesting using convoy-nfs which is considered more mature, but I have not seen an rancher catalog application to actually setup the nsf server, just the convoy nfs client driver.

Any pointers ?

Thanks,
K

HI,

You can check this discussion:

I didn’t see anyone having good performance with Convoy-GlusterFs. There are a problem somewhere. But no idea How to fix it. Possibly something in the way the network works with Rancher:

I’m still testing Rancher, so no production feedback, but yes definitely NFS is working better. You can configure an NFS server (without replication) relatively quickly without using Rancher and so connectecing Convoy on it.