Convoy-glusterfs performance help

[kkvilekval]

I’ve deployed convoy-glusterfs on four ec2 m4.large hosts with SSD storage.

I’ve mounted these on a test container running on the same hosts with the gluster-server

From my test container I see:

root@068297389b78:/# df
Filesystem             1K-blocks     Used Available Use% Mounted on
none                   264084624  8914220 244333832   4% /
tmpfs                    4087120        0   4087120   0% /dev
tmpfs                    4087120        0   4087120   0% /sys/fs/cgroup
glusterfs:/sharedspace 264084608 20829824 232418176   9% /shared
/dev/xvda1             264084624  8914220 244333832   4% /etc/hosts
shm                        65536        0     65536   0% /dev/shm

Reasonable performance when access the EBS directory from the container.

root@068297389b78:/# dd of=/tmp/BIG if=/dev/zero bs=1024 count=1000
1000+0 records in
1000+0 records out
1024000 bytes (1.0 MB) copied, 0.00302365 s, 339 MB/s

But pretty awful performance (and consistent) while utilizing the shared space.

root@068297389b78:/# dd of=/shared/BIG if=/dev/zero bs=1024 count=1000
1000+0 records in
1000+0 records out
1024000 bytes (1.0 MB) copied, 0.200207 s, 5.1 MB/s
root@068297389b78:/#rancher

From a glusterfs-server I see:

[root@glusterfs_glusterfs-server_1 /]# gluster volume status
Status of volume: sharedspace
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.42.29.186:/data/glusterfs/brick1/s
haredspace                                  49152     0          Y       15
Brick 10.42.65.165:/data/glusterfs/brick1/s
haredspace                                  49152     0          Y       16
Brick 10.42.222.19:/data/glusterfs/brick1/s
haredspace                                  49152     0          Y       15
NFS Server on localhost                     N/A       N/A        N       N/A
Self-heal Daemon on localhost               N/A       N/A        Y       38
NFS Server on 10.42.65.165                  N/A       N/A        N       N/A
Self-heal Daemon on 10.42.65.165            N/A       N/A        Y       43
NFS Server on 10.42.222.19                  N/A       N/A        N       N/A
Self-heal Daemon on 10.42.222.19            N/A       N/A        Y       42

Task Status of Volume sharedspace
------------------------------------------------------------------------------
There are no active volume tasks

Are there some basic configuration or tuning tips I can use to increase the performance of convoy-glusterfs?

I have seen some posts suggesting using convoy-nfs which is considered more mature, but I have not seen an rancher catalog application to actually setup the nsf server, just the convoy nfs client driver.

Any pointers ?

Thanks,
K

[seb2411]

HI,

You can check this discussion:

Convoy-gluster on specific storage hosts

I was thinking the limitation is possibly the network. All the VPS propose in general 100Mbps or 200Mbps. If I’m not wrong : 200 Mbps = 25MB/S so To send 1GO we need a minimum of 40s. So we need a 4Gbps network to allow similar performances to a SSD.

I didn’t see anyone having good performance with Convoy-GlusterFs. There are a problem somewhere. But no idea How to fix it. Possibly something in the way the network works with Rancher:

Overlay network performance

Hi, I’m testing rancher (cattle) and I’m noticing slow network performances. I’ve created a rancher environment with 2 agent nodes (public1, public2), added a container stack with a container in each node (agent1, agent2) these are the measurements I get with iperf(iperf -i 1 -t 3): from public2 to container1 [ 3] 0.0- 1.0 sec 113 MBytes 948 Mbits/sec [ 3] 1.0- 2.0 sec 113 MBytes 947 Mbits/sec [ 3] 2.0- 3.0 sec 112 MBytes 940 Mbits/sec [ 3] 0.0- 3.0 sec 338 MBytes…

I’m still testing Rancher, so no production feedback, but yes definitely NFS is working better. You can configure an NFS server (without replication) relatively quickly without using Rancher and so connectecing Convoy on it.