General IO performance of Longhorn storage backend

Hello everyone,

I’ve been testing Longhorn for a while and the power and easy use of this storage backend is amazing.

What surprised me from beginning was sluggish performance, 10x slower than pod with mounted directory directly on the same host, some numbers below:

Longhorn PVC
test-with-only-ssd-detail

Mounted same folder to pod

Test-with-mapped-direct-folder-detail

Underlying HW
Rancher master and node are VMs running on top of Hyper-v with Storage pools where are 4 SSD drives binded in one big pool.

As I haven’t seen discussed anywhere, I assume it’s something with my setup. How performs your Longhorns in the wild?

Hi, @al3sss

It’s impossible to expect the distributed storage software has the same speed as the underlying hardware, since it added the feature and high availability on top of the underlying hardware… Though we’re still in the process of tuning Longhorn. So far we haven’t spent much effort on performance since we want it to be easy to use and powerful first. Once it’s fully tuned you can expect much better performance.

And you can also try the other open source distributed storage softwares and see how they perform as well.

Another thing is the benchmark software makes huge differences, since there are many factors determined the speed of the storage. Which benchmark software you’re using?

Hi @yasker,

Thank you for honest answer. You are right, every single layer of storage logic is cutting down performance. I was not seeking for IO from beginning but I noticed delays on my app stacks…

For sure will do, or wait for 1.0 of Longhorn itself :wink:

I used from both pods https://bench.sh/

Time flies and Longhorn is not a baby anymore… I repeated tests above.

TLDR It’s better but not much. Looking for your numbers and setups folks!

I used NVME for storage: micron 1100 512GB

Longhorn

Mounted folder on same drive on host

Thanks for the info.

As some discussed above, longhorn is a distributed data storage, so some cost is required for atomic operations across replicas for crash consistency claim. In 1.1.0, data locality feature introduced, it supports best-effort mode to keep the mounted volume along with the replica/data, so it should provide some extra benefit in some conditions.

ref: Longhorn | Documentation

We also plan to have some performance benchmark after the coming release, so probably we can provide more info for reference in the future.

That’s quite good actually, considering the amount of fsync involved.