RWX-volume is significantly slower than RWO-volume

Longhorn
1

I am seeing a significant performance difference between RWX (ReadWriteMany) and RWO (ReadWriteOnce) types of volumes.

Here is the yaml that creates these two volumes

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-rwx
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: longhorn
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-rwo
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: longhorn

Here is a job/pod that mounts these two volumes inside it. let’s say the rwx volume is mounted at /volume-rwx, likewise, the rwo volume is mounted at /volume-rwo

Here is the yaml for the job

apiVersion: batch/v1
kind: Job
metadata:
  name: test
spec:
  backoffLimit: 1
  template:
    spec:
      containers:
      - args:
        - |
          tail -f /dev/null
        command:
        - sh
        - -c
        image: alpine:3.16
        name: test
        volumeMounts:
        - mountPath: /volume-rwx
          name: volume-rwx
        - mountPath: /volume-rwo
          name: volume-rwo
      restartPolicy: Never
      volumes:
      - name: volume-rwx
        persistentVolumeClaim:
          claimName: pvc-rwx
      - name: volume-rwo
        persistentVolumeClaim:
          claimName: pvc-rwo

For testing, I will be at each mounted location inside the test pod. later I will download Virtual-Box source (not important) and extract it.

Here is the result when the extract extract performed from /volume-rwo

/volume-rwo # wget https://download.virtualbox.org/virtualbox/6.1.38/VirtualBox-6.1.38.tar.bz2
Connecting to download.virtualbox.org (23.201.200.86:443)
saving to 'VirtualBox-6.1.38.tar.bz2'
VirtualBox-6.1.38.ta 100% |************************************************************************************************************************************|  158M  0:00:00 ETA
'VirtualBox-6.1.38.tar.bz2' saved
/volume-rwo # time -p tar -xf VirtualBox-6.1.38.tar.bz2 
real 49.29
user 43.55
sys 5.74
/volume-rwo # 
```on 

Also, this is the result when the same task is performed at /volume-rwx
```sh
/volume-rwx # wget https://download.virtualbox.org/virtualbox/6.1.38/VirtualBox-6.1.38.tar.bz2
Connecting to download.virtualbox.org (23.221.32.85:443)
saving to 'VirtualBox-6.1.38.tar.bz2'
VirtualBox-6.1.38.ta 100% |************************************************************************************************************************************|  158M  0:00:00 ETA
'VirtualBox-6.1.38.tar.bz2' saved
/volume-rwx # time -p tar -xf VirtualBox-6.1.38.tar.bz2 
real 574.35
user 56.88
sys 44.23
/volume-rwx #

evidently, rwx took 10 times as much time to perform the same extraction.
I have checked and found this issue was fixed at a very early stage.
Whereas I am using Longhorn (v1.3.1).

Please help me solve it.
Thank you