what is needed for LVM mirroring over two data centers?

SUSE Linux Enterprise Extensions SLES High Availability Extension
1

Hi experts,

we are having two data centers. On each data center site there is a SAN and many different virtual machines whose file systems are running on several LUNs of these SANs.

Per one virtual machine, I have added two LUNs for the basic operating system file system layout. EAch vritual machine has one LUN in the storage of DAta Center A and one LUN of Data Center B. Each LUN is accessible over two HBA’s via multipathing. I have made a volume group vg_rootfs, which contains several logical volumes. I have the logical volumes
Name - File System - Mount Point
lv_rootfs - ext3 -/
lv_root - ext3 - /root
lv_tmp - xfs - /tmp
lv_opt - ext3 - /opt
lv_usr - ext3 - /usr
lv_var_log - ext3 - /var/log
lv_var - ext3 - /var
lv_home - ext3 - /home
lv_swap - swap - swap

Then I configured a mirror between the two LUNs, so the LUN on data center A mirrors all it’s content to LUN on data center B via the logical volume manager. So I added a mirror to each logical volume via lvconvert. The command
OS>lsblk

correctly tells me that there are two mirror images, each one on a different LUN.

Then we did some HA tests. First we just removed one of the paths. The command
OS>multipath -l
correctly recognizes that one of the paths has become inactive, but the other still remains active and the OS can work.

Next we just took one of the LUN’s away from the virtual machines and that’s where the problems began.

After removing one of the LUNs from the system, the operating system is not able to write any files on the file systems anymore and shutdown as well as restarting the virtual machine is not possible until we give the LUN back to the virtual machine.

So I am asking: What do I have to do that the system still works after the storage/LUN on one data center fails? I am currently having the following concerns

  • Do I have to use another file system, like a “cluster aware file System”, like ofcs2?
  • Do I have to create a separate logical volume for the mount point /boot?
  • Do I have to setup clustered Logical Volume Manager for this? From my understanding cLVM is not what I want, because cLVM is for making one single LUN available for two hosts. I want to do the opposie. I want to make two Stroage-LUNs available for a single host.

Thank you for your input.

2

dafrk,

It appears that in the past few days you have not received a response to your
posting. That concerns us, and has triggered this automated reply.

These forums are peer-to-peer, best effort, volunteer run and that if your issue
is urgent or not getting a response, you might try one of the following options:

Be sure to read the forum FAQ about what to expect in the way of responses:
http://forums.suse.com/faq.php

If this is a reply to a duplicate posting or otherwise posted in error, please
ignore and accept our apologies and rest assured we will issue a stern reprimand
to our posting bot…

Good luck!

Your SUSE Forums Team
http://forums.suse.com

3

hi experts,

I gave up on trying lvm mirroring and switched to mdadm, still not working though (see other thread).

however, on Ubuntu, lvm mirror takeover is working. so something must have been wrong in our SLES environment, but I don’t know what it might have been.

4

On 07/28/2016 08:14 AM, dafrk wrote:[color=blue]

Hi experts,

we are having two data centers. On each data center site there is a SAN
and many different virtual machines whose file systems are running on
several LUNs of these SANs.[/color]
… lots of confusing stuff removed.

So… I think you want mirrored storage off two different storage devices (and
possible each in a different SAN).

Used mdadm to create a mirrored storage block device, each LUN of the mirror is
one from each storage device. To grow this I highly recommend you consider LVM
concatentation, in other words create new mirrored sets and add the mdadm block
device into the LVM VG and extend the LV and grow the fs.

Ideally what will happen is the multipaths for the failed storage will go down
and thus that storage goes down like a disk failure.

When that disk returns, ideally you’d hope it would be like a disk replace and
the mirror would resilver the returned LUN. Now… if that doesn’t happen,
then perhaps the returned LUN just stays “bad”. If it somehow comes back “good”
as a part of the mirror without rebuild, something is wrong and it will be
corrupted.

LVM should be none the wiser.

Latencies could be a big problem if the devices are remote and different.
iSCSI sounds like a good idea (route your “disk connection” across high latency
routes, etc), but the variables are immense, and you’ve sort of more than
doubled the amount of variables with your scenario.

Hope you find an acceptable solution. IMHO, one day something really bad will
happen and it won’t be recoverable. I recommend you do some very expensive and
high risk testing of your environment if you’re able to just so your management
doesn’t get any unrealistic expectations.

I’ve only done (I think) what you’re suggesting using two storage devices in the
same rack using fibre channel. While there’s still some risk, it’s much lower
since it’s fibre channel and very little latency issues (all local).

Other answers… IBM makes a sort of “abstractor” device that can mirror across
mixed vendor devices that sit behind it. Of course, IBM will tell you that you
need at least 3 of these so that you have one sitting in front of two… etc…
(a backup of a backup of a backup sort of thing). Nice part about these devices
is that to you it just looks like a single storage device, no mdadm involved at all.

I used to work for IBM and they are absolute evil (anti-FOSS), but if interested:

http://www-03.ibm.com/systems/storage/software/virtualization/svc/