SLES11 SP2 XEN VM install fails on a fc san multipath device

SUSE Linux Enterprise Server SLES Virtualization
1

Our environment:

  • two xen servers sles11 sp2 and HA-EXT
  • both attached to an ibm fibre channel ds4300 san storage with two controller (active/passive)
  • multipathing is configured
  • on both servers we have the same san luns:

mpatha_harley_sys (3600axxxxxxxxxxxxxxxxxxxxxxxabea4) dm-1 IBM,1722-600
size=30G features=‘1 queue_if_no_path’ hwhandler=‘1 rdac’ wp=rw
|-± policy=‘round-robin 0’ prio=6 status=active
| - 2:0:0:0 sdb 8:16 active ready running -± policy=‘round-robin 0’ prio=1 status=enabled
`- 2:0:1:0 sde 8:64 active ghost running

When I start the installation via the vm-install command the domu is going active and starts the installation on the san volume. After the first reboot during the installation the domu is reading the kernel and the initrd but the domu is not able to access the xvda device (the configured san volume).

What I found out:
After the first reboot during the installation of the vm, the partitioning of the multipath device comes viewable on the dom0 server, then the domu cannot access to this device any more.

When I remove the partitioning on the dom0 server with the following command:

kpartx -d -v -p “_part” /dev/mapper/mpath_harley_sys

I can continue the installation of the vm with a new reboot. When the new vm is active and I will call the partprobe command on the dom0 server the partitioning of the vm will not be viewable. When I shutdown the vm and call again the partprobe command on the dom0 server, the partitioning comes viewable but the vm will not start again.

Seems there is a device mapper problem.

Our VM configuration file:
name=“vm-harley”
description=“None”
uuid=“58c72121-bcb7-77a8-2584-0d5f251adea5”
memory=2048
maxmem=4096
vcpus=1
on_poweroff=“destroy”
on_reboot=“restart”
on_crash=“destroy”
localtime=0
keymap=“de”
builder=“linux”
bootloader="/usr/bin/pygrub"
boextra=""
disk=[ ‘phy:/dev/mapper/mpath_harley_sys,xvda,w’, ]
vif=[ ‘mac=00:16:3e:7a:9d:f2,bridge=br1’, ]
vfb=[‘type=vnc,vncunused=1’]otargs=""

This configuration runs well on SLES11 SP1
I have also tested the patched xen kernel form April 23 rd, same result.
I reported this also as a bug to novell, any ideas.

Rainer

2

You need to blacklist the xvd* devices in multipath.conf
If you don’t have one, check the example file /usr/share/doc/packages/multipath-tools/multipath.conf.annotated

Then, multipath won’t claim the xvd* devices for the Dom0.

edit: it may be, that it’s mdadm… check /usr/share/doc/packages/mdadm/mdadm.conf-example too.

3

Thank you for the response,

but the “physical” multipath device /dev/mapper/mpatha_harley_sys will mapped as device xvda to the domu. The xvda device is not visable at the dom0.
When I run this configuration on SLES11-SP1 everything works fine. There is no need for blacklisting. I also have to restart the multipathd at the dom0 to renew the blacklisting for multipathing.

The problem begins when the partitioning, which was made during the autoyast installation inside the domu, is viewable at the dom0. At this moment the domu cannot access to the mapped partitioning /dev/mapper/mpatha_harley_sys, /dev/mapper/mpatha_harley_sys_part1, …

It seems to me as a bug or a new device mapper feature. In SLES11 SP1 you can do parallel access to the a mapped fibre channel devices.

Rainer

[QUOTE=enovaklbank;4613]You need to blacklist the xvd* devices in multipath.conf
If you don’t have one, check the example file /usr/share/doc/packages/multipath-tools/multipath.conf.annotated

Then, multipath won’t claim the xvd* devices for the Dom0.

edit: it may be, that it’s mdadm… check /usr/share/doc/packages/mdadm/mdadm.conf-example too.[/QUOTE]

4

Hi Rainer,

Your setup is different to how I normally set this, so I stepping in blurry… My initial question reading your post: how are you initially hiding the domU device from dom0? Which configurations do you have in place to have this so?

-Willem

5

Not 100% sure, but “multipath -r” might be all that is needed instead of a full restart

[QUOTE=unixgrp;4660]The problem begins when the partitioning, which was made during the autoyast installation inside the domu, is viewable at the dom0. At this moment the domu cannot access to the mapped partitioning /dev/mapper/mpatha_harley_sys, /dev/mapper/mpatha_harley_sys_part1, …[QUOTE=unixgrp;4660]

Are you using LVM within the domU? Or just plain primary partitions?

-Willem

6

Hi Willem,
We do not hide anything. When the san device has no partition you can access the device from dom0 and also from the domu. Our problem begins with the partitioning inside the domu.

regards
Rainer

7

Yes it is.

8

[QUOTE=unixgrp;4700]Hi Willem,
We do not hide anything. When the san device has no partition you can access the device from dom0 and also from the domu. Our problem begins with the partitioning inside the domu.

regards
Rainer[/QUOTE]

Ok, thanks for clarifying… was not sure it you had some extended configurations going on there.

In any case, I also agree with Enovaklbank’s suggestion that multipath blacklisting might be required in this case. I understand this seems to be a new requirement in your environment, as you did not face this issue with SLES 11 SP1. I am however curious what SLES support has to say about this… so please also keep us in the loop, where you can, with the details of you SR.

The reason I’m asking about LVM is that I saw something similar happening with domU guest LVM VG’s coming through at dom0 level. Nothing to serious happening there as the storage was not getting claimed or dev mappings changed. I did however put a filter on the Xen host so these VG’s would not be picked up by dom0/Xen host.

-Willem

9

I will keep you informed.
What I really don’t understand is the blacklisting task from Enovaklbank. Why should I blacklist /dev/xvda at dom0. The Dom0 don’t have a xvda Device. Xvda is the device name inside the domu VM for the multipath device:

disk=[ ‘phy:/dev/mapper/mpath_harley_sys,xvda,w’, ]

Rainer

10

[QUOTE=unixgrp;4730]I will keep you informed.
What I really don’t understand is the blacklisting task from Enovaklbank. Why should I blacklist /dev/xvda at dom0. The Dom0 don’t have a xvda Device. Xvda is the device name inside the domu VM for the multipath device:

disk=[ ‘phy:/dev/mapper/mpath_harley_sys,xvda,w’, ]

Rainer[/QUOTE]

I’m not sure which bit either as I’ve not seen this specifically, my first hunch would be to look into blacklisting the partition bits (_part) that are seen after /dev/mapper/mpath_harley_sys. In any case I agree this should not be happening.

Hopefully SUSE support will find the cause quickly and possibly a patch is needed.

-Willem

11

Hi Rainer,

you are right about the xvd* devices. Honestly, I did not fully read and understand your reply just focused on the “there is no need for blacklisting” sentence. You do need blacklisting, on the block device which is given to the Dom0, and whose partitions are seen as xvd*, whichever that is. xvd* devices == partitions on the parent block device, which are found by mpath then. It was really misleading.

enovak

12

Problem solved !

After convincing the SuSE Support that we have a real problem, we got the solution from a multipathing developer via SuSE Support.
With SLES 11 SP2 the behavior of the multipathing an the partitioning at dom0 has changed.
We have to add an additional feature (no_partitions) configuration at /etc/multipath.conf for the IBM san storages ds4300 and also the IBM SVC storage.

This prevents multipathing to call kpartx and add the partitioning for the device at /dev/mapper/mpath_xxxx

You can configure it for a single lun, but I decided to change it for the complete device, so it will work for each additional lun we will add. You can have a look to the default configuration of the support devices with multipath -t. For IBM SAN products the default features line is:

features “1 queue_if_no_path”

you have to change this line to:

features “2 queue_if_no_path no_partitions”

I have added the complete device configuration for IBM SVC and IBM DS4300 to the /etc/multipath.conf file:

devices {
device {
vendor “IBM”
product “1722-600”
product_blacklist “Universal Xport”
path_grouping_policy “group_by_prio”
getuid_callout “/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n”
path_selector “round-robin 0”
path_checker “rdac”
features “2 queue_if_no_path no_partitions”
hardware_handler “1 rdac”
prio “rdac”
failback “immediate”
rr_weight “uniform”
no_path_retry 300
rr_min_io 1000
rr_min_io_rq 1
}

device {
	vendor "IBM"
	product "2145"
	path_grouping_policy "group_by_prio"
	getuid_callout "/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n"
	path_selector "round-robin 0"
	path_checker "tur"
	features "2 queue_if_no_path no_partitions"
	hardware_handler "0"
	prio "alua"
	failback "immediate"
	rr_weight "uniform"
	rr_min_io 1000
	rr_min_io_rq 1
}

}

Rainer

13

Hi Rainer,

Thanks for your detailed feed back! I’m sure this thread will also help others and I for one will be keeping an eye on this while updating our Xen sites to SLES 11 SP2.

The no_partitions option sounds interesting to me - particularly for Xen setups where the dom0 only has to pass trough a raw device to the domU’s and in that sense partitions on devices are not relevant on host level. Definitely something to look into & again thanks for sharing.

Cheers,
Willem

14

[QUOTE=unixgrp;4971]Problem solved !

After convincing the SuSE Support that we have a real problem, we got the solution from a multipathing developer via SuSE Support.
With SLES 11 SP2 the behavior of the multipathing an the partitioning at dom0 has changed.
We have to add an additional feature (no_partitions) configuration at /etc/multipath.conf for the IBM san storages ds4300 and also the IBM SVC storage.

This prevents multipathing to call kpartx and add the partitioning for the device at /dev/mapper/mpath_xxxx

You can configure it for a single lun, but I decided to change it for the complete device, so it will work for each additional lun we will add. You can have a look to the default configuration of the support devices with multipath -t. For IBM SAN products the default features line is:

features “1 queue_if_no_path”

you have to change this line to:

features “2 queue_if_no_path no_partitions”

I have added the complete device configuration for IBM SVC and IBM DS4300 to the /etc/multipath.conf file:

devices {
device {
vendor “IBM”
product “1722-600”
product_blacklist “Universal Xport”
path_grouping_policy “group_by_prio”
getuid_callout “/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n”
path_selector “round-robin 0”
path_checker “rdac”
features “2 queue_if_no_path no_partitions”
hardware_handler “1 rdac”
prio “rdac”
failback “immediate”
rr_weight “uniform”
no_path_retry 300
rr_min_io 1000
rr_min_io_rq 1
}

device {
	vendor "IBM"
	product "2145"
	path_grouping_policy "group_by_prio"
	getuid_callout "/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n"
	path_selector "round-robin 0"
	path_checker "tur"
	features "2 queue_if_no_path no_partitions"
	hardware_handler "0"
	prio "alua"
	failback "immediate"
	rr_weight "uniform"
	rr_min_io 1000
	rr_min_io_rq 1
}

}

Rainer[/QUOTE]

Hi Rainer,

Well… I can confirm the no_partitions option/feature is also needed on iSCSI environments in SLES 11 SP2.

It just bit me as well with a new Xen SLES 11 SP2 setup with the Equallogic HIT kit installed. Sneaky it is as I first did not notice the issue until I had rebooted the Xen hosts to push kernel updates through. Thought it was something related to the updates… but then I remembered this thread.

Difference in my case was that, as we use LVM volumes for the VM’s/domU’s OS disk, the VM would boot - but no secondary disks would be seen - even though xm reported the devices to have connected successfully. Quite hard to pinpoint, so thank you for posting your resolution and description of what was happening!

For now I had to revert to the regular multipath configuration (vs the multipath enhancement the Dell Equallogic HIT kit offers) as to get the multipath feature in.

As reference I’ll post the needed configuration change for the HIT kit once I have it.

Thanks,
Willem

15

[QUOTE=Magic31;5611]…
As reference I’ll post the needed configuration change for the HIT kit once I have it.
…[/QUOTE]

Still waiting on the parameter to use in combination with the Dell Eqaullogic HIT Kit.

In the meantime I’ve made a small script that removes the managed paths for partitions on the volumes. The script runs at Xen host boot and that is enough for now to keep things running.

For reference the script that provides a workaround for us:

#!/bin/bash
# temporary script to work around the Xen host multipathing claiming domU partitions
# For use with Equallogic HIT kit 1.1.0 and SLES 11.2 only
# WMeens - 16 7 2012

#List active dm paths and filter the output to write only partition paths on eql devices into a temporary file
dmsetup ls| grep eql-.*.*p[1-4][[:space:]] |cut -f 1 > /tmp/DMpartitions.dmp

#read the temporary file and remove the dm mappings for the partition paths listed
while read line; do dmsetup remove "$line"; done < /tmp/DMpartitions.dmp