I am trying to get my head around this one using OES servers with NCS. I
have done this using iscsi which is easy. I am not quite sure how you do
this with FC. The layout would be 3 hosts connected to the fc switch and a
fc san connected to the switch. I will be running the vm’s off of a portion
of the san and they will be setup with HA between the servers with ocsfs for
managing which nodes they will be run on. My confusion comes from how to get
these vm’s than to see the clustered nss volumes on the san. I have tried to
speculate on this but so far have not gotten an understanding. These will be
running on Xen hosted on Sles 11 sp2.
[QUOTE=Rick B;4879]I am trying to get my head around this one using OES servers with NCS. I
have done this using iscsi which is easy. I am not quite sure how you do
this with FC. The layout would be 3 hosts connected to the fc switch and a
fc san connected to the switch. I will be running the vm’s off of a portion
of the san and they will be setup with HA between the servers with ocsfs for
managing which nodes they will be run on. My confusion comes from how to get
these vm’s than to see the clustered nss volumes on the san. I have tried to
speculate on this but so far have not gotten an understanding. These will be
running on Xen hosted on Sles 11 sp2.
Thanks
Rick[/QUOTE]
Hi Rick,
I’m not sure what you are looking for. If you have setup with 3 Xen hosts that all have a simultaneous connection to the SAN stores, the OES vm’s can also access devices/partitions/disks. Adding an exclamation (!) to the shared disks defined in the vm configuration files followed by deleting and reimporting the vm definitions into Xen with the ‘xm new [vm name]’ command, should allow the vm’s to access the shared disks simultaneously.
I have not run OES NCS domU’s on Xen before, but have done so successfully with NetWare based clusters a couple of times in the past. (for NetWare vm’s it involved also setting the “set XenBlk Barriers=0” on each NetWare domU, I don’t think such an equivalent is needed for OES domU’s).
Some more info on what you are finding to be an issue might give a better response.
I am trying to get my head around this one using OES servers with NCS. I
have done this using iscsi which is easy. I am not quite sure how you
do
this with FC. The layout would be 3 hosts connected to the fc switch
and a
fc san connected to the switch. I will be running the vm’s off of a
portion
of the san and they will be setup with HA between the servers with
ocsfs for
managing which nodes they will be run on. My confusion comes from how
to get
these vm’s than to see the clustered nss volumes on the san. I have
tried to
speculate on this but so far have not gotten an understanding. These
will be
running on Xen hosted on Sles 11 sp2.
Thanks
Rick[/color]
Hi Rick,
I’m not sure what you are looking for. If you have setup with 3 Xen
hosts that all have a simultaneous connection to the SAN stores, the OES
vm’s can also access devices/partitions/disks. Adding an exclamation (!)
to the shared disks defined in the vm configuration files followed by
deleting and reimporting the vm definitions into Xen with the ‘xm new
[vm name]’ command, should allow the vm’s to access the shared disks
simultaneously.
I have not run OES NCS domU’s on Xen before, but have done so
successfully with NetWare based clusters a couple of times in the past.
(for NetWare vm’s it involved also setting the “set XenBlk Barriers=0”
on each NetWare domU, I don’t think such an equivalent is needed for OES
domU’s).
Some more info on what you are finding to be an issue might give a
better response.
Cheers,
Willem
[/color]
Sorry for not following up sooner but just very swamped. I am in the middle
of this now trying to get virtualized servers using npiv. Does anyone know
what the correct settings to put in the disks field. I have tried using the
setting npiv:wwn,lun and can’t connect. Is there somewhere that explains
how to do this that is understandable. There is not much docs on this. I am
using a qlogic 4g fc switch and qlogic hba’s. I have a Fujitsu DX60 san.
[QUOTE=Rick B;5174][…] I am in the middle
of this now trying to get virtualized servers using npiv. Does anyone know
what the correct settings to put in the disks field. I have tried using the
setting npiv:wwn,lun and can’t connect. Is there somewhere that explains
how to do this that is understandable.[…]
Thanks[/QUOTE]
Concerning WWNN and WWPN, I once looked up the value for “locally defined numbers” and decided that 29001100aaXXXXXX (with XXXXXX being any six-digit hex number unique to your fabric) is a good range for WWPNs and 29011100aaXXXXXX the one for WWNNs. (I use NPIV WWPN and WWNN in pairs, so those are unique per DomU.)
case “$command” in
add)
# Params is one big arg, with fields separated by hyphens:
# FABRIC-VPWWPN-VPWWNN-TGTWWPN-LUN#
# arg 2 - Fabric Name
# arg 3 - VPORT’s WWPN
# arg 4 - VPORT’s WWNN
# arg 5 - Target’s WWPN
# arg 6 - LUN # on Target
# no wwn contains a leading 0x - it is a 16 character hex value
# You may want to optionally pick a specific adapter ?[/CODE]
A word of warning: From bad experience I suggest to take extra precautions so that only a single node will write to a specific LUN, for instance by setting up DomU locking within xend… having the DomU started on more than one node will get you into trouble for sure…
And another experience: When using the above syntax, the LUN will be attached to Dom0 and handed to the DomU like any other local Dom0 resource. I recommend to configure LVM on Dom0 to only use the devices that actually carry Dom0-PVs (that are meant for the Dom0) by specifying the appropriate filters in /etc/lvm/lvm.conf. Additionally, there were/are some minor flaws when using LVM inside the DomUs, independant from LVM filter settings: If you use the same VG name on Dom0 and DomU and you have a LV on the DomU that is not present on the Dom0 VG, then the DomU boot process will hang - this persists until you reboot the Dom0 and affects any subsequent DomU start. Changing /lib/udev/activate_vg to not change the VG to active if the new VG has the same name as Dom0 VGs is the fix.
With regards,
Jens
PS: If you find a way to specify an NPIV adapter for forwarding to the DomU (AKA pvSCSI), let me know. It should be possible manually, but I’ve found no /etc/xen/vm/*-support for this so far. Using pvSCSI would avoid seeing the NPIV adapter in Dom0 at all, keeping it limited to its intended use: inside the DomU…
Good points Jens (locking is certainly one to watch for, especially when using cluster unaware filesystems like LVM2 vs cLVM)!
Also the LVM heads up is a good one to mention, and also added that a best practice is to not use LVM configured storage for partitions the dom0/host is using - configure storagre/partitioning for the Xen host/dom0 itself with non LVM “traditional primary partitions” and use LVM type partitions (or direct phy access to partitions) for the domU/guests.
The tip on /lig/udev/activate_vg has gone into my little book of notes!
When you use an NPIV “adapter” per DomU, use of cluster FSs usually doesn’t come to mind: It’s like assigning a dedicated disk to the single DomU.
But I have to admit that my advice might sound illogical - even when activating the same WWPN on another Dom0 (by starting the DomU there), your fabric should route packets to only a single NPIV HBA (the one with the latest login). It’s just that I had to deal with a bug in SCST that resulted in double-assignments of LUNs to seemingly two different WWPNs, although configured only for a one-on-one assignment. We’ve submitted a fix, which is in the trunk code of SCST, but I doubt that many vendors have switched to using “trunk” rather than one of the “stable” SCST releases - which do not have that fix applied. Details are available here.
I’m too much a fan of LVM to follow this advice - but I believe to be one of the “I know what I’m doing, wanted to do it and can handle the consequences” variety Had it not been for the activate_vg bug, the filters in Dom0’s lvm.conf were sufficient for me to stay out of trouble.
[QUOTE=Magic31;5207]The tip on /lig/udev/activate_vg has gone into my little book of notes!
Cheers,
Willem[/QUOTE]
Then I better provide my version of activate_vg here (since the message from 2010 on the Novell forum was another victim of the Novell/Suse forum split):
echo “$now: not acting on $0 $1 $2, VG $2 unwanted with system VG $SYSTEMVG” >> /var/log/lvm2.log
exit 0
if [ “$SYSTEMVG” = “$VGNAME” ] ; then
echo “$now: not acting on $0 $1 $2, VG $2 conflicts with system VG $SYSTEMVG” >> /var/log/lvm2.log
else
echo “$now: SYSTEMVG=$SYSTEMVG” >> /var/log/lvm2.log
echo "$now: " /sbin/lvm vgchange --config “devices {filter = [ "a|/dev/disk/by-id/lvm2-vgpv-$VGUUID-.|", "r/./" ]}” -a y $VGNAME >> /var/log/lvm2.log
/sbin/lvm vgchange --config “devices {filter = [ "a|/dev/disk/by-id/lvm2-vgpv-$VGUUID-.|", "r/./" ]}” -a y $VGNAME
fi[/CODE]
I failed to reproduce this problem end of April (with then latest patches) - but today I’ve seen new log entries from May and June, so this might still be a problem. I’ll have to re-test this when I find some time…
Good points Jens (locking is certainly one to watch for, especially when
using cluster unaware filesystems like LVM2 vs cLVM)![/color]
When you use an NPIV “adapter” per DomU, use of cluster FSs usually
doesn’t come to mind: It’s like assigning a dedicated disk to the single
DomU.
But I have to admit that my advice might sound illogical - even when
activating the same WWPN on another Dom0 (by starting the DomU there),
your fabric should route packets to only a single NPIV HBA (the one with
the latest login). It’s just that I had to deal with a bug in SCST that
resulted in double-assignments of LUNs to seemingly two different WWPNs,
although configured only for a one-on-one assignment. We’ve submitted a
fix, which is in the trunk code of SCST, but I doubt that many vendors
have switched to using “trunk” rather than one of the “stable” SCST
releases - which do not have that fix applied. Details are available
‘here’
(http://technik.blogs.nde.ag/2012/02/10/dss-npiv-and-scst-pitfalls-dug-[/color]
deep/).[color=blue]
Magic31;5207 Wrote:[color=green]
Also the LVM heads up is a good one to mention, and also added that a
best practice is to not use LVM configured storage for partitions the
dom0/host is using - configure storagre/partitioning for the Xen
host/dom0 itself with non LVM “traditional primary partitions” and use
LVM type partitions (or direct phy access to partitions) for the
domU/guests.[/color]
I’m too much a fan of LVM to follow this advice - but I believe to be
one of the “I know what I’m doing, wanted to do it and can handle the
consequences” variety Had it not been for the activate_vg bug, the
filters in Dom0’s lvm.conf were sufficient for me to stay out of
trouble.
Magic31;5207 Wrote:[color=green]
The tip on /lig/udev/activate_vg has gone into my little book of notes!
Cheers,
Willem[/color]
Then I better provide my version of activate_vg here (since the message
from 2010 on the Novell forum was another victim of the Novell/Suse
forum split):
echo “$now: not acting on $0 $1 $2, VG $2 unwanted with system VG
$SYSTEMVG” >> /var/log/lvm2.log exit 0
if [ “$SYSTEMVG” = “$VGNAME” ] ; then
echo “$now: not acting on $0 $1 $2, VG $2 conflicts with system VG
$SYSTEMVG” >> /var/log/lvm2.log else
echo “$now: SYSTEMVG=$SYSTEMVG” >> /var/log/lvm2.log
echo "$now: " /sbin/lvm vgchange --config “devices {filter = [
"a|/dev/disk/by-id/lvm2-vgpv-$VGUUID-.|", "r/./" ]}” -a y $VGNAME[color=green][color=darkred]
/var/log/lvm2.log /sbin/lvm vgchange --config “devices {filter = [[/color][/color]
"a|/dev/disk/by-id/lvm2-vgpv-$VGUUID-.|", "r/./" ]}” -a y $VGNAME
fi
I failed to reproduce this problem end of April (with then latest
patches) - but today I’ve seen new log entries from May and June, so
this might still be a problem. I’ll have to re-test this when I find
some time…
YMMV
With regards,
Jens
[/color]
Thanks Jens And Willem. I have this now running with 2 vm’s so far. They are
booth oes11. I will be doing clustering at this level from the domu’s. Can I
assign all the cluster devices using the same format to all vm’s that are in
the cluster? I would assume since it is a device it would show up and than
be able to create pools. Also the sbd can be addressed the same way?
I know there are so may ways to skin the cat these days.
[QUOTE=Rick B;5213]Thanks Jens And Willem. I have this now running with 2 vm’s so far. They are
booth oes11. I will be doing clustering at this level from the domu’s. Can I
assign all the cluster devices using the same format to all vm’s that are in
the cluster? I would assume since it is a device it would show up and than
be able to create pools. Also the sbd can be addressed the same way?
I know there are so may ways to skin the cat these days.
Thanks again.[/QUOTE]
you’d create NPIV HBAs per DomU (so with two DomUs, you have two different NPIV addresses) and configure your SAN to give both vHBAs access to the share resource(s).
From that point on, it should be the same as with any other type of shared disk (both VMs have concurrent access to the same disk space).
As I have no experience with OES, I cannot answer the “pool” part of your question.
AFAIK, sbd fencing can be done on any shared device, ie a separate partition on a single “shared disk”. But as we use power switches for fencing, I only know the theory.
Thanks Jens And Willem. I have this now running with 2 vm’s so far. They
are
booth oes11. I will be doing clustering at this level from the domu’s.
Can I
assign all the cluster devices using the same format to all vm’s that
are in
the cluster? I would assume since it is a device it would show up and
than
be able to create pools. Also the sbd can be addressed the same way?
I know there are so may ways to skin the cat these days.
Thanks again.[/color]
you’d create NPIV HBAs per DomU (so with two DomUs, you have two
different NPIV addresses) and configure your SAN to give both vHBAs
access to the share resource(s).
From that point on, it should be the same as with any other type of
shared disk (both VMs have concurrent access to the same disk space).
As I have no experience with OES, I cannot answer the “pool” part of
your question.
AFAIK, sbd fencing can be done on any shared device, ie a separate
partition on a single “shared disk”. But as we use power switches for
fencing, I only know the theory.
Regards,
Jens
[/color]
I don’t know whether I said this earlier. I am on Sles 11 sp2. Is there an
easier way to add additional disks to vm’s rather than having to edit files
each time and than deleting and creating new all the domu’s?
Thanks Jens And Willem. I have this now running with 2 vm’s so far. They
are
booth oes11. I will be doing clustering at this level from the domu’s.
Can I
assign all the cluster devices using the same format to all vm’s that
are in
the cluster? I would assume since it is a device it would show up and
than
be able to create pools. Also the sbd can be addressed the same way?
I know there are so may ways to skin the cat these days.
Thanks again.[/color]
you’d create NPIV HBAs per DomU (so with two DomUs, you have two
different NPIV addresses) and configure your SAN to give both vHBAs
access to the share resource(s).
From that point on, it should be the same as with any other type of
shared disk (both VMs have concurrent access to the same disk space).
As I have no experience with OES, I cannot answer the “pool” part of
your question.
AFAIK, sbd fencing can be done on any shared device, ie a separate
partition on a single “shared disk”. But as we use power switches for
fencing, I only know the theory.
Regards,
Jens
[/color]
I don’t know whether I said this earlier. I am on Sles 11 sp2. Is there an
easier way to add additional disks to vm’s rather than having to edit
files each time and than deleting and creating new all the domu’s?[/color]
Think I figured it out. Looks like the block attach updates the config as
well.
[QUOTE=Rick B;5217]I don’t know whether I said this earlier. I am on Sles 11 sp2. Is there an
easier way to add additional disks to vm’s rather than having to edit files
each time and than deleting and creating new all the domu’s?[/QUOTE]
well, I definitely prefer to maintain control (and to generate config files - our company is specialized in highly automated systems management) - but yes, there are visual tools to edit Xen VM configurations (I assume that it’s still called “virt-manager”).
OTOH, what good is a visual tool that let’s you make choices with your mouse, if there’s no proper entry in the list? Nothing on your Dom0 knows about the NPIV address you’re going to use for the DomU (or rather, to access it’s disk) since there’s no corresponding adapter present on the system. And I believe there’s no tool yet that’ll “synthesize” the NPIV disk string… that’s still all manual: get the fabric name from the corresponding device entry, come up with proper WWPN and WWNN for the vHBA, determine the WWPN of the SAN server offering the LUN, ask the admin for the LUN number(s)… and enter that as the “disk id”, properly formatted.
I say: Go ahead and create the first config file via some tool, then simply copy&edit it (name, uuid, NPIV numbers, MAC address, maybe VNC display number - and use some numbering scheme to simplify things)… just my 2 cents.
Regards,
Jens
PS: If you use some tool, then keep in mind that ie virt-manager registers the DomUs with the local xend - when running things in a Xen cluster, that might turn out to be a problem and so you need to remember to delete the DomU (but not its config file under /etc/xen/vm) after using virt-manager… If “xm list” lists your DomU although it’s not started, you might run into problems later (“xm start” vs. “xm create”).
[QUOTE=Rick B;4879]The layout would be 3 hosts connected to the fc switch and a
fc san connected to the switch. I will be running the vm’s off of a portion
of the san and they will be setup with HA between the servers with ocsfs for
managing which nodes they will be run on.[/QUOTE]
[QUOTE=Rick B;5218]Think I figured it out. Looks like the block attach updates the config as
well.
Thanks[/QUOTE]
you are you running a Xen cluster with clustered OES DomUs, aren’t you? Then from my experience, you should keep your hands off of persisted DomU configurations.
Background: In a Xen cluster, you keep the configuration file-based, and share those between the Dom0s (via a shared /etc/xen/vm FS). No matter where you start the DomU (via “xm create”, i.e. via a pacemaker agent), it will always have the proper configuration.
If you’d use persistent configurations, you’d have to update those on all xends consistently (and use “xm start” to kick off the DomUs), which is too error-prone.
As most of our setups are iSCSI based these days (SLES 11 with Equallogic SAN’s & the Linux HIT kits work like a champ)… NPIV is not something I can “play” with… at least, I always thought this was a fiber only thing with SLES Xen.
To elaborate a bit more, I do use LVM allot within the Xen setups - too mcuh value there indeed: all the OS/systems disks of the domU’s are running on/with an assigned LVM volume. Usually I create two or the volume groups on the host (which are placed on SAN storage), say VGOS1, VGOS2 and VGOS3. Each of these volume groups are +/- 600 GB in size. Spread throughout them are the OS disks of the domU’s.
This way we can utilize LVM snapshots on the Xen host, to capture and dump the domU’s volume and stream it into a zipped tar file. Great for DR scenarios.
The LVM filter on the Xen host is setup as to only look/use devices found under /dev/mapper/vgos*. All other LVM volumes are ignored. And indeed, VG names on the Xen hosts are always different to the VG named in the domU’s. Have not hit any issues with this setup.
On a side note (drifting): One thing I still want/need to dig into are the options lvconvert can add to the LVM snapshot feature (easy rollback). Hope to be able to get that sorted and implemented (is stable/robust enough) end of this summer.
Added to that, also looking into adding the partclone tool to the mix to aid in faster storage migration options, when needed… but that’s drifting far off topic Do mean to fully agree that having LVM in the mix definitely has added value.
What I meant with not using LVM storage for dom0 = All the mountpoints that are used for dom0(e.g. the Xen host / filesystem and related directories) are on non LVM storage.
[QUOTE=jmozdzen;5210]Then I better provide my version of activate_vg here (since the message from 2010 on the Novell forum was another victim of the Novell/Suse forum split):
echo “$now: not acting on $0 $1 $2, VG $2 unwanted with system VG $SYSTEMVG” >> /var/log/lvm2.log
exit 0
if [ “$SYSTEMVG” = “$VGNAME” ] ; then
echo “$now: not acting on $0 $1 $2, VG $2 conflicts with system VG $SYSTEMVG” >> /var/log/lvm2.log
else
echo “$now: SYSTEMVG=$SYSTEMVG” >> /var/log/lvm2.log
echo "$now: " /sbin/lvm vgchange --config “devices {filter = [ "a|/dev/disk/by-id/lvm2-vgpv-$VGUUID-.|", "r/./" ]}” -a y $VGNAME >> /var/log/lvm2.log
/sbin/lvm vgchange --config “devices {filter = [ "a|/dev/disk/by-id/lvm2-vgpv-$VGUUID-.|", "r/./" ]}” -a y $VGNAME
fi[/CODE]
I failed to reproduce this problem end of April (with then latest patches) - but today I’ve seen new log entries from May and June, so this might still be a problem. I’ll have to re-test this when I find some time…
[/QUOTE]
Thanks Jens! You sure are one of the Xen masters …or is that Zen master (in Japanese sense)?
Had it not been for the activate_vg bug, the filters in Dom0’s lvm.conf were sufficient for me to stay out of trouble.