Move SLES DomU from SLES10 SP4 host to SLES11 SP2 host-no go

kjhurni wrote:
[color=blue]

You cannot just do the:
xm new -f file

Because it’ll complain about the DomID (how do you determine the domid
of the Xen host)?[/color]

When you reboot a Xen host, the hypervisor starts Domain 0. Being the
first domain, it is assigned a domain ID of 0 (zero). It is the same
every time the host is rebooted. Each time a DomU is started, it is
assigned the next available sequential number.

The first DomU started will be assigned “1”. The second one started
will be assigned “2”. If the first one is stopped and restarted, it
will be assigned “3”. And so it goes.

Before re-importing a DomU configuration into the XenStore, I make sure
the original has been removed, after backing it up, of course. Make
sure to shutdown the DomU (MyDomU) first.

xm list -l MyDomU > MyDomu.py
xm delete MyDomU
xm new -f MyRevisedDomU.py

I have never has any issues with this procedure.


Kevin Boyle - Knowledge Partner
If you find this post helpful and are using the web interface,
show your appreciation and click on the star below…

[QUOTE=jmozdzen;9626]I think they’re talking about the “other” one - at least that’s what we’re using… and while the files are there until you delete them, you are right that they are not updated when you alter the definitions stored in the Xen config DB. Once you’re into clustering, you’ll want to only work with the files, as modifications to the Xen DB will only persist on a single Xen server, while you can share the files across many Xen Dom0s. But is should be easy to create up-to-date files, either manually or using virtmanager (just create a new VM with the proper settings and i.e. use the results as a template). Just keep in mind that you’ll have to actually DELETE the DomU definitions from the Xen store, else "xm create " wont have the desired effect and "xm start " will use the xen store definitions, rather than the config files.

Actually, they’re not different. SLES11 has simply dropped the Xen scripts to set up all the bridging environment - you do that in advance (outside the Xen configuration, ie via YaST) and then reference the bridge name you want the VIF to attach to, in the DomU config file. We did the same in SLES10 already, as we had other ideas of bridge names etc than the way Xen handled it :smiley:

I believe the DomU config file version will remain active for quite some time - it’s more difficult to run a Xen cluster with via a shared Xen store - AFAIK this isn’t available yet and IMO there’s no “business case” to implement this just to replace config files. But as boot loaders change and new features are available, the syntax and/or available commands within the config files will change over time… like when going from SLES10 to SLES11.

Indeed a valid approach - just make sure those changes are compatible with the SLES10 environment, else you’re fixed to running a SLES11 Dom0 :wink:

With regards,
Jens[/QUOTE]

Well actually the network bridging is different. I just attempted to import the “old” config (from the SLES10 machine) and the networking doesn’t even match up with how it is in SLES 11. Although that may have been due to the fact I was using bridging with custom script (the only way you really could do it in SLES 10).

Anyway, it seems the TID that Novell has is almost completely worthless. The first option is invalid as this isn’t a new install. The second option is vague and I can’t even get NTS to answer me back as to what it means.

Third option I cannot get to work because it won’t let me chroot /mnt or whatever (I get the error about /bin/bash not found or something). Never had that happen before, but then again I usually only do that via rescue installation.

4th option also doesn’t work if you ever made a change to the configuration. And I’m not sure if the command is listed properly. Novell has you running the xm with a -f instead of a -F. I believe the -f only works with the legacy file vs. the -F works with a file you exported via the xm list -l blah>blah command.

Now, I COULD get the 4th option to semi-work by using the -F command to import the exported VM. But I still got the error, but at that point it would let me modify the bootloader line.

Once I did that (paravirtualized DomU, BTW), it would boot, but then it refused to find the /dev/xvda3 for some reason (it IS there because it obviously boots fine on SLES10 Domu). Oh, and that’s also where I found that the bridging is totally different. It was trying to use “eth1” which isn’t even valid anymore, you have to pick paravirtualized - eth1 (br1) or whatever from the list.

So my whole opinion of the process is that you should give up hope on upgrading a SLES10 SP3 Dom0 to SLES11 SP2 and just start the entire thing from scratch and manually reinstall all your DomU.

Apparently nobody has ever bothered to actually TEST taking a DomU from SLES10 sP3 and trying to get it to run under SLES11 SP2 Dom0. At least the authors of the TID didn’t try it because none of their stuff works.

I may be daring and snap the LUNs and see what happens on an actual upgrade of the host OS (Dom0), but so far it seems one cannot easily take a DomU created on SLES10 SP3 and run it on SLES11 SP2 Dom0.

KBOYLE wrote:
[color=blue]

xm list -l MyDomU > MyDomu.py xm delete MyDomU xm new -f MyRevisedDomU.py [/color]

Ooops! Should have used a capital F.

xm list -l MyDomU > MyDomu.py
xm delete MyDomU
xm new -F MyRevisedDomU.py


Kevin Boyle - Knowledge Partner
If you find this post helpful and are using the web interface,
show your appreciation and click on the star below…

[QUOTE=KBOYLE;9727]KBOYLE wrote:
[color=blue]

xm list -l MyDomU > MyDomu.py xm delete MyDomU xm new -f MyRevisedDomU.py [/color]

Ooops! Should have used a capital F.

xm list -l MyDomU > MyDomu.py
xm delete MyDomU
xm new -F MyRevisedDomU.py


Kevin Boyle - Knowledge Partner
If you find this post helpful and are using the web interface,
show your appreciation and click on the star below…[/QUOTE]

Yes, that’s where I messed up as well. The TID said to use “-f” not “-F”
However, apparently they meant the legacy config files in the /etc/xen directory only work with the -f
But those don’t work for me since they’re like 2 years old and we’ve changed things since then, so I have no choice apparently in this whole mess.

I did ask NTS to ask about IMPORTING a VM from SLES10 into SLES11, and they’re checking.

Otherwise it looks like I will either have to:

  1. Upgrade the XEN Host (something I’m loathe to do)
    or
  2. Reinstall the VM (DomU) on SLES11 from scratch. In THIS case, it’s NAM stuff, so it’s not QUITE that bad for the IDP and LAG/MAG.
    But it would be a PITA for the Admin Console.

Makes me want to use Vmware.

kjhurni wrote:
[color=blue]

The boot fails in that you cannot even start the virtual machine via
virt manager.[/color]

I came across this TID which may help.

TID 7001376 How to manually install GRUB after a failed installation or
disk corruption of the MBR.
http://www.novell.com/support/kb/doc.php?id=7001376

If TID 7002815 suggests your issue can occur when grub is not installed
on the DomU and TID 7001376 says there are instances when grub was not
installed properly, then maybe…


Kevin Boyle - Knowledge Partner
If you find this post helpful and are using the web interface,
show your appreciation and click on the star below…

[QUOTE=KBOYLE;9788]kjhurni wrote:
[color=blue]

The boot fails in that you cannot even start the virtual machine via
virt manager.[/color]

I came across this TID which may help.

TID 7001376 How to manually install GRUB after a failed installation or
disk corruption of the MBR.
http://www.novell.com/support/kb/doc.php?id=7001376

If TID 7002815 suggests your issue can occur when grub is not installed
on the DomU and TID 7001376 says there are instances when grub was not
installed properly, then maybe…


Kevin Boyle - Knowledge Partner
If you find this post helpful and are using the web interface,
show your appreciation and click on the star below…[/QUOTE]

Thanks for the info, but unfortunately I am not using LVM or EVMS, so the TID doesn’t really help much.

I’m ASSUMING grub is installed because I can go into Yast → bootloader and it shows grub being used. However, with paravirtualized DomU I vaguely remember there’s some odd/funkiness with the boot stuff as to where it’s really at (the Dom0 or the DomU)?

Hi kjhurni,

However, with paravirtualized DomU I vaguely remember there’s some odd/funkiness with the boot stuff as to where it’s really at (the Dom0 or the DomU)?

as far as our Xen setup is concerned (SLES10 and SLES11SP1), booting a DomU involves the following steps:

  • mount the DomU’s virtual disk in Dom0 for bootup

  • determine initrd and kernel (see comment below)

  • copy kernel and intrd to Dom0 disk space

  • unmount DomU’s virtual disk in Dom0

  • mount DomU’s virtual disk(s) for normal operation

  • start VM using Dom0’s copy of DomU’s initrd & kernel

  • delete Dom0’s copy of DomU’s initrd & kernel

Determining the initrd and kernel file is the part done by the different boot loader settings. It seems that “pygrub” fetches it’s information from DomU’s menu.lst, that why GRUB “needs to be installed” or at least a proper menu.lst is required. Of course you can also set up direct pointers to these files, even up to files stored locally on Dom0 (i.e. using bootloader_args and a supporting bootloader like “domUloader.py”).

So in order to boot your DomU, three things have to work:

  • access to the DomU kernel/initrd prior to DomU startup (meaning “Dom0 can access DomU’s disk to copy those files”)
  • working DomU kernel/initrd (DomU kernel must work with Xen version of Dom0)
  • proper setup (boot loader must be able to determine which DomU kernel/initrd to copy)

“boot loader didn’t return any data” usually means that it couldn’t locate/copy kernel/initrd.

If you can identify the kernel and initrd file you’d like to use, maybe using domUloader will get the VMs up&running?

I’m unsure what the current problem state is: What boot loader (and it’s configuration) is failing under SLES11? What’s in xend.log after (failed) DomU start?

Regards,
Jens

[QUOTE=jmozdzen;9809]Hi kjhurni,

However, with paravirtualized DomU I vaguely remember there’s some odd/funkiness with the boot stuff as to where it’s really at (the Dom0 or the DomU)?

as far as our Xen setup is concerned (SLES10 and SLES11SP1), booting a DomU involves the following steps:

  • mount the DomU’s virtual disk in Dom0 for bootup

  • determine initrd and kernel (see comment below)

  • copy kernel and intrd to Dom0 disk space

  • unmount DomU’s virtual disk in Dom0

  • mount DomU’s virtual disk(s) for normal operation

  • start VM using Dom0’s copy of DomU’s initrd & kernel

  • delete Dom0’s copy of DomU’s initrd & kernel

Determining the initrd and kernel file is the part done by the different boot loader settings. It seems that “pygrub” fetches it’s information from DomU’s menu.lst, that why GRUB “needs to be installed” or at least a proper menu.lst is required. Of course you can also set up direct pointers to these files, even up to files stored locally on Dom0 (i.e. using bootloader_args and a supporting bootloader like “domUloader.py”).

So in order to boot your DomU, three things have to work:

  • access to the DomU kernel/initrd prior to DomU startup (meaning “Dom0 can access DomU’s disk to copy those files”)
  • working DomU kernel/initrd (DomU kernel must work with Xen version of Dom0)
  • proper setup (boot loader must be able to determine which DomU kernel/initrd to copy)

“boot loader didn’t return any data” usually means that it couldn’t locate/copy kernel/initrd.

If you can identify the kernel and initrd file you’d like to use, maybe using domUloader will get the VMs up&running?

I’m unsure what the current problem state is: What boot loader (and it’s configuration) is failing under SLES11? What’s in xend.log after (failed) DomU start?

Regards,
Jens[/QUOTE]

Thanks Jens

The original issue was that I could not import a DomU created in SLES10 SP3 on a SLES11 SP2 Xen Host. I would get the original error in my original post (boot failed, blah blah).

I opened an SR and that’s where they pointed me to the TID that indicates due to kernel changes, that you cannot do that. The TID had some suggestions, most of which either assume that you’ve never ever changed the original VM (ie, your legacy files are identical to your currently exported ones–which in my case they are NOT), or you can manually create/adjust the files.

I ended up basically creating a new DomU that was “empty” (I just needed the machine definition in the Virt Manager), and then edited it to point to the "sles10 DomU’ disk, along with adjusted the bootloader args per the TID.

At that point, the paravirtualized SLES10-created DomU will actually boot, but fails to find the / partition and all you get is the lovely $ prompt.

The FULLY virtualized machine created in SLES10 will actually “import” (for lack of a better word) into SLES11 Xen Host, but it suffers the same fate. You bootup, see the Grub menu, it starts to boot and then cannot find the / partition.

Originally I thought it was due to the MPIO as someone posted that issue here, so I rebuilt the SLES11 XEN host without MPIO and the same thing occurs.

I know normally (on a physical machine) the “cannot find /dev/blah /” errors are usually due to bad menu.lst or /etc/fstab or something, and normally I’d boot into the rescue system, but of course, doing THAT on paravirtualized XEN is a royal nightmare (I tried the old TID on that but the “chroot /mnt” command gives me errors about not finding /bin/bash)

It seems so far, that the only info Novell has on the whole situation is to Upgrade the XEN host, as moving/running the DomU around isn’t supported UNLESS you’re lucky enough to have the legacy files match your current state of your DomU in which case supposedly it would work.

Hi kjhurni,

thanks for the summary - I was briefly following this thread over the days, but somehow got lost :[

At that point, the paravirtualized SLES10-created DomU will actually boot, but fails to find the / partition and all you get is the lovely $ prompt.

You might have already checked: Is the device name of your virtual disk (as the DomU will see it) the same as before the “upgrade”? Looking at the new configuration should give the new name, even if the DomU doesn’t start:
like “disk=[ ‘some_Dom0_disk_description_,xvda,w’, ]” in the configuration file. If the disk’s name (“xvda” in this example) doesn’t match, then most likely the DomU kernel loader (or rather it’s initrd script) will not be able to find the device and you might want to add to your config something like
extra=“root=/dev/xvda3”
or however the root file system is to be identified inside the DomU.

Regards,
Jens

kjhurni wrote:
[color=blue]

At that point, the paravirtualized SLES10-created DomU will actually
boot, but fails to find the / partition and all you get is the lovely
$ prompt.[/color]

Have you tried to run mkinitrd at this point? if the DomU is frozen,
perhaps you can try the technique described in:
[color=blue]

TID 7001376 How to manually install GRUB after a failed installation
or disk corruption of the MBR.
http://www.novell.com/support/kb/doc.php?id=7001376[/color]


Kevin Boyle - Knowledge Partner
If you find this post helpful and are using the web interface,
show your appreciation and click on the star below…

[QUOTE=jmozdzen;9812]Hi kjhurni,

thanks for the summary - I was briefly following this thread over the days, but somehow got lost :[

At that point, the paravirtualized SLES10-created DomU will actually boot, but fails to find the / partition and all you get is the lovely $ prompt.

You might have already checked: Is the device name of your virtual disk (as the DomU will see it) the same as before the “upgrade”? Looking at the new configuration should give the new name, even if the DomU doesn’t start:
like “disk=[ ‘some_Dom0_disk_description_,xvda,w’, ]” in the configuration file. If the disk’s name (“xvda” in this example) doesn’t match, then most likely the DomU kernel loader (or rather it’s initrd script) will not be able to find the device and you might want to add to your config something like
extra=“root=/dev/xvda3”
or however the root file system is to be identified inside the DomU.

Regards,
Jens[/QUOTE]

I’m pretty sure I double-checked. The physical disk is correct, (/dev/disk/by-id/scsi-GUIDTHINGY-part3)
I’ll have to look inside, but I’m 99.9% sure it’s still:
/dev/xvda3

[QUOTE=kjhurni;9819]I’m pretty sure I double-checked. The physical disk is correct, (/dev/disk/by-id/scsi-GUIDTHINGY-part3)
I’ll have to look inside, but I’m 99.9% sure it’s still:
/dev/xvda3[/QUOTE]

Okay the DomU is most certainly using grub (if the Yast → Bootloader is correct).
The “edit configuration files” seems to indicate that the menu.lst for grub is set to
root (hd0,0)
kernel /boot/vmlinuz-2.6.16.60-0.832.2-xen

ramdisk:
/boot/initrd-2.6.16.60-0.83.2-xen

The SLES10 DomU config file has the bootloader args as:
(bootloader_args ‘–entry=xvda1:/boot/vmlinuz-xen,/boot/initrd-xen’)

whereas, the SLES11 DomU would have something like:
(bootloader_args -q)

whereas, the SLES11 DomU would have something like:
(bootloader_args -q)

have you tried setting

(bootloader_args ‘–entry=xvda1:/boot/vmlinuz-2.6.16.60-0.832.2-xen
,/boot/initrd-2.6.16.60-0.83.2-xen’)

(using the appropriate syntax for your file, of course)

Regards,
Jens