We are running a demo POC of SUSE XEN. my customer is running two virtual machines. They installed SAP on both virtual machines.
Customer found(complain) that both virtual machine are performing slow comparatively(they have done a similar setup/test using ESX 5 with 3x better results).
Though I haven’t got the chance to look into the logs, or monitor(top, vmstat) the server yet, but would appreciate if you guys can please share something that can help me understand the problem or find the reason behind the issue
Our Environment:
XEN Host OS: SLES 11 SP2 x864
HW: HP Proliant DL 380 G6
CPU: two Physical Quad core processors.
Memory: 32 GB
On XEN host server we can see 16 processors when we ran “cat /proc/cpuinfo”
we created two SLES 11 SP1 virtual machines ‘vbox1’ and ‘vbox2’
resources assigned to vbox1
CPU: 6
Mem: 16 GB
HDD: Two disks(each 1 TB)
OS: SLES 11 SP1
resources assigned to vbox2
CPU: 6
Mem: 10 GB
HDD: 1 TB single disk
OS: SLES 11 SP1
Note: All the disks are SATA and attached locally, no SAN or multipathing.
[QUOTE=ab;4993]
Paravirtualized or fully virtualized? [/QUOTE]
as the virtual machines are SLES 11 SP1 therefore I am sure that both virtual machines are Paravirtualized.
[QUOTE=ab;4993]
How did they determine the performance of the two systems (exactly… details please)?[/QUOTE]
As per the customer(SAP Guys), whatever they do in SAP e.g taking backup via SAP native tool, uploading data in their SAP system, performing installation of SAP modules etc… i.e in short
their day-to-day SAP related tasks are slower(takes more time to complete) when comparing with ESX.
On same Hard Ware, this customer has created two SLES 11 SP1 virtual machines atop ESX, and with the same SAP modules/setup, and as per them they found that the current(XEN based) is at least 3 time slower(takes more time to complete) then ESX.
One more question here, do you recommend me to mount the /var/lib/xen/images file system with noatime and barrier=0 options ?
With such VM’s and size of the disks I’d opt to give each SLES guest a physical disk rather than file based.
With the setup you are describing this must be done one the local systems disks, so considering limitations of MBR with traditional partitioning: I’d say go with disk assignments based on LVM. Size the LVM volumes appropriately for each SLES guest system and assign the volume as physical connection to the SLES guest servers.
If the used hw raid controller has the option to split the storage up in correctly sized volumes/devices, that would be even better as you can directly assign these as devices without needing to use LVM to manage the guest storage on the host itself.
Hope that is somewhat clear… and in any case it should give the best performance (phy: over file based disks).
One more thing… if this is a POC, why SLES 11 SP1 and not SLES 11 SP2 (which has the newer kernel and Xen version)?
Also, have they installed any specific HP controller software? Could be the used storage driver is not optimized or the controller itself does not have a decent cache controller included?
[QUOTE=Magic31;4998]
With the setup you are describing this must be done one the local systems disks.[/QUOTE]
Yes all the disks are local.
If I got you correctly then you are saying me to create two LV on XEN Host and then assign those LVes to virtual machines, e.g
LV Name: /dev/xenvg/xenimage-vbox1( for vm1)
LV Name: /dev/xenvg/xenimage-vbox2( for vm2)
and then assign the above LVes to the vm. (but it will kill the flexibility)
one question here, creating Logical Volume provides more performance then simply creating two partitions ?, if e.g I create and assign the partitions(rather LV) to vm as:
/dev/cciss/c0d1p3 to vm1
/dev/cciss/c0d1p4 to vm2
so will it be comparatively slow then assigning LV to the vm ?
I am sorry didnt get you properly… can you please provide me a link(url) that I can pass to the HWare guys ?
[QUOTE=Magic31;4998]
One more thing… if this is a POC, why SLES 11 SP1 and not SLES 11 SP2 (which has the newer kernel and Xen version)?[/QUOTE]
XEN host OS: SLES 11 SP2
vm os: SLES 11 SP1
The SAP partner(customer) here is not ready to do this POC atop SLES 11 SP2
Also we mistakenly let the “Create the Sparse Image” file check enabled during the virtual machine (file base disk) creation… would it be a reason/issue only ?
By the way do you think that assigning 6 CPUs to both domU(vm) and only 4 cpus to domain0 is a good idea ? We just assign 6 CPUs when we were creating vm from
virt-manager… i,e we didnt perform the cpu pinning(http://www.suse.com/documentation/sled11/singlehtml/book_xen/book_xen.html#xen.config.cpupinning)… any suggestion on this would again highly appreciated
I got the chance to monitor the servers, and and its disk I/O thats the cause of problem.
I found that the virtual machines has full virtual disks(/dev/hdX) rather then paravirtual disks(/dev/xvdX)… I dont know how these virtual machine(SLES 11 SP 1) got the the full virtual disks.
now what should I do ? is there any virtual machine driver pack available for SUSE Linux SP1 ?
[QUOTE=sharfuddin;5079]I got the chance to monitor the servers, and and its disk I/O thats the cause of problem.
I found that the virtual machines has full virtual disks(/dev/hdX) rather then paravirtual disks(/dev/xvdX)… I dont know how these virtual machine(SLES 11 SP 1) got the the full virtual disks.
now what should I do ? is there any virtual machine driver pack available for SUSE Linux SP1 ?
Thanks
Muhammad Sharfuddin[/QUOTE]
Hi Muhammad,
Sorry for my delay in answering to your previous post…
In any case, coming back to your previous post, I meant performance will always be better when using RAW/physical devices for the VM’s… where possible with no to as little as possible management on host/dom0 side.
With local storage using partitions for the domU/guests is not always possible due to partition table limitations and what the hardware (raid controller mainly) will allow you to do in how storage is presented.
Using LVM on the host/dom0 to manage volumes that get handed down to the guests is a means to be able to give the domU’s exactly the mount of storage they need and hand out a volume per guest disk - and LVM does not impose very much management overhead on the host while giving flexibility. One best practice when using LVM for the domUs/guests, is to keep the partitions for the host/dom0 traditional/non LVM.
Long story shot: phy access for the storage of the domU’s should perform (much) better than file based disk’s. And indeed, for production systems creating a sparse file for the disk makes disk performance drop further (with the current methods).
I did not know SAP was not supported (yet) on SLES 11 SP2, in that case SLES 11 SP1 is fine.
There is no VMDP (virtual driver pack) for SLES as it runs paravirtual and all the needed optimizations are in the guest kernel (that’s also running the Xen kernel in guest mode). No extra options should be needed there.
Sometimes it can help to play with the scheduler settings on host and/or guest (choosing the NOOP scheduler for example)… but that depends on your setup and I’ve found with SLES 11 the defaults are already properly tuned.
Getting the SLES domU’s to run on phy connected storage would be the best way to continue the POC. If you have the room try cloning the current domU file based disks to LVM or raw partitions (using tools like dd, partimage, etc).
Would you mind placing the plain VM config/or the disk definition section (found in /etc/xen/vm by default) on this forum so we can have a look at what’s been configured?
[QUOTE=Magic31;5088]Would you mind placing the plain VM config/or the disk definition section (found in /etc/xen/vm by default) on this forum so we can have a look at what’s been configured?
[QUOTE=sharfuddin;5111]Also I got the following response from Support:
[QUOTE]Response from our backline:
The reason why these disks are showing as full virtualization is because the guest was build as Full Virtualization (FV). By default, SLES guests are installed using PV. I can say that the best way now is reinstall this guest using PV. I’ve tested try to convert from FV to PV and it didn’t work. Customer has to reinstall this guest and restore data from backup
[/QUOTE][/QUOTE]
Hi Muhammad,
Indeed, that would be my advice as well, try converting or reinstalling using phy disk for the guests. As the disks are sized at 1TB, reinstalling would seem a quicker way than trying to convirt… on the other hand, depending on guest filesystem and how much data is really in use… a tool like CloneZilla could be a way to clone the disks (directly from disk to disk or via disk > image > disk) to a new physical partition(s) or LVM volume(s) that has been configured as guest disk(s).
Hope that gives the POC you are hoping for (and a little more )
)