by running “vmstat 1” and “iostat -N 1” both on the DomU (VM) and the Dom0 (host). “iostat” is from the sysstat package, highly recommend at least on the Dom0 Pay special attention to the i/o waits in conjunction with actual disk writes and try to get an idea of what’s happening (when is what data going where, and why is it so slow?).
“I/O scheduling” refers to the algorithms used to decide when to write what data from the OS block buffer to the physical media, see “more /sys/block/*/queue/scheduler”. You’ll probably see “cfq” selected for the disks, which is ok if you have direct access to physical disks. But both within virtual systems, as well as when connected to a SAN, that scheduler can have no idea how to optimize write operations, only the SAN server will. So you could set the scheduler to “deadline” (which gives you a fair amount of tuning options, the defaults may be ok) or even to “noop” (by echoing the according string to the file: “echo deadline > /sys/block/yourDiskDevice/queue/scheduler”). Fine-tuning requires knowledge of your actual usage, I wouldn’t want to recommend this to you at this stage. I would not expect a facto 12 improvement by changing the scheduler, it’s just something to keep in mind for later.
An interesting question would be if you’re just not seeing the problems on Dom0 - it may be writing to the buffers similarily ( 5 vs. 8 seconds), but on Dom0, the disk accesses for system disks and that data LUN go via different controllers. Thus your system is still responsive, while in the background it still is struggling to write that data to the SAN server. On the DomU, everything goes through that single virtual controller to Dom0, thus flushing the buffers impacts the system disks, too. How long does “sync” hang on the Dom0, after the test?
Sounds pretty straight-forward Did I understand that correctly, you tested throughput to “/dev/emcpowera” from within Dom0 with sufficient results? Then how’s your CPU load (Dom0) while the VM flushes its buffers?
Oh, I got that wrong, sorry - reading “blade” I implied you will be running multiple Xen servers (multiple blades). Of course no inter-Xend locking is required with only a single xend
I just installed the sysstat tools on both host and guest. It will take me a bit to get familiar with what I’m looking at here and how to properly interpret the results.
Regards Dom0, yes, throughput with the physical machine (aka host aka Dom0) to the SAN space is excellent (I use /dev/emcpowerb for the physical; a different lun, /dev/emcpowera for the vm), it’s only when writing (copying) from the vm guest that the throughput takes a dive. Note that all luns regardless here though come from the same array, same storage group/zone etc…
During my troubleshooting yesterday, I found something interesting though. I started using the dd command for tests instead of cp, and got very different results.
I ran " dd if=/dev/zero of=/SAN/testfile bs=2M oflag=direct " on the phys host, and it copied the 1 GB in about 2 seconds.
When I ran the equivalent on the vm (dd if=/dev/zero of=/san_lun/testfile bs=2M oflag=direct)I expected the same delay, but it also completed in about 2 seconds, this time, no difference. I really don’t know why.
I ran tests multiple times, in every instance, the results using dd were identical between host and guest, unlike when I use cp. I’m not sure what that means. It seems like the file meta data written using cp is what takes so long to write.
Incidentally, here’s what I have regarding the scheduler, for both host and guest: I assume [cfg] means that’s the current setting?
[INDENT]noop deadline [cfg][/INDENT]
Since this server is not a production server but was created exclusively for testing, I’ll try the deadline setting and it’s defaults for now, and see if that helps.
I’ll get back next week probably with some results from iostat and vmstat.
Thanks again for all the information, I’m in your debt.