I need to build 1 or 2 systems to do some testing of virtualization technologies. They need to be as cheap as possible and there is no need for them to be particularly powerful. I decided to document my search here.

I’ve concluded that file-based storage for hard disk images is more convenient from a management perspective than using LVM. However, even if one decides to go the route of using files there is still the question of which format to use. Xen supports several options at the moment with more to come (hopefully).

A new kernel, 2.6.24-17, has been released for Ubuntu 8.04 (Hardy Heron) in the hardy-updates repository. This kernel may fix at least 1 xen bug (bug 218126, bug 204010) which I’m anxious to test. I’ve held off upgrading any Xen user domains until these were resolved. I think it would be a good time to at least test an upgrade, and since virtualization makes it easy to rollback I think a test is in order. I also noticed that Ubuntu 7.10 got some new kernel modules in the gutsy-updates repository. Not sure what the changes are but it’s something to be aware of.

I’ve been sharing my thoughts about various virtualization solutions a lot over the last few weeks. Several people have asked me for my opinions on the strengths and weaknesses of various products. I’ve also commented online about these issues. I’ve decided to list some of my observations in order to have something to refer to, for myself and others. I’ve listed options separately for servers and desktops and by order of aptitude to those roles.

I really liked VirtualBox and got a lot of use out of it when running my old Windows XP system under Ubuntu. However, the lack of support for 64-bit systems proved too problematic. In testing Ubuntu 8.04 (Hardy Heron) I was only interested in testing the 64-bit versions as that was all I would be running. As some bugs are architecture dependent (iproute and monsterz come to mind) it's important to perform testing on the architecture you will actually use. Under Ubuntu 7.10 (Gutsy Gibbon) I had some issues with QEMU and KVM though I was able to use them enough to do testing (though often only through QEMU with specific parameters at a less than ideal speed). However in Ubuntu 8.04 I found both applications improved and decided to search for a nice front-end to replace the functionality I was used to with VirtualBox.

I have tended to setup Xen systems with LVM storage due to the performance increases I’ve observed compared to file-based storage. Recently I’ve decided to largely transition to file-based storage primarily for the increase in manageability that file-based virtual machine images provides. Files are much easier to manipulate for backups, migrating to other systems, and similar tasks. While I will still use LVM in certain situations I think the file-based images is a better choice until one runs into performance bottlenecks. Even at that point there may be other alternatives available.

In an effort to help with testing the beta release of Ubuntu 8.04 I wanted to set up a number of virtual machines running the OS so I could perform various tests, such as whether certain but in Ubuntu 7.10 (Gutsy Gibbon) had been fixed. Unfortunately I have not been successful so far. As every system save 1 is running the amd64 architecture that is the architecture I would prefer to test. However, it seems that trying to have a virtual amd64 guest on an amd64 host using open source software is problematic.

I just did some reading on unkillable processes. I've seen this happen a few times before and wondered what caused it. I experienced it again today when using KVM. Apparently when processes go into the "D" state they cannot be killed because they are usually waiting for some IO to complete. I have not found any way to fix these processes while a system is running though I continue to investigate. I did find that the 2.6.25 Linux kernel will add a new TASK_KILLABLE state to help alleviate this problem. Hopefully software including KVM will be quickly updated to take advantage of this new state which functions just like TASK_UNINTERRUPTIBLE but allows the process to be killed.

I noticed that I had an field in top labeled "st" and that it was busy once I switched some Xen domains to be file backed (tap:aio) instead of LVM backed (phy). Anyway, after doing some investigation I found that "st" means "steal_time" and is "the cpu time spent in involuntary wait." This is time spent executing other virtual domains in Xen or other virtualization applications. I couldn't find the details when searching for information about the top command (top being one of those difficult search terms). But since top gets its values from /proc/stat I was able to use that to aid my research. I ended up finding the details at http://www.linuxhowtos.org/System/procstat.htm and http://kerneltrap.org/node/8235.

I’ve seen this cause problems on both Xen and VMware though I suspect the problem is widespread. When a virtual ethernet device has checksumming enabled it can cause all sorts of network problems. I once got deep into debugging and found that with checksumming enabled packets larger than the MTU would be sent which would then cause a number of fragmentation messages to be sent. It was a mess, but easily correctable. Some specifics follow.

I kind of dug right into Xen but today as I was investigating pygrub and hvmloader I came across some documentation that I wish I had seen when I first started using Xen. It’s not so much that Xen is poorly documented as the good documentation is hard to find. So here are some good places to start:

• Xen: Frequently Confusing Issues
• Xen Basics
• Xen Users’ Manual
• Virtualization in Xen 3.0

Xen (or any emulation virtualization technology)
Pro: Difficult to overcommit memory
Con: Requires more memory

Linux-VServers (or any OS-level virtualization technology)
Pro: Requires less memory
Con: Trivial to overcommit memory

I just tried to install a 64-bit OS (Ubuntu Server) under VirtualBox OSE 1.5.0 and received the message, "Your CPU does not support long mode. Use a 32bit distribution. I tried toggling the "Enable VT-x/AMD-V" setting but that didn't help. I haven't investigated this yet but I will be disappointed if this is a limitation of VirtualBox OSE.