I’ve concluded that file-based storage for hard disk images is more convenient from a management perspective than using LVM. However, even if one decides to go the route of using files there is still the question of which format to use. Xen supports several options at the moment with more to come (hopefully).

I’ve been reading through the launchpad bugs for Xen and thought that I could use them to illustrate some tips and tricks for using Xen in Ubuntu or Debian. I ran into some problems with Xen today on a system after having been upgraded from Ubuntu 7.10 to Ubuntu 8.04 and from Xen 3.1 to Xen 3.2. This should help as a checklist when diagnosing various Xen problems.

A new kernel, 2.6.24-17, has been released for Ubuntu 8.04 (Hardy Heron) in the hardy-updates repository. This kernel may fix at least 1 xen bug (bug 218126, bug 204010) which I’m anxious to test. I’ve held off upgrading any Xen user domains until these were resolved. I think it would be a good time to at least test an upgrade, and since virtualization makes it easy to rollback I think a test is in order. I also noticed that Ubuntu 7.10 got some new kernel modules in the gutsy-updates repository. Not sure what the changes are but it’s something to be aware of.

I have tended to setup Xen systems with LVM storage due to the performance increases I’ve observed compared to file-based storage. Recently I’ve decided to largely transition to file-based storage primarily for the increase in manageability that file-based virtual machine images provides. Files are much easier to manipulate for backups, migrating to other systems, and similar tasks. While I will still use LVM in certain situations I think the file-based images is a better choice until one runs into performance bottlenecks. Even at that point there may be other alternatives available.

I noticed that I had an field in top labeled "st" and that it was busy once I switched some Xen domains to be file backed (tap:aio) instead of LVM backed (phy). Anyway, after doing some investigation I found that "st" means "steal_time" and is "the cpu time spent in involuntary wait." This is time spent executing other virtual domains in Xen or other virtualization applications. I couldn't find the details when searching for information about the top command (top being one of those difficult search terms). But since top gets its values from /proc/stat I was able to use that to aid my research. I ended up finding the details at http://www.linuxhowtos.org/System/procstat.htm and http://kerneltrap.org/node/8235.

I’ve seen this cause problems on both Xen and VMware though I suspect the problem is widespread. When a virtual ethernet device has checksumming enabled it can cause all sorts of network problems. I once got deep into debugging and found that with checksumming enabled packets larger than the MTU would be sent which would then cause a number of fragmentation messages to be sent. It was a mess, but easily correctable. Some specifics follow.

I think a sign of security-mindedness for a server GNU/Linux distribution is whether it can withstand simple attacks out of the box. At the moment Ubuntu fails this test as a simple fork bomb from any user or any compromised service can render the system useless.

I was reading up on Xen cpu schedulers, which is not easy information to find. My interest is due to a xen domain that has a high load. The credit scheduler is clearly fair as this domain is doing far more work than others on the system. However, the work it is doing is important and I’d like to allocate more cpu to that domain even at the expense of some others. For Xen 3 I can only find information about 3 schedulers: credit, sedf, and bvt. According to the documentation credit is favored over both sedf and bvt. While setting parameters for credit might solve my high load issue I’m not sure it addresses latency issues. Further research is needed.

• Credit-Based CPU Scheduler
• Xen CPU Scheduling
• Xen interface manual

I kind of dug right into Xen but today as I was investigating pygrub and hvmloader I came across some documentation that I wish I had seen when I first started using Xen. It’s not so much that Xen is poorly documented as the good documentation is hard to find. So here are some good places to start:

• Xen: Frequently Confusing Issues
• Xen Basics
• Xen Users’ Manual
• Virtualization in Xen 3.0

Xen (or any emulation virtualization technology)
Pro: Difficult to overcommit memory
Con: Requires more memory

Linux-VServers (or any OS-level virtualization technology)
Pro: Requires less memory
Con: Trivial to overcommit memory