Last friday afternoon, I quietly pushed out an update to the QEMU admin tool (being used for the OLPC SDK) which allows QEMU virtual machines to be connected up to the host network stack. Previously only a private VM-to-VM networks over IP multicast, or SLIRP-based outbound TCP only networks were working with QEMU-Admin. This new capability sets up a private network using VDE (Virtual Distributed Ethernet) switch to which QEMU VMs can connect. The dnsmasq daemon provides DNS & DHCP to this private network, allowing for zero-config networking in the QEMU guests. Finally a little VDE+IPTables magic connects the virtual network to the real host network via a TAP device. For those using NetworkManager, there is even an action script for the NetworkManager dispatcher, to automatically update the IPTables forwarding/masquerading rules when you switch between wired & wireless modes.
There’s probably still a number of bugs in this setup, but this host network bridging is the next step towards a fully automated harness for testing the OLPC ‘firmware’ images. For further info on setting up the network bridge, I’ve written a short set of instructions
Open source SCM (version control) systems are somewhat like buses: you wait hours for a bus to come along, and then 3 arrive all at once. Well with SCM systems, for years everyone has been using (and complaining about) CVS, and then in the space of little over a year, 4 or more really good contenders spring into existance. Well actually it was a little more complicated that this – probably 2-3 years ago Subversion and GNU Arch started to gain a significant userbase. Subversion has frequently been criticised (rightly or wrongly, I’m not going to debate that now) for not being forward thinking / abitious enough – in terms of architecture its basically no more advanced that CVS with atomic changelists / commits – although the underlying technologies are definitely saner. GNU Arch took a big step forward by switching to a decentralized architecture, but its command syntax is incredibly obtuse & the implementation has some questionable techical decisions – such as ignoring POSIX/UNIX standard APIs from (G)LibC and writing them from scratch.
Getting back to my main point, however, the combination of the great Linux Kernel BitKeeper debacle, and dissatisfaction with the tradeoffs between Subversion & GNU Arch, appears to have accelerated the development of open source SCM to an even faster rate. We now have a choice of GIT, Monotone, Mercurial, Bazaar-NG, CodeVille, and Darcs. One thing these SCM systems all have in common is a distributed architecture, secondly learning from the problems of GNU Arch, they also strive to be as easy to learn & use as CVS
When starting work on the OLPC project we needed to pick an SCM system & we were determined that it would not be CVS; Subversion was also discounted due to its inability to work offline, so we required a distributed SCM system. I’d like to say we did a thorough analysis of the distributed SCM systems listed above, but we didn’t – David & both just came to agreement that Mercurial looked like a good tool and so it came to be. It turned out to be a very good choice – all day-to-day operations are incredibly fast; you can trivially work offline (as with any distributed SCM); the core commands were pretty ease to learn – pretty much regular CVS commands, with addition of “clone”, “push” & “pull” for synchronizing with remote repositories; it has minimal pre-requisite dependancies; trivial publishing of readonly anonymous repositories; did I mention it is fast yet ? After a month and a half of using it, I’ve only found one aspect that irritates me – when you pull down changes from a remote repository & you have local changes not yet pushed, it temporarily creates two HEADs, which you then have to merge, even if the changelists don’t conflict. Still this merge is a trivial process, so nothing to be worried about.
I’ve been so impressed with using it for OLPC, that I’ve decided to switch all my personal projects over to Mercurial for future development. For this process I chose to use Tailor which is a sub-project of Darcs providing a general purpose, bi-directional change history converter for a large number of SCM systems. After a little playing around to optimize the conversion, I now have 20 projects up and running, with 5 years of history intact. Very impressive.
Those of you running on Windows, or those for whom QEMU is too slow, might like to try out running the OLPC firmware images within VMWare Player. There’s two steps required to try this out, converting the disk image to VMWare format, and creating a machine configuration file. To convert the disk image, the QEMU disk management tool ‘qemu-img’ is used:
# qemu-img convert olpc.img -O vmdk olpc.vmdk
Now, the next step is to create a configuration file describing the type of machine we need. The core bits are, an IDE harddrive pointing to the olpc.vmdk image, a network device, 128 MB of ram, and optionally a sound card, and USB support. There are online tools to create a suitable config, or just take this example I created earlier:
config.version = "8"
virtualHW.version = "3"
MemAllowAutoScaleDown = "FALSE"
MemTrimRate = "-1"
uuid.location = "56 4d 4a 49 37 3e de 3c-a8 71 14 0f b4 e4 ce cc"
uuid.bios = "56 4d 4a 49 37 3e de 3c-a8 71 14 0f b4 e4 ce cc"
uuid.action = "create"
checkpoint.vmState = ""
displayName = "OLPC Fedora Linux"
guestOS = "other26xlinux"
memsize = "128"
ethernet0.present = "TRUE"
ethernet0.connectionType = "nat"
ethernet0.addressType = "generated"
ethernet0.generatedAddress = "00:0c:29:e4:ce:cc"
ethernet0.generatedAddressOffset = "0"
usb.present = "TRUE"
usb.generic.autoconnect = "FALSE"
sound.present = "TRUE"
sound.virtualdev = "es1371"
scsi0.present = "FALSE"
floppy0.present = "FALSE"
ide0:0.present = "TRUE"
ide0:0.fileName = "olpc.vmdk"
ide0:0.deviceType = "disk"
ide0:0.mode = "persistent"
ide0:0.redo = ""
ide0:0.writeThrough = "TRUE"
ide0:0.startConnected = "TRUE"
ide0:1.present = "FALSE"
ide1:0.present = "FALSE"
ide1:1.present = "FALSE"
With the initial disk image released a couple of weeks ago, the X server won’t work, but the next image will include the display driver for VMWare. Anyway, save the above configuration file (or an equivalent) as olpc.vmx and then run vmplayer olpc.vmx to get started
Chris mentioned…
make sure that when I pointed the simulator to the disk image (the olpc-simulator create command) I had to make sure I used the full path to the disk image, otherwise the olpc-simulator start would just silently exit. Other people I talked to stumbled over these issues as well, so I figured I would pass them along.
If you have such trouble with the machine not starting up, then a quick tip (which will help you narrow down the problem) is to view the runtime logs using the ‘olpc-simulator log [MACHINE]’ command. For example, lets create a machine with a disk image without specifying an absolute path & see what error is produced:
[berrange@localhost ~]$ olpc-simulator create olpc.img demo
[berrange@localhost ~]$ olpc-simulator start demo
[berrange@localhost ~]$ olpc-simulator log demo
Starting qemu at 01:56:06 on 20 February 2006
qemu -monitor pty -hda olpc.img -net nic,vlan=0 -net user,vlan=0 -serial null -parallel null -M pc -boot c -m 128 -smp 1 -title OLPC demo
With environment variables
DISPLAY -> :0.0
qemu: could not open hard disk image 'olpc.img'
[berrange@localhost ~]$
So, with Chris announcing our work to the world, I can now be a little less coy with my postings. As I mentioned previously, we decided that we needed the management API to be a first class peer of the management UI. One of the first examples of its benefit was in creation of the OLPC Simulator tool. This simple command line tool provides a way to quick way to create machine instances for testing/running OLPC firmware images:
$ olpc-simulator create /path/to/olpc-2006_02_06_16_08.ext3 demo
$ olpc-simulator start demo
It calls out to the same DBus API as the main QEMU Admin GUI, so all changes it makes immediately appear in the GUI too! Another plan we have is to write an extension to the Test-AutoBuild workflow engine providing a means to automate the testing of OS images by having the test harness run them with QEMU. For example, the code for a stage to start a VM would be as short as
my $qemu = $bus->get_service("com.redhat.qemu.Manager");
my $manager = $qemu->get_object("/qemu/manager");
if (!$manager->has_machine($self->option("vm-name"))) {
$self->fail("no virtual machine called " . $self->option("vm-name") . " is available");
return;
}
my $machine = $manager->get_child_object("/machine/" . $self->option("vm-name"));
my $delay = $self->option("startup-delay");
$delay = 5 unless defined $delay;
if ($machine->is_running) {
$log->warn("Machine was already running, so we stopped it");
$machine->stop;
sleep $delay;
}
$machine->start;
sleep $delay;
unless ($machine->is_running) {
$self->fail("unable to start the virtual machine");
return;
}
Last week I hoooked up the functionality for managing ISO & disk images within QEMU-Admin. This lets one import a disk image into the management tool, and then when adding a new harddrive to a machine, you can quickly clone the disk image for use with this machine. When cloning disks one can also convert formats to, for example, VMWare compatible format. Similarly with the ISO image library, one might import the Fedora install CD isos, so when setting up CDROM devices, they’re just a click away. In time it’d be desirable to have UI to directly launch “Live CD” isos with a generic machine template. To show off this new functionality, here’s a new flash demo