Open source SCM (version control) systems are somewhat like buses: you wait hours for a bus to come along, and then 3 arrive all at once. Well with SCM systems, for years everyone has been using (and complaining about) CVS, and then in the space of little over a year, 4 or more really good contenders spring into existance. Well actually it was a little more complicated that this – probably 2-3 years ago Subversion and GNU Arch started to gain a significant userbase. Subversion has frequently been criticised (rightly or wrongly, I’m not going to debate that now) for not being forward thinking / abitious enough – in terms of architecture its basically no more advanced that CVS with atomic changelists / commits – although the underlying technologies are definitely saner. GNU Arch took a big step forward by switching to a decentralized architecture, but its command syntax is incredibly obtuse & the implementation has some questionable techical decisions – such as ignoring POSIX/UNIX standard APIs from (G)LibC and writing them from scratch.
Getting back to my main point, however, the combination of the great Linux Kernel BitKeeper debacle, and dissatisfaction with the tradeoffs between Subversion & GNU Arch, appears to have accelerated the development of open source SCM to an even faster rate. We now have a choice of GIT, Monotone, Mercurial, Bazaar-NG, CodeVille, and Darcs. One thing these SCM systems all have in common is a distributed architecture, secondly learning from the problems of GNU Arch, they also strive to be as easy to learn & use as CVS
When starting work on the OLPC project we needed to pick an SCM system & we were determined that it would not be CVS; Subversion was also discounted due to its inability to work offline, so we required a distributed SCM system. I’d like to say we did a thorough analysis of the distributed SCM systems listed above, but we didn’t – David & both just came to agreement that Mercurial looked like a good tool and so it came to be. It turned out to be a very good choice – all day-to-day operations are incredibly fast; you can trivially work offline (as with any distributed SCM); the core commands were pretty ease to learn – pretty much regular CVS commands, with addition of “clone”, “push” & “pull” for synchronizing with remote repositories; it has minimal pre-requisite dependancies; trivial publishing of readonly anonymous repositories; did I mention it is fast yet ? After a month and a half of using it, I’ve only found one aspect that irritates me – when you pull down changes from a remote repository & you have local changes not yet pushed, it temporarily creates two HEADs, which you then have to merge, even if the changelists don’t conflict. Still this merge is a trivial process, so nothing to be worried about.
I’ve been so impressed with using it for OLPC, that I’ve decided to switch all my personal projects over to Mercurial for future development. For this process I chose to use Tailor which is a sub-project of Darcs providing a general purpose, bi-directional change history converter for a large number of SCM systems. After a little playing around to optimize the conversion, I now have 20 projects up and running, with 5 years of history intact. Very impressive.
Those of you running on Windows, or those for whom QEMU is too slow, might like to try out running the OLPC firmware images within VMWare Player. There’s two steps required to try this out, converting the disk image to VMWare format, and creating a machine configuration file. To convert the disk image, the QEMU disk management tool ‘qemu-img’ is used:
# qemu-img convert olpc.img -O vmdk olpc.vmdk
Now, the next step is to create a configuration file describing the type of machine we need. The core bits are, an IDE harddrive pointing to the olpc.vmdk image, a network device, 128 MB of ram, and optionally a sound card, and USB support. There are online tools to create a suitable config, or just take this example I created earlier:
config.version = "8"
virtualHW.version = "3"
MemAllowAutoScaleDown = "FALSE"
MemTrimRate = "-1"
uuid.location = "56 4d 4a 49 37 3e de 3c-a8 71 14 0f b4 e4 ce cc"
uuid.bios = "56 4d 4a 49 37 3e de 3c-a8 71 14 0f b4 e4 ce cc"
uuid.action = "create"
checkpoint.vmState = ""
displayName = "OLPC Fedora Linux"
guestOS = "other26xlinux"
memsize = "128"
ethernet0.present = "TRUE"
ethernet0.connectionType = "nat"
ethernet0.addressType = "generated"
ethernet0.generatedAddress = "00:0c:29:e4:ce:cc"
ethernet0.generatedAddressOffset = "0"
usb.present = "TRUE"
usb.generic.autoconnect = "FALSE"
sound.present = "TRUE"
sound.virtualdev = "es1371"
scsi0.present = "FALSE"
floppy0.present = "FALSE"
ide0:0.present = "TRUE"
ide0:0.fileName = "olpc.vmdk"
ide0:0.deviceType = "disk"
ide0:0.mode = "persistent"
ide0:0.redo = ""
ide0:0.writeThrough = "TRUE"
ide0:0.startConnected = "TRUE"
ide0:1.present = "FALSE"
ide1:0.present = "FALSE"
ide1:1.present = "FALSE"
With the initial disk image released a couple of weeks ago, the X server won’t work, but the next image will include the display driver for VMWare. Anyway, save the above configuration file (or an equivalent) as olpc.vmx and then run vmplayer olpc.vmx to get started
Chris mentioned…
make sure that when I pointed the simulator to the disk image (the olpc-simulator create command) I had to make sure I used the full path to the disk image, otherwise the olpc-simulator start would just silently exit. Other people I talked to stumbled over these issues as well, so I figured I would pass them along.
If you have such trouble with the machine not starting up, then a quick tip (which will help you narrow down the problem) is to view the runtime logs using the ‘olpc-simulator log [MACHINE]’ command. For example, lets create a machine with a disk image without specifying an absolute path & see what error is produced:
[berrange@localhost ~]$ olpc-simulator create olpc.img demo
[berrange@localhost ~]$ olpc-simulator start demo
[berrange@localhost ~]$ olpc-simulator log demo
Starting qemu at 01:56:06 on 20 February 2006
qemu -monitor pty -hda olpc.img -net nic,vlan=0 -net user,vlan=0 -serial null -parallel null -M pc -boot c -m 128 -smp 1 -title OLPC demo
With environment variables
DISPLAY -> :0.0
qemu: could not open hard disk image 'olpc.img'
[berrange@localhost ~]$
So, with Chris announcing our work to the world, I can now be a little less coy with my postings. As I mentioned previously, we decided that we needed the management API to be a first class peer of the management UI. One of the first examples of its benefit was in creation of the OLPC Simulator tool. This simple command line tool provides a way to quick way to create machine instances for testing/running OLPC firmware images:
$ olpc-simulator create /path/to/olpc-2006_02_06_16_08.ext3 demo
$ olpc-simulator start demo
It calls out to the same DBus API as the main QEMU Admin GUI, so all changes it makes immediately appear in the GUI too! Another plan we have is to write an extension to the Test-AutoBuild workflow engine providing a means to automate the testing of OS images by having the test harness run them with QEMU. For example, the code for a stage to start a VM would be as short as
my $qemu = $bus->get_service("com.redhat.qemu.Manager");
my $manager = $qemu->get_object("/qemu/manager");
if (!$manager->has_machine($self->option("vm-name"))) {
$self->fail("no virtual machine called " . $self->option("vm-name") . " is available");
return;
}
my $machine = $manager->get_child_object("/machine/" . $self->option("vm-name"));
my $delay = $self->option("startup-delay");
$delay = 5 unless defined $delay;
if ($machine->is_running) {
$log->warn("Machine was already running, so we stopped it");
$machine->stop;
sleep $delay;
}
$machine->start;
sleep $delay;
unless ($machine->is_running) {
$self->fail("unable to start the virtual machine");
return;
}
Last week I hoooked up the functionality for managing ISO & disk images within QEMU-Admin. This lets one import a disk image into the management tool, and then when adding a new harddrive to a machine, you can quickly clone the disk image for use with this machine. When cloning disks one can also convert formats to, for example, VMWare compatible format. Similarly with the ISO image library, one might import the Fedora install CD isos, so when setting up CDROM devices, they’re just a click away. In time it’d be desirable to have UI to directly launch “Live CD” isos with a generic machine template. To show off this new functionality, here’s a new flash demo
Back in November last year I moved from working as a consultant in Red Hat‘s Global Professional Services divison, back to being a member of Engineering. The first project I’ve been involved in is the $100 Laptop, in particular looking at how we can run OS images without needing the real hardware. We quickly identified QEMU as being a reasonable starting point on which to build out a basic ‘Laptop Simulator’. The base QEMU engine is command line driven, and although there is KQEMU and GTK QEMU provide some form of management frontend, neither was entirely satisfactory.
VMWare GSX server has a basic API which lets one create and control virtual machines from Perl. The API, however, was clearly tacked onto the admin UI as an afterthought, and while the capabilities it exposes are very useful, one is frustrated by the bits not exposed to the API. Anticipating the need to construct autmated test harnesses to control QEMU virtual machines, it was clear that any management tool for QEMU must treat its API as a first-class peer to the actual UI. In addition exposing the API as Perl limits the level of interoperability for developers who might want to write their tools in C, or Python, or Java. Being an active member of the DBus community, the answer to this goal was clear:
- Write a management API as a DBus service
- Write a completely separate admin UI as a DBus client
After a few weeks hard work, we now have not only a DBus service for managing QEMU, but a general purpose Python admin UI providing similar levels of functionality to that of VMWare GSX server, a command line Perl tool for quickly launching pre-built OS images with a virtual hardware configuration aligned to that of the $100 laptop, and the beginnings of an automated test suite. The full code trees will be opened up to the community over the next few weeks, but as a taster of what’s to come, there is a flash movie here