It has been a busy start to the week, both on work & non-work related projects. In the former case we made a new snapshot of virt-manager available – version 0.2.0. The major milestone in this release is the wizard for easy creation of Xen guest virtual machines. RPMs are not yet built into rawhide, but in the meantime there are screenshots of the installation process.
Even more important to me than virt-manager though, is the (long delayed) availability of the new Test-AutoBuild stable release series. Test-AutoBuild is a build & test automation harness aimed at upstream developmers, enabling “continuous integration” over a codebase. ie 24×7 the build harness checks latest code out of SCM repository, builds it, tests it, publishes the results, and starts process again. If you work in the Java world you may be aware of a project called CruiseControl – well, Test-AutoBuild is like CruiseControl on steriods – rather than just building & testing a single code module, it can be setup to perform full regression testing across a suite of dependant software packages.
For example, consider how it might apply to the virt-manager application – this depends on libvirt & xeninst; xeninst in turn depends on libvirt; libvirt in turn depends on xen. Because it is bleeding edge development, virt-manager is tracking latest upstream development branches of libvirt, xeninst & xen. Test-AutoBuild would enable build & test of the entire stack – checking out & building xen, then libvirt, then xeninst & finally virt-manager. Doing this 24×7 mean we can get feedback on any changes/breakage in the dependancy stack in a matter of hours if not sooner. If one weren’t building & testing the against the full stack the breakage might not come to light until months later, by which time resolving it could become a much more time consuming issue. The sooner you know about a problem, the sooner it can be resolved, and the less time is wasted debugging in the long term.
Anyway back to my original point – the new release. Its hard to believe we first started work on this release branch about 2 years back with major re-factoring of the build workflow engine. Shortly after we started work to support caching an arbitrary number of historic biulds, extraction of SCM repository changelogs, and a whole host of other features. We could have released the code much sooner, with so much code change it seemed only prudent to invest an equally large amount of time ensuring the code is stable / reliable. So I spent as much time as possible over the following year basically just writing unit tests for the code & fixing the issues they uncovered.
The upshot is that this new release will provide a very good foundation for further development of Test-AutoBuild – with such an increased level of test coverage it ought to be possible to get new releases cut on a much more frequent basis – back to the classic open source mantra of release early, release often. With recent switch of DBus bindings from CVS to GIT, one of my highest priority development tasks is going to be the addition of GIT support (Test-AutoBuild currently supports Subversion, CVS, Mercurial, GNU Arch, Perforce & local filesystem).
Oh and if you’re thinking, I’m only one person I don’t need such automation of my code, think again. I develop on Fedora normally, but my autobuild server runs on Debian – I’ve lost count of number of occassions it has highlighted some incompatibility I mistakenly introduced – for example using a feature only supported in a very new version of a particular library or app, not commonly available.
Why aren’t more applications using gnome-keyring? is the question asked on the main page of the pam-keyring module. Their posited answer is that it is inconvenient for users / lacks integration with the authentication proess. Now this is indeed true – it would be very desirable from a user’s POV if they didn’t have to separately unlock the keyring – it should just be unlocked when logging in. This is not, however, the reason why more applications don’t use gnome-keyring – it is merely a user inconvenience. Having just spent some time adding support for gnome-keyring to virt-manager, the real answer is crystal clear: There is zero documentation about either the C or Python APIs for gnome-keyring. Even having read the source from existing apps using it like NetworkManager, many aspects of the API are far from clear – particularly all the asynchronous methods. Its no wonder all apps I can find use the synchronous methods instead – you can at least take a reasonable guess at their API contract in most cases. That said when the API is all about securely managing passwords, it seems very wrong to be guessing about the best way to use it. If someone were to document gnome-keyring, the barrier to use for application developers would be dramatically lower.
For several years now I’ve been using SpamAssassin as my primary tool to fight the spam arriving at my mail server. The performance of SpamAssassin has never been great (and using spamd/spamc is not a real fix – it merely avoids the startup overhead, doing nothing for the speed of the classification engine). Thus, about 6 months ago, to lower the overhead on my machine (a UserModeLinux host from bytemark with a mere 64 MB of RAM), I moved the RBL checks from SpamAssassin into Exim. This was a big help, but detecting and filtering spam from blacklisted domains only stops approx 30-40% of spam – the rest still goes through the heavy SA.
To compound problems, spammers have increasingly been finding ways to get around the Bayesian DB I’ve trained with SpamAssassin, which means I’ve spent a large amount of time re-training SpamAssassin with mails it got wrong. Well SpamAssassin is a really slow learner and today I’ve had enough, thrown it away and decided to pursue a new strategy of “total annihilation”. What this means in practice is that I’m now using 3 bayesian classifiers at once – BMF, QSF and BogoFilter – all of which are written in C, so crazy fast (training on a few thousand mails takes seconds instead of minutes). Every incoming message is fed through all three of these engines and if 2 or more of them agree on it being spam it gets sent to a spam folder. If one of them didn’t agree, but the other two gave it a 100% spam rating, that dissenting one will be auto-trained. In addition I’ve also added Rik’s PSBL blacklist to my exim config, to backup my existing use of SpamHaus.
NB. I can’t really take credit for having come up with this plan – I’ve used the procmail recipes from acme.com‘s bayesian mail filtering pages with minimal change. Hopefully this will prove to be a winning formula – certainly early indications are very promising, but of course only time will tell.
Hugh Brock, Jeremy Katz and myself have been working hard on various aspects of the Xen GUI management tools. Jeremy’s re-factored the xenguest-install.py script from Fedora Core 5 into a formal python module / library API called ‘xeninst’ which should be appearing in rawhide fairly soon. Hugh, meanwhile has been tackling the Xen guest installation problem from the UI side, producing a wizard for the virt-manager app to collect the info neccessary provision new guest instances. Today we reached the major milestone of connecting the wizard to the xeninst libray and were able to perform a (near) end-to-end graphical install of a guest OS. With seemless progression from the data collection wizard, to the embedded VNC (or serial) console in virt-manager, installation of Xen guest operating systems in a Fedora host ought to become a whole lot more friendly RSN. A few days more bug fixing / tidying up loose ends and we’ll release another snapshot virt-manager and make re-try pushing RPMs into rawhide for wider testing. Today also saw a major new release of libvirt providing much better support for hardware assisted virtualization (aka HVM / VT), automated testing via the mock hypervisor driver, and a plethora of bug fixes / minor feature enhancements.
So at the start of the year I moved from London over to Boston. In that time I’ve slowly figured out the way of life over here, but of course at the same time I’ve been trying to keep in touch with British culture. So what does my ‘little britain’ consist of…. The first stage is to find a house in a place with a solid English name – in my case Cambridge. Next up is finding a good source of news (ie not Fox) – the excellant BBC news is great for online news, and more recently I got the BBC World Service streaming straight to my SqueezeBox. On that topic, SqueezeBox is setup out of the box to receive 100’s of radio stations from all over the world, so I can still listen to favourites like Virgin Radio – although it tends to lead to timezone confusion – from about 7 in the evening US time, you’re listening to the UK graveyard shift of DJ’s – oh, and you get all the UK advertising still which is a little amuzing. Back to news though – I discovered you can also get a The Guardian Weekly edition delivered anywhere in the world – a high density / condensed round-up of the weeks happenings.
While back in the UK a couple of weeks ago I paid a visit to the Plymouth Gin Distillery which is England’s oldest Gin distiller having been in operation since at least 1793. Even today every Royal Navy ship commissioned is provided with a wooden casket containing two bottles of the special “Navy Strength” (100% proof – 57% by volume) gin – so named because it is the concentration at which gin can be spilt on gunpowder and still be able to ignite the powder. Well you can’t get the navy strength version over here, but a number of liquor stores carry the regular Plymouth Gin. For a summer’s afternoon though, one should really be drinking Pimm’s – the drink of choice at garden parties, or events like Royal Ascot & Wimbledon – its not well known over here, but its surprisingly easily available none the less.
Back to Fedora related news, we’ve been working hard developing the capabilities in libvirt and virt-manager, racing to be ready with a really useful tool for managing Xen VMs in Fedora Core 6. Virt-manager didn’t make it into FC6-test2 (primarily thanks to an inconveniently timed 2-week vacation on my part ;-), but I’m hopeful that we’ll be able to slip it in before test3. Since the 0.1.5 release 2 weeks back, we’ve got the serial console embedded directly in the app, added ability to tune memory allocation & number of CPUs on the fly, and designed & mostly implemented a wizard for creating new VMs (a GUI equivalent of the xenguest-install.py script), so things are looking very promising on the features front.