Following the libvirt CVS repository using Mercurial, Tailor and patch queues

Posted: October 20th, 2008 | Filed under: Coding Tips, libvirt, Virt Tools | 2 Comments »

The libvirt project uses CVS as its primary SCM system. Our project code submission process requires that people submit patches for review & approval on the mailing list before committing them to CVS. This is great for catching stupid bugs and working through various design options, but CVS really isn’t conducive to this kind of workflow. It is not unusual to have a number of outstanding patches going through review, and you want to keep them all tracked separately during development. In other words you want a queue of pending patches.

There are many ways to implement such a workflow. A long time ago Andrew Morton had a set of scripts for managing patches, which evolved into Quilt. In these modern times, many projects (including the kernel of course) use GIT, and its ability to rebase branches to achieve this workflow. We even maintain a GIT mirror of libvirt which could be used for managing outstanding patches. While certainly a very capable tool, GIT has a somewhat steep learning curve, and even once learnt it violates the principle of least surprise on a daily basis with seemingly simple operations working in the most bizarre way. So with libvirt though I’ve been using a Mercurial mirror, and its ‘mq’ extension for patch management. This isn’t the only way of skinning the cat – you could also use branches, or the new ‘rebase’ command inspired by GIT, but my mind works in terms of patch queues, so I chose ‘mq’. For benefit of anyone else with an interest in Mecurial, here’s a quick overview of my workflow

The first task is to get a reliable process to synchronize the CVS repository to Mercurial. For this purpose I chose to use Tailor, though there are plenty of other tools which do the job fine too. For a repeatable conversion, you need to create a recipe describing what you want. I have the following script saved as $HOME/work/libvirt-hg.tailor, with execute permission enabled:

$ sudo yum install tailor
$ cd $HOME/work
$ cat > libvirt-hg.tailor <<EOF
#!/usr/bin/env tailor

"""
[DEFAULT]
verbose = True

[project]
target = hg:target
start-revision = INITIAL
root-directory = /home/berrange/work
state-file = libvirt-hg.state
source = cvs:source
subdir = libvirt-hg

[cvs:source]
module = libvirt
repository = :pserver:anoncvs@libvirt.org:2401/data/cvs

[hg:target]

"""
EOF
$ chmod +x libvirt-hg.tailor
$ ./libvirt-hg.tailor

When invoking this script for the first time it’ll take a seriously long time, checking out every revision of every file in CVS, figuring out the changesets, and populating a mercurial repository with them. When complete it will have left a directory libvirt-hg containing the mercurial repository mirroring the CVS repo, and a file libvirt-hg.state recording how much work it has done. On a daily basis (or more/less often as desired), re-running libvirt-hg.tailor will make it “catch up” with any new changes in CVS, updating the libirt-hg repository with the new changesets.

While you could work directly in the libvirt-hg repository, my preference is to work in a clone of it, and leave that as a pristine mirror of the CVS repository. So I create a repository called ‘libvirt-work’ for my day-to-day patch queue. If I want to try out someone else’s work – for example Ben Guthro’s libvirt events patchset, I create a patch queue just for that, by again cloning the pristine libvirt-hg repository.

 $ cd $HOME/work
 $ hg clone libvirt-hg libvirt-work

For some reason Mercurial extensions aren’t enabled by default, so if you’ve not used the ‘mq’ extension before, create a $HOME/.hgrc containing:

$ cat $HOME/.hgrc
[ui]
username="Daniel P. Berrange <berrange@redhat.com>"
[diff]
git = 1
showfunc = 1
[extensions]
hgext.hgk=
hgext.mq=
hgext.extdiff =

This does a number of things. The ‘showfunc’ setting makes it include function names in context diffs. The ‘hgk’ extension is a graphical repository browser, ‘mq’ is the patch queue extension, and ‘extdiff’ allows use of more interesting diff options. With that in place I can create a queue for my new bit of work in libvirt

$ cd $HOME/work
$ cd libvirt-work
$ hg qinit

The first patch I want to work on is refactoring mac address handling, so I’ll add a patch to track this work

$ hg qnew mac-address-refactor

Now off editing files as normal – ‘hg add’ / ‘hg remove; to create / delete files as needed, ‘hg status’ or ‘hg diff’ to see what’s changed, etc. The only difference comes when the work is complete – instead of using ‘hg commit’ to record a permanent changeset, I want to update the patch queue. This is done with the ‘hg qrefresh’ command, and the ‘hg qdiff’ command will show you the contents of the patch file

$ hg qrefresh
$ hg qdiff | diffstat
 capabilities.c  |   16 +++++++++++++++-
 capabilities.h  |   11 +++++++++++
 domain_conf.c   |   34 +++++++---------------------------
 domain_conf.h   |    8 ++------
 lxc_conf.c      |    3 +++
 lxc_driver.c    |    4 +++-
 openvz_conf.c   |    2 +-
 qemu_conf.c     |    3 +++
 qemu_driver.c   |    6 ++++--
 util.c          |   24 +++++++++++++++++++++++-
 util.h          |   12 +++++++++++-
 xen_internal.c  |    3 +++
 xend_internal.c |    8 ++++++--
 xm_internal.c   |   34 +++++++++++-----------------------
 14 files changed, 103 insertions(+), 65 deletions(-)

This bit of work was a pre-cursor to the real thing I wanted to work on, the OpenVZ networking code. With this refactoring out of the way, I want to add support for the ‘get version’ operation in OpenVZ driver, so I can start a new patch

$ hg qnew openvz-version

Patches stack up in a series, each building on the last

$ hg qseries
mac-address-refactor
openvz-version

Fast forward another hour, and I’ve got the version operation implemented, and ready for the final patch, enhancing the OpenVZ networking support.

$ hg qnew openvz-network
$ hg qseries
mac-address-refactor
openvz-version
openvz-network

In working through this 3rd patch though, I realize there was a problem in the first one, so I save my current work, and then ‘pop’ patches off the queue until I get back to the first.

$ hg qrefresh
$ hg qtop
openvz-network
$ hg qpop
Now at: openvz-version
$ hg qpop
Now at: mac-address-refactor

Once I’ve fixed the mistakes in this patch, I can then ‘push’ patches back onto the queue. If you’re lucky Mercurial can resolve / merge changes, but as with any rebase, sometimes there are conflicts which require fixing up by hand. If this occurrs, ‘.rej’ reject files are saved which must be manually addressed, and then the updated patch saved with ‘hg qrefresh’ before continuing.

$ hg qpush
applying openvz-version
Now at: openvz-version
$ hg qpush
applying openvz-network
Now at: openvz-network

When a piece of work is done the raw patch files all live in $REPO/.hg/patches/ and can be submitted for review to the mailing list.

If a bit of work goes on for a long time, it is often neccessary to re-sync with latest upstream CVS repository. First step in this re-base is to get Tailor to pull down all latest changes into the the local libvirt-hg repository

$ cd $HOME/work
$ ./libvirt-hg.tailor

Then in the work-ing repository to be rebased, first ‘pop’ off all local patches. Then pull in the new changesets from libvirt-hg, before re-applying the patches, fixing up any merge conflicts which arise

$ hg qpop --all
Patch queue now empty
$ hg pull
pulling from /home/berrange/work/libvirt-hg
searching for changes
adding changesets
adding manifests
adding file changes
added 15 changesets with 67 changes to 48 files
(run 'hg update' to get a working copy)
$ hg update
48 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg qpush
applying mac-address-refactor
Now at: mac-address-refactor
$ hg qpush
applying openvz-version
Now at: openvz-version
$ hg qpush
applying openvz-network
Now at: openvz-network

This time I was lucky and didn’t have any merge conflicts.

While I remmeber, another reason for keeping the libvirt-hg repository pristine, is that I typically do work on different features across different machines. On one machine I make be doing OpenVZ work, while another has a patch queue of KVM work, while yet another may be Xen work. Instead of running Tailor on every machine, I can just have one master mercurial mirror, and push that where needed. Then each local machine has its own independant libvirt-work cloned repository with relevant patchqueue.

There is soo much more I could describe here – I’ve not even mentioned most of the commands available in the ‘mq’ extension – ‘hg help’ will show them all. For further reading, turn to the Mercurial mq wiki page

New Java bindings for libvirt

Posted: June 26th, 2008 | Filed under: libvirt, Virt Tools | No Comments »

DV has recently been looking at the issue of Java bindings for libvirt. A few months back a libvirt community member, Tóth István, contributed most of the code for Java bindings to libvirt. Daniel has now taken this codebase added a build system, and is hosting it in the libvirt CVS repository and done a formal release. This should be hitting Fedora 10 rawhide in the near future, meaning we now have bindings for C, Perl, Python, OCaml, Ruby and Java. Now who wants to do a PHP binding….that’s the only other language commonly requested

Announcing oVirt: a web based virtual machine management application

Posted: February 14th, 2008 | Filed under: libvirt, Virt Tools | No Comments »

Way back in Fedora Core 5, we introduced the libvirt management library for virtual machines. Shortly thereafter in Fedora Core 6 we introduced Virtual Machine Manager (aka virt-manager) desktop application. In Fedora Core 7 we introduced KVM as a virtualization technology. In Fedora 8 we introduced secure remote management of virtual machines using TLS/x509, further enhanced to support Kerberos in Fedora 9. Meanwhile the FreeIPA project has been busy working to integrate Kerberos and LDAP (Fedora Directory Server).

Clearly the time has come to move beyond managing handfuls of virtual machines with a desktop application, and take full advantage of all the infrastructure we’ve painstakingly prepared. To move to a web-based management capable of scaling from a handful of machines, up to an entire data center, allowing administration from any client OS whether Linux or Windows or Mac OS-X.

It is time for oVirt.

Quoting Hugh’s announcement

oVirt is:

    * A small OS image that runs libvirt and hosts virtual machines
    * A Web-based virtual machine management console

oVirt goals:

    * Empower virtual machine owners without giving up control of
      hardware
    * Automate virtual machine clustering, load balancing, and SLA
      maintenance
    * Simplify management of large numbers of machines
    * Work across platforms and architectures

oVirt uses:

    * A kerberos/LDAP server for authentication and authorization
      (oVirt ships with FreeIPA)
    * DNS/DHCP services on the local LAN -- or provides them for oVirt
      hosts over a private network if desired
    * Libvirt for virtual machine management, storage management, and
      secure remote communication
    * collectd for stats gathering and monitoring
    * Rails for rapid, flexible development

oVirt mailing list: http://www.redhat.com/mailman/listinfo/ovirt-devel
oVirt IRC: irc.freenode.net/#ovirt
oVirt website: http://ovirt.org

Fedora 9 (current rawhide) is providing the underlying OS distribution for both the “managed nodes” running guests, and the management application itself. We’re targetting libvirt as the mangement API, to avoid being tied into any single virtualization technology, though KVM is our current technology of choice due to its clean integration with the Linux kernel, and its ever improving performance.

PolicyKit and libvirt integration

Posted: January 11th, 2008 | Filed under: libvirt, Virt Tools | 3 Comments »

For Fedora 9 one of the new feature’s we’ve got pending for virtualization is integration with PolicyKit. This will allow virt-manager to manage local hypervisor connections without having to run as root via consolehelper. Although the virt-manager part of this won’t be ready for a while yet, the libvirt bits were made available in libvirt 0.4.0 just before christmas. As a sneak preview this is now in updates-testing and already gives you the ability to run virsh as non root.

For example, currently if you run virsh as non-root you’l lsee something like

$ virsh --connect qemu:///system
libvir: Remote error : authentication failed
error: failed to connect to the hypervisor

Now with PolicyKit support you can use ‘polkit-grant’ to authenticate and then you’ll be able to run virsh without issue!

$ polkit-grant --gain org.libvirt.unix.manage
Attempting to gain the privilege for org.libvirt.unix.manage.
Authentication is required.
Password:
Keep this privilege for the session? [no/session]?
session
Successfully gained the privilege for org.libvirt.unix.manage.
$ virsh --connect qemu:///system
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh #  start VirtTest
virsh # list
 Id Name                 State
----------------------------------
  1 VirtTest             running

In other news, Rich Jones has created a Mozilla Plugin for GTK-VNC so there’s at last a less-sucky replacement for the terrible Java VNC plugins out there

Announcing a libvirt CIM provider

Posted: November 5th, 2007 | Filed under: libvirt, Virt Tools | No Comments »

For the short story, read the announcement. For the long story, read on…..

The libvirt project provides a hypervisor agnostic API for managing virtual machines (and their associated resources like their network & storage). The libvirt API has 3 core characteristics – simplicity – minimal code required to get useful work done; standard – the same API can be used across any virtualization backend; stable – guarenteed stable public API and XML descriptions across releases. In the short time it has been around, libvirt has proved impressively popular, finding its way in all the main Linux distributions, as well as Open Solaris. There are drivers in libvirt for Xen, QEMU, KVM, OpenVZ and soon Linux-VServer. If someones contributes VMWare, UML, and VirtualBox support we’ll basically have complete coverage for all common Linux virtualization platforms in a single open source API, usable by open & closed source apps alike (libvirt is LGPL licensed explicitly to enable use by closed source apps).

In the enterprise world, people like to form committees to come up with comprehensive standards to cover everything you can think of, and then go off and write multiple implementations of these standards ;-) For virtualization, the ‘standard’ is based around the DMTF / CIM specification. Pretty much since the start of the libvirt project, people have asked why we didn’t just implement the CIM APIs directly. We always wanted the core of our public APIs to be simpler to use though. At the same time, we recognise that some people really want to use a CIM API for their virtualization management tools. Thus we’ve always supported the idea of writing a CIM provider on top of libvirt. Not only would this mean only a single CIM provider implementation is needed for all of the virt platforms supported in libvirt, but it gives good interoperability between tools using libvirt vs CIM.

Over the past year & a half (or more), the Xen project has developed a CIM provider using the Xen-API as the underlying implementation. Indeed at one time this actually used libvirt instead of Xen-API, but back when the switch was made to Xen-API, KVM wasn’t around so the benefits of a hypervisor agnositic libvirt API were more hypothetical than tangible. KVM does eventually appear on the scene & sure enough, the topic of developing CIM providers for KVM soon comes up. KVM though doesn’t have any form of existing management API to deal with – a KVM guest is ‘managed’ by running a QEMU process, and killing it when you’re done. QEMU provides a mini admin shell for controlling various aspects of its behaviour. Turning this into a formal API is hard work – libvirt has already done it once and you don’t want to have to duplicate that :-)

At the recent KVM forum, IBM announced that they were going to develop a CIM provider based on libvirt, and release it to the open source community. Fast forward a few months and I’m happy to announce their plans have come to fruition. Furthermore, this is not just a 3rd party add-on, the CIM provider is to be a core part of the libvirt project & ecosystem. We’re collaborating on ideas, API requirements, hosting for our infrastructure (websites, SCM repos, mailing lists, IRC channels, etc) and above all working towards the common goal of providing state-of-the-art open source management APIs, capable of working across any virtualization platform.