The historical philosophy of libvirt is for all our core libraries to be written in C and then create bindings to other programming languages or mappings to alternative object models. Thus far we have bindings to Python, Perl, Ruby, OCaml, Php, C#, Java and mappings to the QMF (Matahari), CIM and SNMP object models. The virt-install and virt-manager applications use the python binding to libvirt, but even very early in development of virt-manager it was clear that the libvirt python API is not a natural fit for an application using GTK, since it does not integrate with GObject and in particular GObject signals. Thus virt-manager wraps the libvirt python objects to create real GObjects it then works with. This has been quite successful, but because all the virt-manager code is in python other applications have not been able to take advantage of the higher level libvirt API virt-manager has evolved. In addition the virt-install code (which is called internally by virt-manager) contains a set of Python objects which represent the various libvirt XML schemas as plain old objects with properties and setters/getters. If you’ve developed applications against libvirt, you’ll likely appreciate just how useful such an API would be. Again though, because the API is in Python and (technically) internal to the virt-install codebase, it is not accessible to many other applications
There was clearly space for an independent library mapping the libvirt API and XML schemas to GObject, which could then be used by any application. The task of creating a libvirt GObject library API is large enough, without considering the task of also ensuring it is accessible from all the non-C programming languages. Fortunately, with the release of GNOME-3, GObject introspection has now matured to the point where it can really be used in anger for real application development. The upshot is that it is now feasible to attempt development of a proper libvirt GObject API.
The libvirt-glib package is the result, and it actually contains three related libraries
- libvirt-gib – non-object based glue code between GLib and libvirt. In particular this has APIs to convert libvirt virErrorPtr instances into GError instances, and provides an implementation of the libvirt event loop contract, using the GLib GMain APIs.
- libvirt-gconfig – object based APIs which map libvirt XML documents/schemas into GObject classes. This library explicitly has no direct link to the libvirt API, solely concerning itself with XML management. This is to allow use of libvirt-gconfig from applications which are using one of the object mappings like QMF/CIM/SNMP, instead of the direct libvirt API. This where the current virt-install XML handling objects will be replicated
- libvirt-gobject – object based APIs which map libvirt types and APIs into GObject classes. This library depends on libirt-glib and libvirt-gconfig, and is where the current virt-manager object mapping APIs will be replicated. This library is also adopting the GIO paradigm for allowing asynchronous API invocation & completion, for long running applications. This eliminates much of the need for applications to explicitly use threads (thread usage is hidden behind the async API impl).
From day 1, all the APIs are being developed with GObject introspection in mind., so all methods are fully annotated, and we are generating the glue layer for Vala bindings as standard in order to support the GNOME Boxes application. It is still very early days for development and very little of the libvirt API has been mapped into GObject thus far and work is only just starting on the XML object mappings. The overall target, however, is to develop the library to the state where it can support the aforementioned GNOME Boxes application in Fedora 17, as well as an application sandbox framework I am developing for Fedora 17 (more on that in a later blog post).
For more information
With Fedora 16 out, I have to perform upgrades on my Fedora machines, two laptops, one mac mini and five servers. Officially the only supported way to upgrade Fedora is to use the regular Anaconda installer, or use pre-upgrade. In the years I’ve been using Fedora (since Fedora Core 5), I have never used Anaconda for upgrades because I rarely have direct access to the machine to boot CDs/DVDs, and whenever I have tried pre-upgrade it always fails. This time it failed on one server because /boot was on software-RAID, and failed on the other two servers because /boot was too small. So instead I’ve always ended up doing live upgrades using yum directly. The first few times I did this, there were some surprises, but in the end I’ve settled into a recipe that has a high success rate.
I’m NOT encouraging people follow my approach, but if you have ended up in a situation where a live yum upgrade is your only option, these notes might help you avoid some pitfalls.
- Tip 1: Only attempt to upgrade 1 release at a time, don’t try to skip over releases. This reduces the chances of problems with yum calculating a suitable upgrade path
- Tip 2: Avoid having any 3rd party repositories configured with yum, unless you know they already support both the current and target distro release. In practice this means you want to avoid anything except official Fedora, RPM Fusion and Livna repositories.
The actual upgrade process I follow is this:
- Step 1: Ensure the current install is fully updated.
# yum -y update
- Step 2: Remove any orphaned packages, ie locally installed packages which are not present in any of your active YUM repositories. Orphaned packages are the most common cause for unresolvable RPM dependencies during upgrade. If you choose to skip this step be aware that you will almost certainly need to revisit it, if yum fails to resolve an upgrade path.
# package-cleanup --orphans
# rpm -e ...for each orphan you want to remove...
- Step 3: Purge all YUM cached data about current repos. This just frees up space by removing cached data files that won’t be needed anymore
# yum clean all
- Step 4: Install the fedora-release, fedora-release-rawhide and fedora-release-notes RPMs for the new release.
# wget http://mirror.bytemark.co.uk/fedora/linux/releases/16/Everything/x86_64/os/Packages/fedora-release-16-1.noarch.rpm
# wget http://mirror.bytemark.co.uk/fedora/linux/releases/16/Everything/x86_64/os/Packages/fedora-release-notes-16.1.0-1.fc16.noarch.rpm
# wget http://mirror.bytemark.co.uk/fedora/linux/releases/16/Everything/x86_64/os/Packages/fedora-release-rawhide-16-1.noarch.rpm
# rpm -Uvh fedora-release*16*.rpm
- Step 5: Perform the actual upgrade.
# yum update
Sometimes even after removing all orphan packages and 3rd party packages, the last step will still fail with unresolvable dependencies. This might be the case if an RPM was purged from latest Fedora and thus not having an upgrade path. When this happens, it is usually sufficient to just remove the obsolete package and retry the upgrade.
One significant change between Fedora 15 and 16 that is important for the upgrade process, is the switch from Grub1 to Grub2. While the live YUM upgrade will result in grub2 getting installed, it does not update your bootsector. So there are two post-upgrade tasks that must be done before reboot.
- Generate the master grub2 config file
# grub2-mkconfig > /etc/grub2.cfg
- Install grub2 into the boot sector
# grub2-install /dev/sda
As mentioned earlier, on one of the machines I had software RAID configured, so I wanted grub2 installed on both disks
# grub2-install /dev/sda
# grub2-install /dev/sdb
After rebooting into the new kernel, I once again run ‘package-cleanup –orphans’ to ensure find any obsolete packages which can be killed off.
That’s all there was to it. Where I have followed these instructions I have a 100% success rate in live yum upgrades. The only upgrade which failed was the one where I forgot to generate the grub2 config file before rebooting :-)
I have written before about what a great benefit GObject Introspection is, by removing the need to write dynamic language bindings for C libraries. When I ported GTK-VNC to optionally build with GTK3, I did not bother to update the previous manually created Python binding. Instead application developers are now instructed to use GObject Introspection if they ever want to use GTK-VNC from non-C languages. As a nice demo of the capabilities I have written the bare minimum “Hello World” like example for GTK-VNC in Perl, Python and JavaScript. The only significant difference between these examples is syntax for actually importing a particular library. The Perl binding is the most verbose for importing libraries, which is a surprise, since Perl is normally a very concise language. Hopefully they will invent a more concise syntax for importing soon.
Perl “hello world” VNC client
#!/usr/bin/perl
use Gtk3 -init;
Glib::Object::Introspection->setup(basename => 'GtkVnc', version => '2.0', package => 'GtkVnc');
Glib::Object::Introspection->setup(basename => 'GVnc', version => '1.0', package => 'GVnc');
GVnc::util_set_debug(1);
my $win = Gtk3::Window->new ('toplevel');
my $dpy = GtkVnc::Display->new();
$win->set_title("GTK-VNC with Perl");
$win->add($dpy);
$dpy->open_host("localhost", "5900");
$win->show_all;
Gtk3::main;
Python “hello world” VNC client
#!/usr/bin/python
from gi.repository import Gtk;
from gi.repository import GVnc;
from gi.repository import GtkVnc;
GVnc.util_set_debug(True)
win = Gtk.Window()
dpy = GtkVnc.Display()
win.set_title("GTK-VNC with Python")
win.add(dpy)
dpy.open_host("localhost", "5900")
win.show_all()
Gtk.main()
JavaScript “hello world” VNC client
#!/usr/bin/gjs
const Vnc = imports.gi.GtkVnc;
const GVnc = imports.gi.GVnc;
const Gtk = imports.gi.Gtk;
Gtk.init(0, null);
GVnc.util_set_debug(true);
var win = new Gtk.Window();
var dpy = new Vnc.Display();
win.set_title("GTK-VNC with JavaScript");
win.add(dpy);
dpy.open_host("localhost", "5900");
win.show_all();
Gtk.main();
In the next week or two, we will be releasing libvirt 0.9.7, which comes with a new URI alias feature, which will be particularly useful for users of tools like virsh and virt-install.
In the same way that SSH allows you to setup hostname aliases in $HOME/.ssh/config, libvirt will now allow you to setup URI aliases in $HOME/.libvirt/libvirt.conf (if you are running unprivileged) or /etc/libvirt/libvirt.conf (if you are running as root). NB do not confuse this file with libvirtd.conf which is a server side libvirtd daemon config file.
The new libvirt.conf configuration file will start with a single ‘uri_aliases’ parameter, which takes a list of entries. Each entry is an alias map in the format “ALIAS=URI”. To avoid potential overlap with otherwise valid URIs, the ALIAS part is restricted to the characters a-z, A-Z, 0-9, -, _. The config file format is fairly self-explanatory if you look at an example:
$ cat $HOME/.libvirt/libvirt.conf
uri_aliases = [
"hail=qemu+ssh://root@hail.cloud.example.com/system",
"sleet=qemu+tls://sleet.cloud.example.com/system",
]
Once this config has been created, I can then replace any commands like
$ virsh -c qemu+ssh://root@hail.cloud.example.com/system list
$ virt-install -c qemu+tls://sleet.cloud.example.com/system ...some args...
with much simpler commands like
$ virsh -c hail list
$ virt-install -c sleet ...some arg...
Application that use libvirt do not have to do anything special to support URI aliases, as they are automatically handled by the virConnectOpen family of APIs. If, however, an application wishes to prevent the use of URI aliases for some reason, it can do so by passing VIR_CONNECT_NO_ALIASES constant as a flag to the virConnectOpenAuth API.
For further documentation on libvirt URIs consult the URI reference and the remote driver reference pages on the libvirt website.
Earlier today I was asked how one would go about debugging early startup of KVM under GDB, when launched by libvirtd. It was not possible to simply attach to KVM after it had been launched by libvirtd, since that was too late. In addition running the same KVM command outside libvirt did not exhibit the problem that was being investigated.
Fortunately, with a little cleverness, it is actually possible to debug a KVM guest launched by libvirtd, right from the start. The key is to combine a couple of breakpoints with use of follow-fork-mode. When libvirtd starts up a KVM guest, it runs QEMU a couple of times in order to detect which command line arguments are supported. This means the follow-fork-mode setting cannot be changed too early, otherwise GDB will end up following the wrong process.
I happen to know that there is only one place in the libvirt code which calls virCommandSetPreExecHook, and that is immediately before launching the real QEMU process. A nice thing about GDB is that when following forked/exec’d children, it will apply any existing breakpoints in the child, even if it is a new binary. So a break point set on ‘main’, while still in libvirtd will happily catch ‘main’ in the QEMU process. The only remaining problem is that if QEMU does not setup and activate the monitor quickly enough, libvirtd will try to kill it off again. Fortunately GDB lets you ignore SIGTERM, and even SIGKILL :-)
The start of the trick is this:
# pgrep libvirtd
12345
# gdb
(gdb) attach 12345
(gdb) break virCommandSetPreExecHook
(gdb) cont
Now in a separate shell
# virsh start $GUESTNAME
Back in the GDB shell the breakpoint should have triggered, allowing the trick to be finished:
(gdb) break main
(gdb) handle SIGKILL nopass noprint nostop
Signal Stop Print Pass to program Description
SIGKILL No No No Killed
(gdb) handle SIGTERM nopass noprint nostop
Signal Stop Print Pass to program Description
SIGTERM No No No Terminated
(gdb) set follow-fork-mode child
(gdb) cont
process 3020 is executing new program: /usr/bin/qemu-kvm
[Thread debugging using libthread_db enabled]
[Switching to Thread 0x7f2a4064c700 (LWP 3020)]
Breakpoint 2, main (argc=38, argv=0x7fff71f85af8, envp=0x7fff71f85c30)
at /usr/src/debug/qemu-kvm-0.14.0/vl.c:1968
1968 {
(gdb)
Bingo, you can now debug QEMU startup at your leisure