Attaching libvirt to an externally launched KVM instance

Traditionally, if you have manually launched a QEMU process, either as root, or as your own user, it will not be visible to/from libvirt’s QEMU driver. This is an intentional design decision because given an arbitrary QEMU process, it is very hard to determine what its current or original configuration is/was. Without knowing QEMU’s configuration, it is hard to reliably perform further operations against the guest. In general, this limitation has not proved a serious burden to users of libvirt, since there are variety of ways to launch new guests directly with libvirt whether graphical (virt-manager) or command line driven (virt-install).

There are always exceptions to the rule, though, and one group of users who have found this a problem is the QEMU/KVM developer community itself. During the course of developing & testing QEMU code, they often have need to quickly launch QEMU processes with a very specific set of command line arguments. This is hard to do with libvirt, since when there is a choice of which command line parameters to use for a feature, libvirt will pick one according to some predetermined rule. As an example, if you want to test something related to the old style ‘-drive’ parameters with QEMU, and libvirt is using the new style ‘-device + -drive’ parameters, you are out of luck & will not be able to force libvirt to use the old syntax. There are other features of libvirt that QEMU developers may well still want to take advantage of though like virt-top or virt-viewer. Thus it is desirable to have a way to launch QEMU with arbitrary command line arguments, but still use libvirt.

A little while ago we did introduce support for adding extra QEMU specific command line arguments in the guest XML configuration, using a separate namespace. This is not entirely sufficient, or satisfactory for the particular scenario outlined above. For this reason, we’ve now introduced a new QEMU-specific API into libvirt that allows the QEMU driver to attach to an externally launched QEMU process. This API is not in the main library, but rather in the separate libvirt-qemu.so library. Use of this library by applications is strongly discouraged and many distros will not supports its use in production deployments. It is intended primarily for developer / troubleshooting scenarios. This QEMU specific command is also exposed in virsh, via a QEMU specific command ‘qemu-attach‘. So now it is possible for the QEMU developers to launch a QEMU process and connect it to libvirt

$ qemu-kvm \
  -cdrom ~/demo.iso \
  -monitor unix:/tmp/myexternalguest,server,nowait \
  -name myexternalguest \
  -uuid cece4f9f-dff0-575d-0e8e-01fe380f12ea  \
  -vnc 127.0.0.1:1 &
$ QEMUPID=$!
$ virsh qemu-attach $QEMUPID
Domain myexternalguest attached to pid 14725

Once attached, most of the normal libvirt commands and tools will be able to at least see the guest. For example, query its status

$ virsh list
 Id Name                 State
----------------------------------
  1 myexternalguest      running

$ virsh dominfo myexternalguest
Id:             1
Name:           myexternalguest
UUID:           cece4f9f-dff0-575d-0e8e-01fe380f12ea
OS Type:        hvm
State:          running
CPU(s):         1
CPU time:       15.1s
Max memory:     65536 kB
Used memory:    65536 kB
Persistent:     no
Autostart:      disable
Security model: selinux
Security DOI:   0
Security label: unconfined_u:unconfined_r:unconfined_qemu_t:s0-s0:c0.c1023 (permissive)

libvirt reverse engineers an XML configuration for the guest based on the command line arguments it finds for the process in /proc/$PID/cmdline and /proc/$PID/environ. This is using the same code as available via the virsh domxml-from-native command. The important caveat is that the QEMU process being attached to must not have had its configuration modified via the monitor. If that has been done, then the /proc command line will no longer match the current QEMU process state

$ virsh dumpxml myexternalguest
<domain type='kvm' id='1'>
  <name>myexternalguest</name>
  <uuid>cece4f9f-dff0-575d-0e8e-01fe380f12ea</uuid>
  <memory>65536</memory>
  <currentMemory>65536</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='i686' machine='pc-0.14'>hvm</type>
  </os>
  <features>
    <acpi/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='cdrom'>
      <source file='/home/berrange/demo.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' unit='0'/>
    </disk>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5901' autoport='no' listen='127.0.0.1'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='static' model='selinux' relabel='yes'>
    <label>unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023</label>
  </seclabel>
</domain>

Tools like virt-viewer will be able to attach to the guest just fine

$ ./virt-viewer --verbose myexternalguest
Opening connection to libvirt with URI
Guest myexternalguest is running, determining display
Guest myexternalguest has a vnc display
Opening direct TCP connection to display at 127.0.0.1:5901

Finally, you can of course kill the attached process

$ virsh destroy myexternalguest
Domain myexternalguest destroyed

The important caveats when using this feature are

• The guest config must not be modified using monitor commands between the time the QEMU process is started and when it is attached to the libvirt driver
• There must be a monitor socket for the guest using the ‘unix’ protocol as shown in the example above. The socket location does not matter, but it must be a server socket
• It is strongly recommended to specify a name using ‘-name’, otherwise libvirt will auto-assign a name based on the $PID

To re-inforce the earlier point, this feature is ONLY targetted at QEMU developers and other people who want to do ad-hoc testing/troubleshooting of QEMU with precise control over all command line arguments. If things break when using this feature, you get to keep both pieces. To re-inforce this, when attaching to an externally launched guest, it will be marked as tainted which may limit the level of support a distro / vendor provides. Anyone writing serious production quality virtualization applications should NEVER use this feature. It may make babies cry and kick cute kittens.

This feature will be available in the next release of libvirt, currently planned to be version 0.9.4, sometime near to July 31st, 2011