Using QCow2 disk encryption with libvirt in Fedora 12

Without muchany fan-fare we slipped a significant new feature into libvirt in Fedora 12, namely the ability to encrypt a virtual machine’s disks. Ordinarily this would have been widely publicised as a Fedora 12 release feature, but the code arrived into libvirt long after the Fedora feature writeup deadline. Before continuing, special thanks are due to Miloslav Trmač who wrote nearly all of this encryption/secrets management code for libvirt!

Why might you want to encrypt a guest’s disk from the host, rather than using the guest OS’s own block encryption capabilities (eg the block encryption support in anaconda) ? There’s a couple of reasons actually…

1. The host is using a network filesystem (like NFS/GFS) for storing guest disks, a guarantee is required that no one can snoop on any guest data, regardless of guest OS configuration.
2. Guest OS can boot without needing any password prompts, since libvirt can supply the decryption key directly to QEMU on the host side when launching the guest.
3. Allows integration with a key management server. Libvirt provides APIs for setting the keys associated with a guest’s disks. An key management service can use these APIs to set/clear the keys for each host to match the list of guests it is intended to run.

There are probably more advantages to managing encryption on the virtualization host but I’m not going to try to think about them now. Instead the rest of this posting will give a short overview of how to use the new encryption capabilities

Secret management

There are many objects managed by libvirt which can conceivably use/require encryption secrets. It is also desirable that libvirt be able to integrate with external key management services, rather than always having to store secrets itself. For these two reasons, rather than directly set encryption secrets against virtual machines, or virtual disks, libvirt introduces a simple set of “secrets” management APIs. The first step in using disk encryption is thus to define a new secret in libvirt. In keeping with all other libvirt objects, a secret is defined by a short XML document

# cat demo-secret.xml
<secret ephemeral='no' private='no'>
  <uuid>0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f</uuid>
  <usage type='volume'>
    <volume>/home/berrange/VirtualMachines/demo.qcow2</volume>
  </usage>
</secret>

The “ephemeral” attribute controls whether libvirt will store a persistent copy of the secret on disk. If you had an external key management server talking to libvirt you would typically set this to ‘yes’, so that keys were never written to disk on individual virtualization hosts. Most people though will want to set this to ‘no’, so that libvirt stores the secret, otherwise you’ll loose all your keys when you reboot which probably isn’t what you want ! When running against a privileged libvirtd instance (eg with the qemu:///system URI), secrets are stored in /etc/libvirt/secrets, while when running unprivileged (qemu:///session), secrets are stored in $HOME/.libvirt/secrets.

The “private” attribute controls whether you can ask libvirt to give you the value associated with a secret. When it is ‘yes’, secrets are “write only”, once you’ve set the value, libvirt will refuse to tell you what it is. Again this is useful if you are using a key management server, because it allows it to load a secret into libvirt in order to start a guest, without allowing anyone else who is connected to libvirt to actually see what its value is.

The “uuid” is simply a unique identifier for the secret, when defining a secret this can be left out and it will be auto-generated.

Finally the “usage” element indicates what object the secret will be used this. This is not technically required, but when you have many hundreds of secrets defined, it is useful to know what objects they’re associated with, so you can easily purge secrets which are no longer used/required.

Having created the XML snippet for a secret as above, the first step is thus to load the secret definition into libvirt. If you are familiar with libvirt API/command naming conventions, you won’t be surprised to find out that this is done using the ‘virsh secret-define’ command

  # virsh secret-define demo-secret.xml
  Secret 1a81f5b2-8403-7b23-c8d6-21ccc2f80d6f created

Notice how we have not actually set the value of the secret anywhere, merely defined metadata. We explicitly choose not to include the secret’s value in the XML, since that would increase the risk of it being accidentally exposed in log files / bug reports / etc, etc. Thus once the secret has been defined, it is neccessary to set its value. There is a special virsh command for doing this called ‘virsh secret-set-value’, which takes two parameters, the UUID of the secret and then the value in base64. If you’re one of these people who can’t compute base64 in your head, then there’s of course the useful ‘base64’ command line tool

  # MYSECRET=`echo "open seseme" | base64`
  # virsh secret-set-value 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f $MYSECRET
  Secret value set

There are a few other virsh commands available, for managing secrets, but those two are the key ones you need to know in order to provision a new guest using encrypted disks. See the ‘virsh help’ output for the other commands

Virtual disks

Being able to define secrets isn’t much fun if you can’t then put those secrets to use. The first interesting task is probably to create an encrypted disk. At this point in time, libvirt’s storage APIs only support encryption of the qcow1, or qcow2 formats. It would be very desirable to support dm-crypt too, but that’s an outstanding feature request for someone else to implement.
I’ve got a directory based storage pool configured in my libvirt host which points to $HOME/VirtualMachines, defined with the following XML

# virsh pool-dumpxml VirtualMachines
<pool type='dir'>
  <name>VirtualMachines</name>
  <source>
  </source>
  <target>
    <path>/home/berrange/VirtualMachines</path>
  </target>
</pool>

To create a encrypted volume within this pool it is neccessary to provide a short XML document describing the volume, what format it shoould be, how large it should be, and what secret it should use for encryption

# cat demo-disk.xml
<volume>
  <name>demo.qcow2</name>
  <capacity>5368709120</capacity>
  <target>
    <format type='qcow2'/>
    <encryption format='qcow'>
      <secret type='passphrase' uuid='0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f'/>
    </encryption>
  </target>
</volume>

Notice that we set the volume format to ‘qcow2’ since that is the type of disk we want to create. The XML then has the newly introduced “encryption” element which says that the volume should be encrypted using the ‘qcow’ encryption method (this is the same method for both qcow1, and qcow2 format disks). Finally it indicates that the ‘qcow’ encryption “passphrase” is provided by the secret with UUID 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f. The disk can now be created using the “virsh vol-create” command, for example,

# virsh vol-create VirtualMachines demo-disk.xml
Vol demo.qcow2 created from demo-disk.xml

An oddity of the qcow2 disk format is that it doesn’t actually need to have the encryption passphrase at the time it creates the volume, since it only encrypts its data, not metadata. libvirt still requires you set a secret in the XML at time of creation though, because you never know when qcow may change its requirements, or when we might use a format that does require the passphrase at time of creation. Oh and if you are cloning an existing volume, you would actually need the passphrase straight away to copy the data.

Virtual Machines

Now that we have created an encrypted QCow2 disk, it would be nice to use that disk with a virtual machine. In this example I’ve downloaded the PXE boot initrd.img and vmlinuz files for Fedora 12 and intend to use them to create a brand new virtual machine. So my guest XML configuration will be setup to use a kernel+initrd to boot, and have a single disk, the encrypted qcow file we just created. The key new feature in the XML format introduced here is again the “encryption” element within the “disk” element description. This is used to indicate what encryption method is used for this disk (again it is the ‘qcow’ method), and then associates the qcow decryption ‘passphrase’ with the secret we defined earlier

# cat demo-guest.xml
<domain type='qemu'>
  <name>demo</name>
  <memory>500200</memory>
  <vcpu>4</vcpu>
  <os>
    <type arch='i686' machine='pc'>hvm</type>
    <kernel>/home/berrange/vmlinuz-PAE</kernel>
    <initrd>/home/berrange/initrd-PAE.img</initrd>
    <boot dev='hd'/>
  </os>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/berrange/VirtualMachines/demo.qcow2'/>
      <target dev='hda' bus='ide'/>
      <encryption format='qcow'>
        <secret type='passphrase' uuid='0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f'/>
      </encryption>
    </disk>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
  </devices>
</domain>

With that XML config written, it is a simple matter to define a new guest, and then start it

# virsh define demo-guest.xml
Domain demo defined from demo-guest.xml

# virsh start demo
Domain demo started

# virt-viewer demo

If everything has gone to plan upto this point, the guest will boot off the kernel/initrd, hopefully taking you into anaconda. Everything written to the guest disk will now be encrypted using the secrets defined.

Future work

You’ll have noticed that all these examples are using the low level virsh command. Great if you are the king of shell scripting, not so great if you want something friendly to use. So of course this new encryption functionality needs to be integrated into virt-install, and virt-manager. They should both allow you say that you want an encrypted guest, prompt you for a passphrase and then setup all the secrets automatically from there.

The libvirtd daemon has the ability to store secrets and their values persistently on disk, but this is not really secure, since the secrets are stored in unencrypted base64 format ! Clearly the next step here is for libvirtd to at the very least have the option of using gpg to encrypt the base64 files. The problem is that this then introduces a boot-strapping problem – what key does libvirt use for gpg ! This is a familiar problem to anyone who’s ever had to setup apache with SSL and wondered how to give apache the key to decrypt its SSL server key upon host startup.

As mentioned earlier on, the libvirt secrets management public API was designed to be easy to integrate with external key management services. For a desktop virtualization application like virt-manager there is an opportunity to integrate with gnome-keyring (or equivalent). When defining secrets in libvirt, virt-manager would mark them all as ephemeral so that libvirt never stored them in itself. At the time of starting a guest, virt-manager would query gnome-keyring for the disk decryption keys, and pass them onto libvirt. This would ensure no one could ever run your guests unless they are able to login & authenticate to your gnome-keyring service. A server virtualization application like oVirt could do much the same, perhaps storing keys in FreeIPA (if it had such a capability).

Being restricted to qcow2 disk formats isn’t all that nice because qcow2 isn’t the fastest virtual disk format to start off with, and adding encryption doesn’t improve matters. Many people, particularly in server virtualization environments, prefer to use LVM or raw block devices (SCSI/iSCSI). There are hacks which let you tell QEMU to write to the block device in qcow2 format, but they make me feel rather dirty. The kernel already comes with a generic block device encryption capability in the form of ‘dm-crypt’. libvirt really ought to support creation of encrypted block devices using dm-crypt.