Improving QEMU security part 3: securely passing in credentials
This blog is part 3 of a series I am writing about work I’ve completed over the past few releases to improve QEMU security related features.
When configuring a virtual machine, there are a number of places where QEMU requires some form of security sensitive credentials, typically passwords or encryption keys. Historically QEMU has had no standard approach for getting these credentials from the user, so things have grown in an adhoc manner with predictably awful results. If the VNC server is configured to use basic VNC authentication, then it requires a password to be set. When I first wrote patches to add password auth to QEMU’s VNC server it was clearly not desirable to expose the password on the command line, so when configuring VNC you just request password authentication be enabled using -vnc 0.0.0.0:0,password and then have to use the monitor interface to set the actual password value “change vnc password“. Until a password has been set via the monitor, the VNC server should reject all clients, except that we’ve accidentally broken this in the past, allowing clients when no server password is set :-( The qcow & qcow2 disk image formats support use of AES for encryption (remember this is horribly broken) and so there needs to be a way to provide the decryption password for this. Originally you had to wait for QEMU to prompt for the disk password on the interactive console. This clearly doesn’t work very nicely when QEMU is being managed by libvirt, so we added another monitor command which allows apps to provide the disk password upfront, avoiding the need to prompt. Fast forward a few years and QEMU’s block device layer gained support for various network protocols including iSCSI, RBD, FTP and HTTP(s). All of these potentially require authentication and thus a password needs to be provided to QEMU. The CURL driver for ftp, http(s) simply skipped support for authentication since there was no easy way to provide the passwords securely. Sadly, the iSCSI and RBD drivers simply decided to allow the password to be provided in the command line. Hence the passwords for RBD and iSCSI are visible in plain text in the process listing and in libvirt’s QEMU log files, which often get attached to bug reports, which has resulted in a CVE being filed against libvirt. I had an intention to add support for the LUKS format in the QEMU block layer which will also require passwords to be provided securely to QEMU, and it would be desirable if the x509 keys provided to QEMU could be encrypted too.
Looking at this mess and the likely future requirements, it was clear that QEMU was in desperate need of a standard mechanism for securely receiving credentials from the user / management app (libvirt). There are a variety of channels via which credentials can be theoretically passed to QEMU:
- Command line argument
- Environment variable
- Plain file
- Anonymous pipe
- Monitor command
As mentioned previously, using command line arguments or environment variables is not secure if the credential is passed in plain text, because they are visible in the processing list and log files. It would be possible to create a plain file on disk and write each password to it and use file permissions to ensure only QEMU can read it. Using files is not too bad as long as your host filesystem is on encrypted storage. It has a minor complexity of having to dynamically create files on the fly each time you want to hotplug a new device using a password. Most of these problems can be avoided by using an anonymous pipe, but this is more complicated for end users because for hotplugging devices it would require passing file descriptors over a UNIX socket. Finally the monitor provides a decent secure channel which users / mgmt apps will typically already have open via a UNIX socket. There is a chicken & egg problem with it though, because the credentials are often required at initial QEMU startup when parsing the command line arguments, and the monitor is not available that early.
After considering all the options, it was decided that using plain files and/or anonymous pipes to pass credentials would be the most desirable approach. The qemu_open() method has a convenient feature whereby there is a special path prefix that allows mgmt apps to pass a file descriptor across instead of a regular filename. To enable reuse of existing -object command line argument and object_add monitor commands for definin credentials, the QEMU object model framework (QOM) was used to define a ‘secret’ object class. The ‘secret‘ class has a ‘path‘ property which is the filename containing the credential. For example it could be used
# echo "letmein" > mydisk.pw # $QEMU -object secret,id=sec0,file=mydisk.pw
Having written this, I realized that it would be possible to allow passwords to be provided directly via the command line if we allowed secret data to be encrypted with a master key. The idea would be that when a QEMU process is first started, it gets given a new unique AES key via a file. The credentials for individual disks / servers would be encrypted with the master key and then passed directly on the command line. The benefit of this is that the mgmt app only needs to deal with a single file on disk with a well defined lifetime.
First a master key is generated and saved to a file in base64 format
# openssl rand -base64 32 > master-key.b64
Lets say we have two passwords we need to give to QEMU. We will thus need two initialization vectors
# openssl rand -base64 16 > sec0-iv.b64 # openssl rand -base64 16 > sec1-iv.b64
Each password is now encrypted using the master key and its respective initialization vector
# SEC0=$(printf "letmein" |
openssl enc -aes-256-cbc -a \
-K $(base64 -d master-key.b64 | hexdump -v -e '/1 "%02X"') \
-iv $(base64 -d sec0-iv.b64 | hexdump -v -e '/1 "%02X"'))
# SEC1=$(printf "1234567" |
openssl enc -aes-256-cbc -a \
-K $(base64 -d master-key.b64 | hexdump -v -e '/1 "%02X"') \
-iv $(base64 -d sec1-iv.b64 | hexdump -v -e '/1 "%02X"'))
Finally when QEMU is launched, three secrets are defined, the first gives the master key via a file, and the others provide the two encrypted user passwords
# $QEMU \
-object secret,id=secmaster,format=base64,file=key.b64 \
-object secret,id=sec0,keyid=secmaster,format=base64,\
data=$SECRET,iv=$(<sec0-iv.b64) \
-object secret,id=sec1,keyid=secmaster,format=base64,\
data=$SECRET,iv=$(<sec1-iv.b64) \
...other args using the secrets...
Now we have a way to securely get credentials into QEMU, there just remains the task of associating the secrets with the things in QEMU that need to use them. The TLS credentials object previously added originally required the x509 server key to be provided in an unencrypted PEM file. The tls-creds-x509 object can now gain a new property “passwordid” which provides the ID of a secret object that defines the password to use for decrypting the x509 key.
# $QEMU \
-object secret,id=secmaster,format=base64,file=key.b64 \
-object secret,id=sec0,keyid=secmaster,format=base64,\
data=$SECRET,iv=$(<sec0-iv.b64) \
-object tls-creds-x509,id=tls0,dir=/home/berrange/qemutls,endpoint=server,passwordid=sec0 \
-vnc 0.0.0.0:0,tls-creds=tls0
Aside from adding support for encrypted x509 certificates, the RBD, iSCSI and CURL block drivers in QEMU have all been updated to allow authentication passwords to be provided using the ‘secret‘ object type. Libvirt will shortly be gaining support to use this facility which will address the long standing problem of RBD/ISCSI passwords being visible in clear text in the QEMU process command line arguments. All the enhancements described in this posting have been merged for the forthcoming QEMU 2.6.0 release so will soon be available to users. The corresponding enhancements to libvirt to make use of these features are under active development.
In this blog series: