The OpenStack Nova project has the ability to inject files into a guest filesystem immediately prior to booting the virtual machine instance. Historically the way it did this was to setup either a loop or NBD device on the host OS and then mount the guest filesystem directly on the host OS. One of the high priority tasks for Red Hat engineers when we became involved in OpenStack was to integrate libguestfs FUSE into Nova to replace the use of loop back + NBD devices, and then subsequently refactor Nova to introduce a VFS layer which enables use of the native libguestfs API to avoid any interaction with the host filesystem at all.
There has already been a vulnerability in the Nova code which allowed a malicious user to inject files to arbitrary locations in the host filesystem. This has of course been fixed, but even so mounting guest disk images on the host OS should still be considered very bad practice. The libguestfs manual describes the remaining risk quite well:
When you mount a filesystem, mistakes in the kernel filesystem (VFS) can be escalated into exploits by attackers creating a malicious filesystem. These exploits are very severe for two reasons. Firstly there are very many filesystem drivers in the kernel, and many of them are infrequently used and not much developer attention has been paid to the code. Linux userspace helps potential crackers by detecting the filesystem type and automatically choosing the right VFS driver, even if that filesystem type is unexpected. Secondly, a kernel-level exploit is like a local root exploit (worse in some ways), giving immediate and total access to the system right down to the hardware level
Libguestfs provides protection against this risk by creating a virtual machine inside which all guest filesystem manipulations are performed. Thus even if the guest kernel gets compromised by a VFS flaw, the attacker then still has to break out of the KVM virtual machine and its sVirt confinement to stand a chance of compromising the host OS. Some people have doubted the severity of this kernel VFS driver risk in the past, but an article posted on LWN today should serve reinforce the fact that libguestfs is right to be paranoid. The article highlights two kernel filesystem vulnerabilities (one in ext4 which is enabled in pretty much all Linux hosts) which left hosts vulnerable for as long as 3 years in some cases:
- CVE-2009-4307: a divide-by-zero crash in the ext4 filesystem code. Causing this oops requires convincing the user to mount a specially-crafted ext4 filesystem image
- CVE-2009-4020: a buffer overflow in the HFS+ filesystem exploitable, once again, by convincing a user to mount a specially-crafted filesystem image on the target system.
If the user has access to an OpenStack deployment which is not using libguestfs for file injection, then “convincing a user to mount a specially crafted filesystem” merely requires them to upload their evil filesystem image to glance and then request Nova to boot it.
Anyone deploying OpenStack with file injection enabled, is strongly advised to make sure libguestfs is installed to avoid any direct exposure of the host OS kernel to untrusted guest images.
While I picked on OpenStack as a convenient way to illustrate the problem here, it is not unique to OpenStack. Far too frequently I find documentation relating to virtualization that suggests people mount untrusted disk images directly on their OS. Based on their documented features I’m confident that a number of public virtual machine hosting companies will be mounting untrusted user disk images on their virtualization hosts, likely without using libguestfs for protection.
When launching a virtual machine, Nova has the ability to inject various files into the disk image immediately prior to boot up. This is used to perform the following setup operations:
- Add an authorized SSH key for the root account
- Configure init to reset SELinux labelling on /root/.ssh
- Set the login password for the root account
- Copy data into a number of user specified files
- Create the meta.js file
- Configure network interfaces in the guest
This file injection is handled by the code in the nova.virt.disk.api module. The code which does the actual injection is designed around the assumption that the filesystem in the guest image can be mapped into a location in the host filesystem. There are a number of ways this can be done, so Nova has a pluggable API for mounting guest images in the host, defined by the nova.virt.disk.mount module, with the following implementations:
- Loop – Use losetup to create a loop device. Then use kpartx to map the partitions within the device, and finally mount the designated partition. Alternatively on new enough kernels the loop device’s builtin partition support is used instead of kpartx.
- NBD – Use qemu-nbd to run a NBD server and attach with the kernel NBD client to expose a device. Then mapping partitions is handled as per Loop module
- GuestFS – Use libguestfs to inspect the image and setup a FUSE mount for all partitions or logical volumes inside the image.
The Loop module can only handle Raw format files, while the NBD module can handle any format that QEMU supports. While they have the ability to access partitions, the code handling this is very dumb. It requires the Nova global ‘libvirt_inject_partition’ config parameter to specify which partition number to inject. The result is that every image you upload to glance must be partitioned in exactly the same way. Much better would be if it used a metadata parameter associated with the image. The GuestFS module is much more advanced and inspects the guest OS to figure out arbitrarily partitioned images and even LVM based images.
Nova has a “img_handlers” configuration parameter which defines the order in which the 3 mount modules above are to be tried. It tries to mount the image with each one in turn, until one suceeds. This is quite crude code really – it has already been hacked to avoid trying the Loop module if Nova knows it is using QCow2. It has to be changed by the Nova admin if they’re using LXC, otherwise you can end up using KVM with LXC guests which is probably not what you want. The try-and-fallback paradigm also has the undesirable behaviour of masking errors that you would really rather consider fatal to the boot process.
As mentioned earlier, the file injection code uses the mount modules to map the guest image filesystem into a temporary directory in the host (such as /tmp/openstack-XXXXXX). It then runs various commands like chmod, chown, mkdir, tee, etc to manipulate files in the guest image. Of course Nova runs as an unprivileged user, and the guest files to be changed are typically owned as root. This means all the file injection commands need to run via Nova’s rootwrap utility to gain root privileges. Needless to say, this has the undesirable consequence that the code injecting files into a guest image in fact has privileges that allow it to write to arbitrary areas of the host filesystem. One mistake in handling symlinks and you have the potential for a carefully crafted guest image to result in compromise of the host OS. It should come as little surprise that this has already resulted in a security vulnerability / CVE against Nova.
The solution to this class of security problems is to decouple the file injection code from the host filesystem. This can be done by introducing a “VFS” (Virtual File System) interface which defines a formal API for the various logical operations that need to be performed on a guest filesystem. With that it is possible to provide an implementation that uses the libguestfs native python API, rather than FUSE mounts. As well as being inherently more secure, avoiding the FUSE layer will improve performance, and allow Nova to utilize libguestfs APIs that don’t map into FUSE, such as its Augeas support for parsing config files. Nova still needs to work in scenarios where libguestfs is not available though, so a second implementation of the VFS APIs will be required based on the existing Loop/Nbd device mount approach. The security of the non-libguestfs support has not changed with this refactoring work, but de-coupling the file injection code from the host filesystem does make it easier to write unit tests for this code. The file injection code can be tested by mocking out the VFS layer, while the VFS implementations can be tested by mocking out the libguestfs or command execution APIs.
Incidentally if you’re wondering why Libguestfs does all its work inside a KVM appliance, its man page describes the security issues this approach protects against vs just directly mounting guest images on the host
sVirt has been available in the libvirt KVM driver for a few years now, both for SELinux and more recently for AppArmour. When using it with SELinux there has been a choice of two different configurations
- Dynamic configuration
- libvirt takes the default base label (“system_u:system_r:svirt_t:s0”), generates a unique MCS label for the guest (“c123,c465”) and combines them to form the complete security label for the virtual machine process. libvirt takes the same MCS label and combines it with the default image base label (“system_u:system_r:svirt_image_t:s0”) to form the image label. libvirt will then automatically apply the image label to all host OS files that the VM is required to access. These can be disk images, disk devices, PCI devices (we label the corresponding sysfs files), USB devices (we label the /dev/bus/usb files), kernel/initrd files, and a few more things. When the VM shuts down again, we reverse the labelling. This mode was originally intended for general usage where the management application is not aware of the existence of sVirt.
- Static configuration
- The guest XML provides the full security label, including the MCS part. libvirt simply assigns this security label to the virtual machine process without trying to alter/interpret it any further. libvirt does not change the labels of any files on disk. The administrator/application using libvirt, is expected to have done all the resource file labelling ahead of time. This mode was originally intended for locked down MLS environments, where even libvirtd itself is not trusted to perform relabelling
These two configurations have worked well enough for the two uses cases they were designed to satisfy. As sVirt has become an accepted part of the libvirt/KVM ecosystem, application developers have started wanting todo more advances things which are currently harder than they should be. In particular some applications want to have full control over the security label generation (eg to ensure cluster-wide unique labels, instead of per-host uniqueness), but still want libvirt to take care of resource relabelling. This is sort of a hybrid between our static & dynamic configuration. Other applications would like to be able to choose a different base label (“system_u:system_r:svirt_custom_t:s0”) but still have libvirt assign the MCS suffix and perform relabelling. This is another variant on dynamic labelling. To satisfy these use cases we have extended the syntax for sVirt labelling in recent libvirt. The “seclabel” element gained a ‘relabel’ attribute to control whether resource relabelling is attempted. A new “baselabel” element was introduced to override the default base security label in dynamic mode. So there are now 4 possible styles of configuration:
-
Dynamic configuration (the default out of the box usage)
<seclabel type='dynamic' model='selinux' relabel='yes'>
<label>system_u:system_r:svirt_t:s0:c192,c392</label> (output only element)
<imagelabel>system_u:object_r:svirt_image_t:s0:c192,c392</imagelabel> (output only element)
</seclabel>
-
Dynamic configuration, with base label
<seclabel type='dynamic' model='selinux' relabel='yes'>
<baselabel>system_u:system_r:svirt_custom_t:s0</baselabel>
<label>system_u:system_r:svirt_custom_t:s0:c192,c392</label> (output only element)
<imagelabel>system_u:object_r:svirt_image_t:s0:c192,c392</imagelabel> (output only element)
</seclabel>
-
Static configuration, no resource labelling (primarily for MLS/strictly controlled environments)
<seclabel type='static' model='selinux' relabel='no'>
<label>system_u:system_r:svirt_custom_t:s0:c192,c392</label>
</seclabel>
-
Static configuration, with dynamic resource labelling
<seclabel type='static' model='selinux' relabel='yes'>
<label>system_u:system_r:svirt_custom_t:s0:c192,c392</label>
</seclabel>
This blog post isn’t exactly “news” to many readers, but I feel it is worth reminding people of the risks of letting their browser remember form data history
Being reasonably paranoid about security / privacy, one of the first things I do when starting firefox on a freshly installed machine is to go to the preferences and change the history settings so that it does not ever remember any form data. Unfortunately on my most recently installed laptop I had forgotten to do this, for inexplicable reasons, so firefox was happily remembering form data. Before clearing the data out, I decided to take a look at just what firefox had remembered over the past couple of months.
- Card numbers: Every credit and debit card number
- CVV/CVC: The 3 digit verification codes from the back of several cards
- Social security / national insurance numbers
- Bank accounts: Several bank account numbers and sort codes
- Addresses: Home address, amongst others
- Date of birth: Handy in combination with the above data
- User names: login name for many many websites, even those where I said not to remember the actual passwords
- Phone numbers: my home and mobile phone numbers, amongst others
- Answers to several “security questions” used to reset passwords on sites
You might wonder where/how firefox keeps all this sensitive data ? In a sqlite database with no encryption in your firefox profile ($HOME/.mozilla/firefox/XXXXXXXX.default/formhistory.sqlite). You can look at what is stored using the sqlite3 command line tool:
# sqlite3 formhistory.sqlite
sqlite> select fieldname, value from moz_formhistory order by value;
Or, if you trust the plugin author, you can install the firefox “Form History Control” addon which lets you browse all the data from a nice UI.
While I do have my laptop disk running with dm-crypt block device encryption, I still have a golden rule that no sensitive data is stored in cleartext in files since, while the OS is running, disk encryption is no real barrier malicious programs that find their way on to the machine. If it is a work provided machine, the data may also be finding its way into the corporate backup system & who knows if that is encrypted well enough. Any files with sensitive data have additional GPG encryption and really sensitive data will be on a separate encrypted USB stick, again with extra encryption for the files themselves. Keeping form history in an unencrypted sqlite database in my laptop home directory clearly violates my security goals. Why would a $EVIL criminal go the trouble of creating a virus which installs a keystroke logger in someone’s browser, when they can get a veritable goldmine of data by simply taking a copy of the formhistory.sqlite file.
Having known about this problem since 2003, a “fix” was applied in 2010 for firefox 4 which attempts to identify credit card numbers in forms and not store them in the form history. Great, now what about all the other data it is storing some of which is just as sensitive as credit card numbers, if not more so ? If credit card details get mis-used, liability is usually on the credit card company, but not so for social security numbers, bank account numbers, etc…
Needless to say, I have fixed my preferences in firefox so that it doesn’t remember form data, and deleted the formhistory.sqlite file to remove any existing stored data it had. I have to wonder how many users of firefox are out there running with the default history settings, without any realization that a great deal of their sensitive personal data is being silently stored in plain sight for any malicious virus/trojan to capture. The annoying thing is form history can be really useful for certain websites where there’s alot of mundane non-sensitive data that needs entering frequently. As the last commenter on the bug 188285 proposed, it would be desirable if firefox prompted before remembering form data with options “Yes, Not at this time, Never for this site”. It would also be desirable for data in the formhistory.sqlite file to be encrypted as firefox can already do when remembering passwords.
To stop firefox recording your form history:
- Select menu Edit -> Preferences
- Navigate to tab Privacy
- Change Firefox will: Remember history to Firefox will: Use custom settings for history
- Unselect Remember search and form history
To clear your existing form history
- Select menu Tools -> Clear Recent History
- Change Time range to clear: Last Hour to Time range to clear: Everything
- Expand details
- Make sure Form History is selected (optionally unselect other bits you don’t want cleared)
- Press Clear Now
If you are super paranoid, then shutdown firefox and delete the file $HOME/.mozilla/firefox/XXXXXXXX.default/formhistory.sqlite and then start firefox again