Secure remote management for Xen/QEMU/KVM virtualization

One of the most frequently requested features for both libvirt and virt-manager is the ability to manage virtual machines remotely. When considering this problem, one immediately thinks of the security implications – most of the communications being used for local management have minimal security. For example, with Xen the XenD offers access over TCP with either XML-RPC or a custom SEXPR protocol. In both cases though, the communication channel is unencrypted and the authentication is minimal. Even though the new XML-RPC protocol does offer authentication against PAM this is totally unsuitable for use off-host because the passwords would be transmitted in clear text :-( The core management APIs are not the only area using a cleartext communication channel – the migration of virtual machines between hosts is also done in the clear – not great if the OS running in the VM has any sensitive data in memory. The virtual console capability is based on the VNC remote framebuffer protocol – again in everything is being transmitted in cleartext.

When raising the question of secure remote management, a frequently suggested solution is to just tunnel everything over SSH. While this would obviously address the issue of wire level encryption just fine, it is less desirable when considering the issues of authentication, access control, and key management. To manage virtual machines using SSH tunnelling one would have to create user accounts on each Dom0 for anyone who can manage VMs. Integrity / security of the Dom0 hosts is critical in a virtualized environment because any single host is running many VMs, so giving people shell access with SSH is not at all desirable. It is possible to restrict a users’ access by only allowing authentication with public keys, and using magic in the authorized_keys file to whitelist a set of commands that can be run.
That said, in a data center with 1000’s hosts, one certainly does not want to be maintaining 10’s of 1000’s of authorized_keys files – just think of the work involved in revoking access to an admin who resigns / is fired. Authentication via SSH is at a rather coarse level – the user can either log into the host, or not – there’s no facility for saying ‘user A can manage virtual machines B and C, but not virtual machine D or E’. So there would need to be a second level of access control on top of that provided by SSH. Thus as well as managing 10’s of 1000’s of authorized_keys files, there’s the additional management hassle of users in the management protocol itself. While on the subject of SSH, when logging into a machine for the first time, how many people actually verify that the host key is correct ? Not many I suspect. I know I usually just type ‘yes’ every time.

So IMHO, using SSH as the foundation for secure remote management of virtual machines in a large data center is not really practical. The answer to these problems is to utilize TLS (the successor to SSL). Contrary to popular belief TLS is not just used for web servers/browsers / HTTPS – it is a general purpose protocol which can be layered onto pretty many any network communication protocol with surprisingly little effort. So, how does it deal with the issues identified above ? WRT to the question of host key checking, the key is that TLS uses X509 certificates and with that comes the concept of a Certificate Authority. Apriori, one only has to trust the certificate authority (CA). Thus, when connecting to a server for the first time, one validates the certificate presented by checking that it was signed by the trusted CA, and compare the server hostname against the ‘common name’ embedded in the signed certificate. Obviously if one is communicating directly using the virtualization management server’s native protocol there is no need to give any shell access to the Dom0 host. Each client connecting to the server would have their own ‘client certificate’ which is signed by the certificate authority. Upon accepting a client connection, the server validates the CA signature on the client’s certificate, and also checks the ‘certificate revokation list’ (CRL) published by the CA. Assuming the CRL is pushed to each Dom0 host on a periodic basis, there is only one place the admin needs to go to revoke a user’s access to all hosts – the CA admin site. Fine grained access control per VM, can be keyed off a whitelist of client certificates, either based on the ‘common name’ or ‘fingerprint’ fields in the cert.

Thus my long term vision for remote management is to TLS enable all the network protocols involved. As a first step Rich Jones is working on a TLS enabled daemon for libvirt which will allow apps to securely manage any virtualization driver supported by libvirt.
This is the core first step in getting virt-manager operating remotely. The obvious next step in this process is to enable TLS in the VNC protocol. The main open source VNC server, RealVNC, does not offer any standard TLS support – they restrict this stuff to the commercial version. Over the years several people have offered patches to add TLS to VNC, but they’ve never been accepted in the RealVNC codebase, which has unfortunately led to a fork – VeNCrypt. Thus for QEMU and the VNC daemon used by Xen paravirt framebuffer, I’d anticipate implementing the VNC protocol extensions defined by VeNCrypt to add TLS support. I’m currently hacking on a GTK VNC viewer widget to replace the current VNC viewer in virt-manager, with the express purpose of supporting TLS. The final stage of the plan will involve adapting the Xen and QEMU migration protocols to be layered over a TLS connection. There are many interesting questions / challenges to be worked out along the way, and certainly a hell of alot of coding to be done. The payoff at the end will be well worth it though.

As I mentioned earlier, Rich Jones is working on the libvirt management protocol, and I’m attacking the VNC client side of things. That still leaves a lot of hacking to be done – the server side VNC TLS impl in QEMU and the Xen framebuffer daemon are both sizeable chunks of work, and I’m not aware of anyone tackling the migration stuff either. So there’s plenty of coding work for people who are interested in this….