Faster rebuilds for python virtualenv trees

When developing on OpenStack, it is standard practice for the unit test suites to be run in a python virtualenv. With the Nova project at least, and I suspect most others too, setting up the virtualenv takes a significant amount of time as there are many packages to pull down from PyPI, quite a few of which need compilation too. Any time the local requirements.txt or test-requirements.txt files change it is necessary to the rebuild the virtualenv. This rebuild is an all-or-nothing kind of task, so can be a significant time sink, particularly if you frequently jump between different code branches.

At the OpenStack design summit in Paris, Joe Gordon showed Matt Booth how to setup devpi and wheel to provide a cache of the packages that make up the virtualenv. Not only does it avoid the need to repeatedly download the same packages from pypi each time, but it also avoids the compilation step, since the cache is storing the final installed pieces for each python module. The end result is that it takes 20-30 seconds or less to rebuild a virtualenv instead of many minutes.

After a few painful waits for virtualenvs today, I decided to set it up too I don’t like installing non-packaged software as root on my machines, so what follows is all done as a regular user account. The first step is to pull down the devpi and wheel packages from pypi, telling pip to install them in under $HOME/.local

# pip install --user devpi
# pip install --user wheel

Since we’re using a custom install location, it necessary to update your $HOME/.bashrc file with new $PATH and $PYTHONPATH environment variables and then source the .bashrc file

# cat >> $HOME/.bashrc <<EOF
export PATH=\$PATH:$HOME/.local/bin
export PYTHONPATH=$HOME/.local/lib/python2.7/site-packages
EOF
# . $HOME/.bashrc

The devpi package provides a local server that will be used for downloads instead of directly accessing pypi.python.org, so this must be started

# devpi-server --start

Both devpi and wheel integrate with pip, so the next setup task is to modify the pip configuration file

# cat >> $HOME/.pip/pip.conf <<EOF
[global]
index-url = http://localhost:3141/root/pypi/+simple/
wheel-dir = /home/berrange/.pip/wheelhouse
find-links = /home/berrange/.pip/wheelhouse
EOF

We’re pretty much done at this point – all that is left is to prime the cache with the all the packages that Nova wants to use

# cd $HOME/src/cloud/nova
# pip wheel -r requirements.txt
# pip wheel -r test-requirements.txt

Now if you run any command that would build a Nova virtualenv, you should notice it is massively faster

# tox -e py27
# ./run_tests.sh -V

That is basically all there is to it. Blow away the virtualenv directories at any time and they’ll be repopulated from the cache. If the requirements.txt is updated with new deps re-running the ‘pip wheel’ command above will update the cache to hold the new bits.

That was all incredibly easy and so I’d highly recommend devs on any non-trivially sized python project make use of it.  Thanks to Joe Gordon for the pointing this out at the summit !