Faster rebuilds for python virtualenv trees
When developing on OpenStack, it is standard practice for the unit test suites to be run in a python virtualenv. With the Nova project at least, and I suspect most others too, setting up the virtualenv takes a significant amount of time as there are many packages to pull down from PyPI, quite a few of which need compilation too. Any time the local requirements.txt or test-requirements.txt files change it is necessary to the rebuild the virtualenv. This rebuild is an all-or-nothing kind of task, so can be a significant time sink, particularly if you frequently jump between different code branches.
At the OpenStack design summit in Paris, Joe Gordon showed Matt Booth how to setup devpi and wheel to provide a cache of the packages that make up the virtualenv. Not only does it avoid the need to repeatedly download the same packages from pypi each time, but it also avoids the compilation step, since the cache is storing the final installed pieces for each python module. The end result is that it takes 20-30 seconds or less to rebuild a virtualenv instead of many minutes.
After a few painful waits for virtualenvs today, I decided to set it up too I don’t like installing non-packaged software as root on my machines, so what follows is all done as a regular user account. The first step is to pull down the devpi and wheel packages from pypi, telling pip to install them in under $HOME/.local
# pip install --user devpi # pip install --user wheel
Since we’re using a custom install location, it necessary to update your $HOME/.bashrc file with new $PATH and $PYTHONPATH environment variables and then source the .bashrc file
# cat >> $HOME/.bashrc <<EOF export PATH=\$PATH:$HOME/.local/bin export PYTHONPATH=$HOME/.local/lib/python2.7/site-packages EOF # . $HOME/.bashrc
The devpi package provides a local server that will be used for downloads instead of directly accessing pypi.python.org, so this must be started
# devpi-server --start
Both devpi and wheel integrate with pip, so the next setup task is to modify the pip configuration file
# cat >> $HOME/.pip/pip.conf <<EOF [global] index-url = http://localhost:3141/root/pypi/+simple/ wheel-dir = /home/berrange/.pip/wheelhouse find-links = /home/berrange/.pip/wheelhouse EOF
We’re pretty much done at this point – all that is left is to prime the cache with the all the packages that Nova wants to use
# cd $HOME/src/cloud/nova # pip wheel -r requirements.txt # pip wheel -r test-requirements.txt
Now if you run any command that would build a Nova virtualenv, you should notice it is massively faster
# tox -e py27 # ./run_tests.sh -V
That is basically all there is to it. Blow away the virtualenv directories at any time and they’ll be repopulated from the cache. If the requirements.txt is updated with new deps re-running the ‘pip wheel’ command above will update the cache to hold the new bits.
That was all incredibly easy and so I’d highly recommend devs on any non-trivially sized python project make use of it. Thanks to Joe Gordon for the pointing this out at the summit !
Thanks for the very useful post! Helps a lot to speed up dependencies installation. However I’ve found out, that parameters `wheel-dir` and `find-links` in `pip.conf` should have `file://` prefixes like follows:
wheel-dir = file:///home/zoresvit/.pip/wheelhouse
find-links = file:///home/zoresvit/.pip/wheelhouse
Really helpful — thanks for posting.
Note: I had to upgrade my version of pip from 1.5.4 to 1.5.6 (easy_install -U pip) in order to get the wheel user install to succeed. (I was getting “ImportError: cannot import name IncompleteRead” with 1.5.4) After that, everything worked great.
daniel,
if i’m not misremembering my previous look into devpi and not misunderstanding the intentions of your instructions, then devpi is just a caching server and you are manually pre-populating the cache.
to get automated building and caching of python packages, i’m experimenting with a different approach: replace “pip install” in tox.ini with a call to a 25-line shell script (eg “install_command = {homedir}/bin/pip-wheel {packages}”). now that’s not a network-wide service, in case you are building on multiple machines and only want one network-wide cache, but you can accomplish that with nfs if really desired.
let’s see if your blog platform will handle shell scripts in comments (no, i’m not trying a command injection attack ;); i don’t have it in a public git repo yet)…
===== 8< =====
#!/bin/sh
set -e
DOWNLOAD_CACHE=${HOME}/.cache/python/download-cache/
WHEEL_DIR=${HOME}/.cache/python/wheel-dir/
ORIGINAL_TOX_PIP_INSTALL_OPTIONS="-U –force-reinstall"
# the only time tox deviates from the normal, and our specified, "
# ” is when it’s installing the application under test, which it does
# in editable mode (ie ” -e “), and when we should just
# run “pip install” without creating or caching wheels.
if [ “${1}” = “-e” ]; then
pip install ${@}
else
# we need wheel, so install it whether it’s already installed or not
# i’m too lazy to figure out the logic to conditionally install it,
# but i’ll write this long comment ;)
pip install wheel
pip wheel –download-cache ${DOWNLOAD_CACHE} –find-links ${WHEEL_DIR} –wheel-dir ${WHEEL_DIR} ${@}
pip install –no-index –find-links ${WHEEL_DIR} ${ORIGINAL_TOX_PIP_INSTALL_OPTIONS} ${@}
fi
exit 0
===== 8< =====
thanks for the motivation to seek out my preferred solution.
in the forth code section you should probably have file:///$HOME instead of /home/berrange. thanks for this.
As a note.
I initially tried the file:///home/USER_NAME/.pip/wheelhouse and met with failure :(
Using /home/USER_NAME/.pip/wheelhouse worked for me.
Of course change USER_NAME to your user name :)