What DevStack does to your host when setting up OpenStack on Fedora 17
As I mentioned in my previous post, I’m not really a fan of giant shell scripts which ask for unrestricted sudo access without telling you what they’re going todo. Unfortunately DevStack is one such script :-( So I decided to investigate just what it does to a Fedora 17 host when it is run. The general idea I had was
- Install a generic Fedora 17 guest
- Create a QCow2 image using the installed image as its backing file
- Reconfigure the guest to use the QCow2 image as its disk
- Run DevStack in the guest
- Compare the contents of the original installed image and the DevStack processed image
It sounded like libguestfs ought to be able to help out with the last step, and after a few words with Rich, I learnt about use of virt-ls for exactly this purpose. After trying this once, it quickly became apparent that just comparing the lists of files is quite difficult because DevStack installs a load of extra RPMs with many 1000’s of files. So to take this out of the equation, I grabbed the /var/log/yum.log file to get a list of all RPMs that DevStack had installed, and manually added them into the generic Fedora 17 guest base image. Now I could re-run DevStack again and do a file comparison which excluded all the stuff installed by RPM.
RPM packages installed (with YUM)
- apr-1.4.6-1.fc17.x86_64
- apr-util-1.4.1-2.fc17.x86_64
- apr-util-ldap-1.4.1-2.fc17.x86_64
- augeas-libs-0.10.0-3.fc17.x86_64
- binutils-2.22.52.0.1-10.fc17.x86_64
- boost-1.48.0-13.fc17.x86_64
- boost-chrono-1.48.0-13.fc17.x86_64
- boost-date-time-1.48.0-13.fc17.x86_64
- boost-filesystem-1.48.0-13.fc17.x86_64
- boost-graph-1.48.0-13.fc17.x86_64
- boost-iostreams-1.48.0-13.fc17.x86_64
- boost-locale-1.48.0-13.fc17.x86_64
- boost-program-options-1.48.0-13.fc17.x86_64
- boost-python-1.48.0-13.fc17.x86_64
- boost-random-1.48.0-13.fc17.x86_64
- boost-regex-1.48.0-13.fc17.x86_64
- boost-serialization-1.48.0-13.fc17.x86_64
- boost-signals-1.48.0-13.fc17.x86_64
- boost-system-1.48.0-13.fc17.x86_64
- boost-test-1.48.0-13.fc17.x86_64
- boost-thread-1.48.0-13.fc17.x86_64
- boost-timer-1.48.0-13.fc17.x86_64
- boost-wave-1.48.0-13.fc17.x86_64
- ceph-0.44-5.fc17.x86_64
- check-0.9.8-5.fc17.x86_64
- cloog-ppl-0.15.11-3.fc17.1.x86_64
- cpp-4.7.2-2.fc17.x86_64
- curl-7.24.0-5.fc17.x86_64
- Django-1.4.2-1.fc17.noarch
- django-registration-0.7-3.fc17.noarch
- dmidecode-2.11-8.fc17.x86_64
- dnsmasq-utils-2.63-1.fc17.x86_64
- ebtables-2.0.10-5.fc17.x86_64
- euca2ools-2.1.1-2.fc17.noarch
- gawk-4.0.1-1.fc17.x86_64
- gcc-4.7.2-2.fc17.x86_64
- genisoimage-1.1.11-14.fc17.x86_64
- git-1.7.11.7-1.fc17.x86_64
- glusterfs-3.2.7-2.fc17.x86_64
- glusterfs-fuse-3.2.7-2.fc17.x86_64
- gnutls-utils-2.12.17-1.fc17.x86_64
- gperftools-libs-2.0-5.fc17.x86_64
- httpd-2.2.22-4.fc17.x86_64
- httpd-tools-2.2.22-4.fc17.x86_64
- iptables-1.4.14-2.fc17.x86_64
- ipxe-roms-qemu-20120328-1.gitaac9718.fc17.noarch
- iscsi-initiator-utils-6.2.0.872-18.fc17.x86_64
- kernel-headers-3.6.6-1.fc17.x86_64
- kpartx-0.4.9-26.fc17.x86_64
- libaio-0.3.109-5.fc17.x86_64
- libcurl-7.24.0-5.fc17.x86_64
- libmpc-0.9-2.fc17.2.x86_64
- libunwind-1.0.1-3.fc17.x86_64
- libusal-1.1.11-14.fc17.x86_64
- libvirt-0.9.11.7-1.fc17.x86_64
- libvirt-client-0.9.11.7-1.fc17.x86_64
- libvirt-daemon-0.9.11.7-1.fc17.x86_64
- libvirt-daemon-config-network-0.9.11.7-1.fc17.x86_64
- libvirt-daemon-config-nwfilter-0.9.11.7-1.fc17.x86_64
- libvirt-python-0.9.11.7-1.fc17.x86_64
- libwsman1-2.2.7-5.fc17.x86_64
- lzop-1.03-4.fc17.x86_64
- m2crypto-0.21.1-8.fc17.x86_64
- mod_wsgi-3.3-2.fc17.x86_64
- mx-3.2.3-1.fc17.x86_64
- mysql-5.5.28-1.fc17.x86_64
- mysql-libs-5.5.28-1.fc17.x86_64
- MySQL-python-1.2.3-5.fc17.x86_64
- mysql-server-5.5.28-1.fc17.x86_64
- netcf-libs-0.2.2-1.fc17.x86_64
- numad-0.5-4.20120522git.fc17.x86_64
- numpy-1.6.2-1.fc17.x86_64
- parted-3.0-10.fc17.x86_64
- perl-AnyEvent-5.27-7.fc17.noarch
- perl-AnyEvent-AIO-1.1-8.fc17.noarch
- perl-AnyEvent-BDB-1.1-7.fc17.noarch
- perl-Async-MergePoint-0.03-7.fc17.noarch
- perl-BDB-1.88-5.fc17.x86_64
- perl-common-sense-3.5-1.fc17.noarch
- perl-Compress-Raw-Bzip2-2.052-1.fc17.x86_64
- perl-Compress-Raw-Zlib-2.052-1.fc17.x86_64
- perl-Config-General-2.50-6.fc17.noarch
- perl-Coro-6.07-3.fc17.x86_64
- perl-Curses-1.28-5.fc17.x86_64
- perl-DBD-MySQL-4.020-2.fc17.x86_64
- perl-DBI-1.617-1.fc17.x86_64
- perl-Encode-Locale-1.02-5.fc17.noarch
- perl-Error-0.17016-7.fc17.noarch
- perl-EV-4.03-8.fc17.x86_64
- perl-Event-1.20-1.fc17.x86_64
- perl-Event-Lib-1.03-16.fc17.x86_64
- perl-Git-1.7.11.7-1.fc17.noarch
- perl-Glib-1.241-2.fc17.x86_64
- perl-Guard-1.022-1.fc17.x86_64
- perl-Heap-0.80-10.fc17.noarch
- perl-HTML-Parser-3.69-3.fc17.x86_64
- perl-HTML-Tagset-3.20-10.fc17.noarch
- perl-HTTP-Date-6.00-3.fc17.noarch
- perl-HTTP-Message-6.03-1.fc17.noarch
- perl-IO-AIO-4.15-1.fc17.x86_64
- perl-IO-Async-0.29-7.fc17.noarch
- perl-IO-Compress-2.052-1.fc17.noarch
- perl-IO-Socket-SSL-1.66-1.fc17.noarch
- perl-IO-Tty-1.10-5.fc17.x86_64
- perl-LWP-MediaTypes-6.01-4.fc17.noarch
- perl-Net-HTTP-6.02-2.fc17.noarch
- perl-Net-LibIDN-0.12-8.fc17.x86_64
- perl-Net-SSLeay-1.48-1.fc17.x86_64
- perl-POE-1.350-2.fc17.noarch
- perl-Socket6-0.23-8.fc17.x86_64
- perl-Socket-GetAddrInfo-0.19-1.fc17.x86_64
- perl-TermReadKey-2.30-14.fc17.x86_64
- perl-TimeDate-1.20-6.fc17.noarch
- perl-URI-1.60-1.fc17.noarch
- ppl-0.11.2-8.fc17.x86_64
- ppl-pwl-0.11.2-8.fc17.x86_64
- pylint-0.25.1-1.fc17.noarch
- python-amqplib-1.0.2-3.fc17.noarch
- python-anyjson-0.3.1-3.fc17.noarch
- python-babel-0.9.6-3.fc17.noarch
- python-BeautifulSoup-3.2.1-3.fc17.noarch
- python-boto-2.5.2-1.fc17.noarch
- python-carrot-0.10.7-4.fc17.noarch
- python-cheetah-2.4.4-2.fc17.x86_64
- python-cherrypy-3.2.2-1.fc17.noarch
- python-coverage-3.5.1-0.3.b1.fc17.x86_64
- python-crypto-2.6-1.fc17.x86_64
- python-dateutil-1.5-3.fc17.noarch
- python-devel-2.7.3-7.2.fc17.x86_64
- python-docutils-0.8.1-3.fc17.noarch
- python-eventlet-0.9.17-1.fc17.noarch
- python-feedparser-5.1.2-2.fc17.noarch
- python-gflags-1.5.1-2.fc17.noarch
- python-greenlet-0.3.1-11.fc17.x86_64
- python-httplib2-0.7.4-6.fc17.noarch
- python-iso8601-0.1.4-4.fc17.noarch
- python-jinja2-2.6-2.fc17.noarch
- python-kombu-1.1.3-2.fc17.noarch
- python-lockfile-0.9.1-2.fc17.noarch
- python-logilab-astng-0.23.1-1.fc17.noarch
- python-logilab-common-0.57.1-2.fc17.noarch
- python-lxml-2.3.5-1.fc17.x86_64
- python-markdown-2.1.1-1.fc17.noarch
- python-migrate-0.7.2-2.fc17.noarch
- python-mox-0.5.3-4.fc17.noarch
- python-netaddr-0.7.5-4.fc17.noarch
- python-nose-1.1.2-2.fc17.noarch
- python-paramiko-1.7.7.1-2.fc17.noarch
- python-paste-deploy-1.5.0-4.fc17.noarch
- python-paste-script-1.7.5-4.fc17.noarch
- python-pep8-1.0.1-1.fc17.noarch
- python-pip-1.0.2-2.fc17.noarch
- python-pygments-1.4-4.fc17.noarch
- python-qpid-0.18-1.fc17.noarch
- python-routes-1.12.3-3.fc17.noarch
- python-setuptools-0.6.27-2.fc17.noarch
- python-sphinx-1.1.3-1.fc17.noarch
- python-sqlalchemy-0.7.9-1.fc17.x86_64
- python-suds-0.4.1-2.fc17.noarch
- python-tempita-0.5.1-1.fc17.noarch
- python-unittest2-0.5.1-3.fc17.noarch
- python-virtualenv-1.7.1.2-2.fc17.noarch
- python-webob-1.1.1-2.fc17.noarch
- python-wsgiref-0.1.2-8.fc17.noarch
- pyxattr-0.5.1-1.fc17.x86_64
- PyYAML-3.10-3.fc17.x86_64
- qemu-common-1.0.1-2.fc17.x86_64
- qemu-img-1.0.1-2.fc17.x86_64
- qemu-system-x86-1.0.1-2.fc17.x86_64
- qpid-cpp-client-0.18-5.fc17.x86_64
- qpid-cpp-server-0.18-5.fc17.x86_64
- radvd-1.8.5-3.fc17.x86_64
- screen-4.1.0-0.9.20120314git3c2946.fc17.x86_64
- scsi-target-utils-1.0.24-6.fc17.x86_64
- seabios-bin-1.7.1-1.fc17.noarch
- sg3_utils-1.31-2.fc17.x86_64
- sgabios-bin-0-0.20110622SVN.fc17.noarch
- spice-server-0.10.1-5.fc17.x86_64
- sqlite-3.7.11-3.fc17.x86_64
- tcpdump-4.2.1-3.fc17.x86_64
- vgabios-0.6c-4.fc17.noarch
- wget-1.13.4-7.fc17.x86_64
- xen-libs-4.1.3-5.fc17.x86_64
- xen-licenses-4.1.3-5.fc17.x86_64
Python packages installed (with PIP)
These all ended up in /usr/lib/python2.7/site-packages :-(
- WebOp
- amqplib
- boto (splattering over existing boto RPM package with older version)
- cinderclient
- cliff
- cmd2
- compressor
- django_appconf
- django_compressor
- django_openstack_auth
- glance
- horizon
- jsonschema
- keyring
- keystoneclient
- kombu
- lockfile
- nova
- openstack_auth
- pam
- passlib
- prettytable
- pyparsing
- python-cinderclient
- python-glanceclient
- python-novaclient
- python-openstackclient
- python-quantumclient
- python-swiftclient
- pytz
- quantumclient
- suds
- swiftclient
- warlock
- webob
Files changed
- /etc/group (added $USER to ‘libvirtd’ group)
- /etc/gshadow (as above)
- /etc/httpd/conf/httpd.conf (changes Listen 80 to Listen 0.0.0.0:80)
- /usr/lib/python2.7/site-packages/boto (due to overwriting RPM provided boto)
- /usr/bin/cq (as above)
- /usr/bin/elbadmin (as above)
- /usr/bin/list_instances (as above)
- /usr/bin/lss3 (as above)
- /usr/bin/route53 (as above)
- /usr/bin/s3multiput (as above)
- /usr/bin/s3put (as above)
- /usr/bin/sdbadmin (as above)
Files created
- /etc/cinder/*
- /etc/glance/*
- /etc/keystone/*
- /etc/nova/*
- /etc/httpd/conf.d/horizon.conf
- /etc/polkit-1/localauthority/50-local.d/50-libvirt-reomte-access.pkla
- /etc/sudoers.d/50_stack_sh
- /etc/sudoers.d/cinder-rootwrap
- /etc/sudoers.d/nova-rootwrap
- $DEST/cinder/*
- $DEST/data/*
- $DEST/glance/*
- $DEST/horizon/*
- $DEST/keystone/*
- $DEST/noVNC/*
- $DEST/nova/*
- $DEST/python-cinderclient/*
- $DEST/python-glanceclient/*
- $DEST/python-keystoneclient/*
- $DEST/python-novaclient/*
- $DEST/python-openstackclient/*
- /usr/bin/cinder*
- /usr/bin/glance*
- /usr/bin/keystone*
- /usr/bin/nova*
- /usr/bin/openstack
- /usr/bin/quantum
- /usr/bin/swift
- /var/cache/cinder/*
- /var/cache/glance/*
- /var/cache/keystone/*
- /var/cache/nova/*
- /var/lib/mysql/cinder/*
- /var/lib/mysql/glance/*
- /var/lib/mysql/keystone/*
- /var/lib/mysql/mysql/*
- /var/lib/mysql/nova/*
- /var/lib/mysql/performance_schema/*
Thoughts on installation
As we can see from the details above, DevStack does a very significant amount of work as root using sudo. I had fully expected that it was installing RPMs as root, but I had not counted on it adding extra python modules into /usr/lib/python-2.7, nor the addition of files in /etc/, /var or /usr/bin. I had set the $DEST environment variable for DevStack naively assuming that it would cause it to install everything possible under that location. In fact the $DEST variable was only used to control where the GIT checkouts of each openstack component went, along with a few misc files in $DEST/files/
IMHO a development environment setup tool should do as little as humanely possible as root. From the above list of changes, the only things that I believe justify use of sudo privileges are:
- Installation of RPMs from standard YUM repositories
- Installation of /etc/sudoers.d/ config files
- Installation of /etc/polkit file to grant access to libvirtd
Everything else is capable of being 100% isolated from the rest of the OS, under the $DEST directory location. Taking that into account my preferred development setup would be
$DEST
+- nova (GIT checkout)
+- ...etc... (GIT checkout)
+- vroot
+- bin
| +- nova
| +- ...etc...
+- etc
| +- nova
| +- ...etc...
+- lib
| +- python2.7
| +- site-packages
| +- boto
| +- ...etc...
+- var
+- cache
| +- nova
| +- ...etc...
+- lib
+- mysql
This would imply running a private copy of qpid, mysql and httpd, ideally all inside the same screen session as the rest of the OpenStack services, using unprivileged ports. Even if we relied on the system instances of qpid, mysql, httpd and did a little bit more privileged config, 95% of the rest of the stuff DevStack does as root, could still be kept unprivileged. I am also aware that current OpenStack code may not be amenable to installation in locations outside of / by default, but the code is all there to be modified to cope with arbitrary install locations if desired/required.
My other wishlist item for DevStack would be for it to print output that is meaningful to the end user running it. Simply printing a verbose list of every single shell command executed is one of the most unkind things you can do to a user. I’d like to see
# ./devstack.sh
* Cloning GIT repositories
- 1/10 nova
- 2/10 cinder
- 3/10 quantum
* Installing RPMs using YUM
- 1/30 python-pip
- 2/30 libvirt
- 3/30 libvirt-client
* Installing Python packages to $DEST/vroot/lib/python-2.7/site-packages using PIP
- 1/24 WebOb
- 2/24 ampqlib
- 3/24 boto
* Creating database schemas
- 1/10 nova
- 2/10 cinder
- 3/10 quantum
By all means still save the full list of every shell command and their output to a ‘devstack.log’ file for troubleshooting when things go wrong.
Agreed we should do better, also over the 6 months I’ve contributed to devstack it’s gotten a lot better than it was.
Patches welcomed! I’ll happy review them and help get them in.
Some good ideas. I’d be interested in having devstack work with a virtualenv, too. Add me as a reviewer on your patches!