Today I purchased one of the lovely new ARM based Samsung Google Chromebooks. While ChromeOS is very slick, I don’t have much interest in running that, I got this machine to serve as a dev machine for doing ARM work on libvirt and related projects. Not co-incidentally, this machine has the ARM Cortex A-15 CPU which is the architecture targeted by the new KVM for ARM work.
Obviously the first task at hand is to actually get Fedora onto the machine. Fortunately Christopher Hewitt has been here already and wrote up some notes for Fedora 17 on Google+. The notes look good, but when you look, the first step in installing Fedora is to use an Ubuntu host to build a SD card. So what follows is my slightly tweaked instructions for doing the same starting from a Fedora 17 host.
Building the bootable SD card
We don’t want to blow away ChromeOS entirely, so this guide is focused on booting using an SD Card. I have a bunch of 8 GB SD cards for my SLR, so one of those is re-purposed for this task.
Creating partitions & filesystems
The first part of the process is to create the partitions and root filesystem. All the instructions below are to be run as root on the Fedora 17 host.
- Insert the SD card in the host machine, let GNOME automount it, and then look to see what device it is
# df
Filesystem 1K-blocks Used Available Use% Mounted on
rootfs 151607468 127703344 16202828 89% /
devtmpfs 1958948 0 1958948 0% /dev
tmpfs 1970892 284 1970608 1% /dev/shm
tmpfs 1970892 1656 1969236 1% /run
/dev/mapper/vg_t500wlan-lv_root 151607468 127703344 16202828 89% /
tmpfs 1970892 0 1970892 0% /sys/fs/cgroup
tmpfs 1970892 0 1970892 0% /media
/dev/sda1 198337 126585 61512 68% /boot
/dev/mmcblk0p1 7753728 7668960 84768 99% /run/media/berrange/NIKON D90
So in this case, I see the device is /dev/mmcblk0 (ignoring the p1 suffix which is the partition number)
- Take a look at the SD card and double-check there are no important files still on it, then unmount it. I found a few photographs that I hadn’t transferred, so happy I checked
- Being a nice modern platform, the Chromebook doesn’t want no stinkin’ MS-Dos partition table. This is a GPT world we live it. Using parted we re-format the SD Card with a GPT table (well, ok, if you must know, this creates a MS-Dos partition table too for back compat)
# parted /dev/mmcblk0
GNU Parted 3.0
Using /dev/mmcblk0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mktable gpt
Warning: The existing disk label on /dev/mmcblk0 will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes
(parted) quit
Information: You may need to update /etc/fstab.
- Now time to create 3 partitions, 2 for storing kernels, and one for storing the filesystem. You could just create 1 for the kernel, if you don’t care about having a backup sane kernel for emergencies
## gdisk /dev/mmcblk0
GPT fdisk (gdisk) version 0.8.4
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
Command (? for help): x
Expert command (? for help): l
Enter the sector alignment value (1-65536, default = 2048): 8192
Expert command (? for help): m
Command (? for help): n
Partition number (1-128, default 1): 1
First sector (34-15523806, default = 8192) or {+-}size{KMGTP}:
Last sector (8192-15523806, default = 15523806) or {+-}size{KMGTP}: +16M
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): 7f00
Changed type of partition to 'ChromeOS kernel'
Command (? for help): n
Partition number (2-128, default 2): 2
First sector (34-15523806, default = 40960) or {+-}size{KMGTP}:
Last sector (40960-15523806, default = 15523806) or {+-}size{KMGTP}: +16M
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): 7f00
Changed type of partition to 'ChromeOS kernel'
Command (? for help): n
Partition number (3-128, default 3): 3
First sector (34-15523806, default = 73728) or {+-}size{KMGTP}:
Last sector (73728-15523806, default = 15523806) or {+-}size{KMGTP}:
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300):
Changed type of partition to 'Linux filesystem'
Command (? for help): w
Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!
Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/mmcblk0.
The operation has completed successfully.
- With that out of the way, it is time to format the filesystem on the 3rd partition, and ext4 is the choice we’re going for here
# mkfs.ext4 /dev/mmcblk0p3
mke2fs 1.42.3 (14-May-2012)
Discarding device blocks: ^C76960/1931259
[root@t500wlan ~]#
[root@t500wlan ~]# mkfs.ext4 /dev/mmcblk0p3
mke2fs 1.42.3 (14-May-2012)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
483328 inodes, 1931259 blocks
96562 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1979711488
59 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
- A filesystem isn’t too interesting, unless it has some data in it. Lets download a suitable Fedora 17 ARM filesystem archive and copy its data across
# wget http://download.fedoraproject.org/pub/fedora-secondary/releases/17/Images/armhfp/Fedora-17-armhfp-xfce.tar.xz
# mount /dev/mmcblk0p3 /mnt
# cd /mnt
# tar Jxvf /root/Fedora-17-armhfp-xfce.tar.xz
When unpacked, the filesystem will consume about 2.4 GB of space, so a 8GB SD card isn’t strictly needed – a 4GB one would suffice.
- The core F17 image is now installed on the SD card, so unmount the card. The next step will move onto the ChromeBook
# umount /dev/mmcblk0p3
Getting the Chromebook into Developer Mode
Before moving onto the next stage, the ChomeBook must be in Developer Mode.
- Booting into Recovery Mode is the first thing todo. This requires holding down the Esc and Refresh keys, while pressing the Power key.
- It should now display a nice scary message that the installation is broken. This is of course a lie, so ignore it.
- Pressing Ctrl-D now will start the switch into Developer Mode.
- It should now display a less scary message asking if you want to turn off verification. We do indeed want todo this, so press Enter.
- The ChromeBook will reboot and tell you that verification is disabled.
- Wait a short while, or press Ctrl-D, and it’ll display another message that it is transitioning to Developer Mode.
- Erasing local data can take a while, so be patient here. Strangely, there’s a crude ascii art progress bar along the top of the screen here, while everything else remains pretty & graphical.
Copying the ChromeBook ARM kernels
Fedora 17 doesn’t come with a suitable kernel for the Cortex A-15 CPU, so this step grabs the one already present in ChromeOS.
- Assuming the ChromeBook is now booted in Developer Mode, insert the SD card.
- Bring up a
croshprompt by typing Ctrl-Alt-T
- From here, run ‘shell’ which gives you a bash session.
crosh> shell
chronos@localhost / $
- Become the root user
$ sudo su -
localhost ~ #
- The kernel image has the boot arguments embedded in it, and these need to be changed for Fedora. This involves a fun little command which bundles up the vmlinuz file, config args and boot keys into one magic image
# cd /tmp
# echo “console=tty1 debug verbose root=/dev/mmcblk1p3 rootwait rw” > /tmp/config
# vbutil_kernel –pack /tmp/newkern –keyblock /usr/share/vboot/devkeys/kernel.keyblock –version 1 –signprivate /usr/share/vboot/devkeys/kernel_data_key.vbprivk –config=/tmp/config –vmlinuz /boot/vmlinuz-3.4.0 –arch arm
- Assuming that worked, the new kernel image can copied across to the new SD card partitions created earlier
# dd if=/tmp/newkern of=/dev/mmcblk1p1
# dd if=/tmp/newkern of=/dev/mmcblk1p2
# rm /tmp/newkern /tmp/config
- Having the kernels in a partition isn’t much use, unless you mark those partitions as bootable
# cgpt add -i 1 -S 1 -T 5 -P 10 -l KERN-A /dev/mmcblk1
# cgpt add -i 2 -S 1 -T 5 -P 5 -l KERN-B /dev/mmcblk1
This says try to boot from the first partition 10 times, then fallback to trying the second partition 5 times. This fallback behaviour is handy if you want to test out custom kernel builds, and have an escape route for when it all goes belly up.
- Much of the ChromeOS kernel is built as modules, so those also need to be copied across to the filesystem image
cp -rf /lib/modules/* /media/removable/External\ Drive\ 1/lib/modules/
cp -rf /lib/firmware/* /media/removable/External\ Drive\ 1/lib/firmware/
- Even in Developer Mode, the Chromebook will ignore the SD/USB card when booting, unless explicitly told to look at them
# crossystemm dev_boot_usb=1
It might print out a couple of warnings, but those can be ignored – it will have done what is required.
- Xorg will work fine out of the box, but the TouchPad sensitivity can do with improving
# cat > /media/removable/External\ Drive\ 1/etc/X11/xorg.conf.d/50-touchpad.conf <<EOF
Section "InputClass"
Identifier "touchpad"
MatchIsTouchpad "on"
Option "FingerHigh" "5"
Option "FingerLow" "5"
EndSection
EOF
Some other guides suggest copying across the Xorg drivers armsoc_drv.so and cmt_drv.so, but I wanted to keep my system as vanilla & open source as possible
- For sound to work, copy the alsa profiles across too
# cp -a /usr/share/alsa/ucm/ /media/removable/External\ Drive\ 1/usr/share/alsa/ucm
WARNING: Under no circumstances use ‘alsamixer’ to adjust setting on your ChromeBook. Multiple people have reported (with pictures) melting/burning out the speakers (no seriously, you can kill the hardware)
- Set things up so that the alsa settings are loaded on startup
# cat > /media/removable/External\ Drive\ 1/etc/rc.d/rc.local <<EOF
#!/bin/sh
alsaucm -C DAISY-I2S set _verb HiFi
EOF
# chmod +x /media/removable/External\ Drive\ 1/etc/rc.d/rc.local
Note that some instructions miss off the ‘set _verb HiFi’ part of this command line. My experiance is that audio did not work unless that was set.
- The final thing todo is configure the root password and user account
# mount -o remount,suid,exec /media/removable/External\ Drive\ 1/
# chroot /media/removable/External\ Drive\ 1/
# passwd
...enter new root password...
# mkdir /home/guest
# rsync -a /etc/skel/ /home/guest
# chown -R guest:users /home/guest
# usermod -d /home/guest -g users guest
# exit
# umount /media/removable/External\ Drive\ 1/
That’s it, you’re now ready to rock-n-roll. Power-cycle the Chromebook, and when the initial boot screen appears press Ctrl-U to boot from the SD card.
First impressions of Fedora 17 on the Chromebook
Writing these instructions was interrupted part way through, when I began to realize that the TrackPad on my Chromebook was faulty. I didn’t work at all on first boot of ChromeOS, but second time around it was working fine, so I thought nothing of it. It just got worse & worse from then on though. Looking at the kernel dmesg logs showed the hardware did not even appear visible 60% of the boot attempts. It is probably just a loose cable inside, but I didn’t fancy opening up the case, so I took it back to PC World for an exchange. Thankfully the 2nd unit worked flawlessly.
The first annoying thing to discover is that it is only possible to boot custom distros on the ChromeBook while in Developer Mode. This means that at every power-on, the boot screen pauses 30 seconds with a scary warning. AFAICT, there is no ability to sign the kernel images, such that you can boot in Release Mode, without the warning. So before distros have even done any work, it appears guaranteed that their user experience will be worse that ChromeOS on this hardware. Thankyou “verified boot”.
Booting up Fedora 17 is significantly slower than booting ChromeOS. While some of this might be Fedora’s fault, I’m not prepared to lay any blame here, since I’m comparing booting from the internal drive for ChromeOS, vs booting from an SD-Card for Fedora. It just isn’t a fair comparison unless I can get Fedora onto the internal drive. It also isn’t as pretty, though again that is mostly my fault for not using graphical boot with Fedora.
If the procedure above is followed, you will get a pretty functional Fedora 17 XFCE desktop environment. Since I skipped the part others recommend about copying over the binary Xorg driver for the graphics card, X performance isn’t going to win any benchmarks though. Playing youtube video in firefox with HTML5, in a smallest video size consumes 95% CPU time. I expect proper graphics drivers would help that significantly.
In anycase I’m not doing any of this for Xorg / graphics performance. This is primarily going to be a dev machine which I remotely access for doing libvirt work
The hope is that when Fedora 18 is released for ARM, there will be a remix done specifically for the new ChromeBook which will simplify the setup process, use a normal Fedora kernel, and just generally “do the right thing”. I may well try to re-do this image using a F18 alpha filesystem as the source instead.
When a program has to kill a process, it may well want to perform some safety checks before invoking kill(), to ensure that the process really is the one that it is expecting. For example, it might check that the /proc/$PID/exe symlink matches the executable it originally used to start it. Or it might want to check the uid/gid the process is running as from /proc/$PID/status. Or any number of other attributes about a process that are exposed in /proc/$PID files.
This is really a similar pattern to the checks that a process might do when operating on a file. For example, it may stat() the file to check it has the required uid/gid ownership. It has long been known that doing stat() followed by open() is subject to a race condition which can have serious security implications in many circumstances. For this reason, well written apps will actually do open() and then fstat() on the file descriptor, so they’re guaranteed the checks are performed against the file they’re actually operating on.
If the kernel ever wraps around on PID numbers, it should be obvious that apps that do checks on /proc/$PID before killing a process are also subject to a race condition with potential security implications. This leads to the questions of whether it is possible to design a race free way of checking & killing a process, similar to the open() + fstat() pattern used for files. AFAICT there is not, but there are a couple of ways that one could be constructed without too much extra support from the kernel.
To attack this, we first need an identifier for the process which is unique and not reusable if a process dies & new one takes it place. As mentioned above, if the kernel allows PID numbers to wrap, the PID is unsuitable for this purpose. A file handle for the /proc/$PID directory though, it potentially suitable for this purpose. We still can’t simply open files like /proc/$PID/exe directly, since that’s relying on the potentially reusable PID number. Here the openat()/readlinkat() system calls come to rescuse. They allow you to pass a file descriptor for a directory and a relative filename. This guarantees that the file opened is in the directory expected, even if the original directory path changes / is replaced. All that is now missing is a way to kill a process, given a FD for /proc/$PID. There are two ideas for this, first a new system call fkill($fd, $signum), or a new proc file /proc/$PID/kill where you can write signal numbers.
With these pieces, a race free way to validate a process’ executable before killing it would look like
int pidfd = open("/proc/1234", O_RDONLY);
char buf[PATH_MAX];
char *exe = readlinkat(pidfd, "exe", buf, sizeof(buf));
if (strcmp(exe, "/usr/bin/mybinary") != 0)
return -1;
fkill(pidfd, SIGTERM);
Alternatively the last line could look like
char signum[10];
snprintf(signum, sizeof(signum), "%d", SIGTERM);
int sigfd = openat(pidfd, "kill", O_WRONLY);
write(sigfd, signum, strlen(signum))
close(sigfd)
Though I prefer the fkill() variant for reasons of brevity.
After reading this, you might wonder how something like systemd safely kills processes it manages without this kind of functionality. Well, it uses cgroups to confine processes associated with each service. Therefore it does not need to care about safety checking arbitrary PIDs, instead it can merely iterate over every PID in the service’s cgroup and kill them. The downside of cgroups is that it is Linux specific, but that doesn’t matter for systemd’s purposes. I believe that a fkill() syscall, however, is something that could be more easily used in a variety of existing apps without introducing too much portability code. Apps could simply use fkill() if available, but fallback to regular kill() elsewhere.
NB I’ve no intention of actually trying to implement this, since I like to stay out of kernel code. It is just an idea I wanted to air in case it interests anyone else.
Two weeks back, I was in Barcelona for LinuxCon Europe / KVM Forum 2012. While there Jeff Cody acquired a photo of many of the KVM community developers. Although already visible on Google+, along with tags to identify all the faces, I wanted to put up an outline view of the photo too, mostly so that I could then write this blog post describing how to create the head outline :-) The steps on this page were all performed using Fedora 17 and GIMP 2.8.2, but this should work with pretty much every version of GIMP out there since there’s nothing fancy going on.
The master photo
The master photo that we’ll be working with is
Step 1: Edge detect
It was thought that one of the edge detection algorithms available in GIMP would be a good basis for providing a head outline. After a little trial & error, I picked ‘Filters -> Edge-detect -> Edge..’, then chose the ‘Laplace’ algorithm.
This resulted in the following image
Step 2: Invert colours
The previous image shows the outlines quite effectively, but my desire is for a primarily white image, with black outlines. This is easily achieved using the menu option ‘Colours -> Invert’
Step 3: Desaturate
The edge detection algorithm leaves some colour artifacts in the images, which are trivially dealt with by desaturating the image using ‘Colours -> Desaturate…’ and any one of the desaturation algorithms GIMP offers.
Step 4: Boost contrast
The outline looks pretty good, but there is still a fair amount of fine detail “noise”. There are a few ways we might get rid of this – in particular some of GIMPs noise removal filters. I went for the easy option of simply boosting the overall image contrast, using ‘Colours -> Brightness/Contrast…’
For this image, setting the contrast to ’40’ worked well, vary according to the particular characteristics of the image
Step 5: Add numbers
The outline view is where we want to be, but the whole point of the exercise is to make it easy to put names to faces. Thus the final step is to simply number each head. GIMP’s text tool is the perfect way to do this, just click on each face in turn and type in a number.
No need to worry about perfect placement, since each piece of text becomes a new layer. Once done, the layer positions can be moved around to fit well.
And that’s the final image completed. In the page I created on the KVM website, a little javascript handled swapping between the original & outline views on mouse over, but that’s all there is to it. The hardest part of the whole exercise is actually remembering who everyone is :-P
As I mentioned in my previous post, I’m not really a fan of giant shell scripts which ask for unrestricted sudo access without telling you what they’re going todo. Unfortunately DevStack is one such script :-( So I decided to investigate just what it does to a Fedora 17 host when it is run. The general idea I had was
- Install a generic Fedora 17 guest
- Create a QCow2 image using the installed image as its backing file
- Reconfigure the guest to use the QCow2 image as its disk
- Run DevStack in the guest
- Compare the contents of the original installed image and the DevStack processed image
It sounded like libguestfs ought to be able to help out with the last step, and after a few words with Rich, I learnt about use of virt-ls for exactly this purpose. After trying this once, it quickly became apparent that just comparing the lists of files is quite difficult because DevStack installs a load of extra RPMs with many 1000’s of files. So to take this out of the equation, I grabbed the /var/log/yum.log file to get a list of all RPMs that DevStack had installed, and manually added them into the generic Fedora 17 guest base image. Now I could re-run DevStack again and do a file comparison which excluded all the stuff installed by RPM.
RPM packages installed (with YUM)
- apr-1.4.6-1.fc17.x86_64
- apr-util-1.4.1-2.fc17.x86_64
- apr-util-ldap-1.4.1-2.fc17.x86_64
- augeas-libs-0.10.0-3.fc17.x86_64
- binutils-2.22.52.0.1-10.fc17.x86_64
- boost-1.48.0-13.fc17.x86_64
- boost-chrono-1.48.0-13.fc17.x86_64
- boost-date-time-1.48.0-13.fc17.x86_64
- boost-filesystem-1.48.0-13.fc17.x86_64
- boost-graph-1.48.0-13.fc17.x86_64
- boost-iostreams-1.48.0-13.fc17.x86_64
- boost-locale-1.48.0-13.fc17.x86_64
- boost-program-options-1.48.0-13.fc17.x86_64
- boost-python-1.48.0-13.fc17.x86_64
- boost-random-1.48.0-13.fc17.x86_64
- boost-regex-1.48.0-13.fc17.x86_64
- boost-serialization-1.48.0-13.fc17.x86_64
- boost-signals-1.48.0-13.fc17.x86_64
- boost-system-1.48.0-13.fc17.x86_64
- boost-test-1.48.0-13.fc17.x86_64
- boost-thread-1.48.0-13.fc17.x86_64
- boost-timer-1.48.0-13.fc17.x86_64
- boost-wave-1.48.0-13.fc17.x86_64
- ceph-0.44-5.fc17.x86_64
- check-0.9.8-5.fc17.x86_64
- cloog-ppl-0.15.11-3.fc17.1.x86_64
- cpp-4.7.2-2.fc17.x86_64
- curl-7.24.0-5.fc17.x86_64
- Django-1.4.2-1.fc17.noarch
- django-registration-0.7-3.fc17.noarch
- dmidecode-2.11-8.fc17.x86_64
- dnsmasq-utils-2.63-1.fc17.x86_64
- ebtables-2.0.10-5.fc17.x86_64
- euca2ools-2.1.1-2.fc17.noarch
- gawk-4.0.1-1.fc17.x86_64
- gcc-4.7.2-2.fc17.x86_64
- genisoimage-1.1.11-14.fc17.x86_64
- git-1.7.11.7-1.fc17.x86_64
- glusterfs-3.2.7-2.fc17.x86_64
- glusterfs-fuse-3.2.7-2.fc17.x86_64
- gnutls-utils-2.12.17-1.fc17.x86_64
- gperftools-libs-2.0-5.fc17.x86_64
- httpd-2.2.22-4.fc17.x86_64
- httpd-tools-2.2.22-4.fc17.x86_64
- iptables-1.4.14-2.fc17.x86_64
- ipxe-roms-qemu-20120328-1.gitaac9718.fc17.noarch
- iscsi-initiator-utils-6.2.0.872-18.fc17.x86_64
- kernel-headers-3.6.6-1.fc17.x86_64
- kpartx-0.4.9-26.fc17.x86_64
- libaio-0.3.109-5.fc17.x86_64
- libcurl-7.24.0-5.fc17.x86_64
- libmpc-0.9-2.fc17.2.x86_64
- libunwind-1.0.1-3.fc17.x86_64
- libusal-1.1.11-14.fc17.x86_64
- libvirt-0.9.11.7-1.fc17.x86_64
- libvirt-client-0.9.11.7-1.fc17.x86_64
- libvirt-daemon-0.9.11.7-1.fc17.x86_64
- libvirt-daemon-config-network-0.9.11.7-1.fc17.x86_64
- libvirt-daemon-config-nwfilter-0.9.11.7-1.fc17.x86_64
- libvirt-python-0.9.11.7-1.fc17.x86_64
- libwsman1-2.2.7-5.fc17.x86_64
- lzop-1.03-4.fc17.x86_64
- m2crypto-0.21.1-8.fc17.x86_64
- mod_wsgi-3.3-2.fc17.x86_64
- mx-3.2.3-1.fc17.x86_64
- mysql-5.5.28-1.fc17.x86_64
- mysql-libs-5.5.28-1.fc17.x86_64
- MySQL-python-1.2.3-5.fc17.x86_64
- mysql-server-5.5.28-1.fc17.x86_64
- netcf-libs-0.2.2-1.fc17.x86_64
- numad-0.5-4.20120522git.fc17.x86_64
- numpy-1.6.2-1.fc17.x86_64
- parted-3.0-10.fc17.x86_64
- perl-AnyEvent-5.27-7.fc17.noarch
- perl-AnyEvent-AIO-1.1-8.fc17.noarch
- perl-AnyEvent-BDB-1.1-7.fc17.noarch
- perl-Async-MergePoint-0.03-7.fc17.noarch
- perl-BDB-1.88-5.fc17.x86_64
- perl-common-sense-3.5-1.fc17.noarch
- perl-Compress-Raw-Bzip2-2.052-1.fc17.x86_64
- perl-Compress-Raw-Zlib-2.052-1.fc17.x86_64
- perl-Config-General-2.50-6.fc17.noarch
- perl-Coro-6.07-3.fc17.x86_64
- perl-Curses-1.28-5.fc17.x86_64
- perl-DBD-MySQL-4.020-2.fc17.x86_64
- perl-DBI-1.617-1.fc17.x86_64
- perl-Encode-Locale-1.02-5.fc17.noarch
- perl-Error-0.17016-7.fc17.noarch
- perl-EV-4.03-8.fc17.x86_64
- perl-Event-1.20-1.fc17.x86_64
- perl-Event-Lib-1.03-16.fc17.x86_64
- perl-Git-1.7.11.7-1.fc17.noarch
- perl-Glib-1.241-2.fc17.x86_64
- perl-Guard-1.022-1.fc17.x86_64
- perl-Heap-0.80-10.fc17.noarch
- perl-HTML-Parser-3.69-3.fc17.x86_64
- perl-HTML-Tagset-3.20-10.fc17.noarch
- perl-HTTP-Date-6.00-3.fc17.noarch
- perl-HTTP-Message-6.03-1.fc17.noarch
- perl-IO-AIO-4.15-1.fc17.x86_64
- perl-IO-Async-0.29-7.fc17.noarch
- perl-IO-Compress-2.052-1.fc17.noarch
- perl-IO-Socket-SSL-1.66-1.fc17.noarch
- perl-IO-Tty-1.10-5.fc17.x86_64
- perl-LWP-MediaTypes-6.01-4.fc17.noarch
- perl-Net-HTTP-6.02-2.fc17.noarch
- perl-Net-LibIDN-0.12-8.fc17.x86_64
- perl-Net-SSLeay-1.48-1.fc17.x86_64
- perl-POE-1.350-2.fc17.noarch
- perl-Socket6-0.23-8.fc17.x86_64
- perl-Socket-GetAddrInfo-0.19-1.fc17.x86_64
- perl-TermReadKey-2.30-14.fc17.x86_64
- perl-TimeDate-1.20-6.fc17.noarch
- perl-URI-1.60-1.fc17.noarch
- ppl-0.11.2-8.fc17.x86_64
- ppl-pwl-0.11.2-8.fc17.x86_64
- pylint-0.25.1-1.fc17.noarch
- python-amqplib-1.0.2-3.fc17.noarch
- python-anyjson-0.3.1-3.fc17.noarch
- python-babel-0.9.6-3.fc17.noarch
- python-BeautifulSoup-3.2.1-3.fc17.noarch
- python-boto-2.5.2-1.fc17.noarch
- python-carrot-0.10.7-4.fc17.noarch
- python-cheetah-2.4.4-2.fc17.x86_64
- python-cherrypy-3.2.2-1.fc17.noarch
- python-coverage-3.5.1-0.3.b1.fc17.x86_64
- python-crypto-2.6-1.fc17.x86_64
- python-dateutil-1.5-3.fc17.noarch
- python-devel-2.7.3-7.2.fc17.x86_64
- python-docutils-0.8.1-3.fc17.noarch
- python-eventlet-0.9.17-1.fc17.noarch
- python-feedparser-5.1.2-2.fc17.noarch
- python-gflags-1.5.1-2.fc17.noarch
- python-greenlet-0.3.1-11.fc17.x86_64
- python-httplib2-0.7.4-6.fc17.noarch
- python-iso8601-0.1.4-4.fc17.noarch
- python-jinja2-2.6-2.fc17.noarch
- python-kombu-1.1.3-2.fc17.noarch
- python-lockfile-0.9.1-2.fc17.noarch
- python-logilab-astng-0.23.1-1.fc17.noarch
- python-logilab-common-0.57.1-2.fc17.noarch
- python-lxml-2.3.5-1.fc17.x86_64
- python-markdown-2.1.1-1.fc17.noarch
- python-migrate-0.7.2-2.fc17.noarch
- python-mox-0.5.3-4.fc17.noarch
- python-netaddr-0.7.5-4.fc17.noarch
- python-nose-1.1.2-2.fc17.noarch
- python-paramiko-1.7.7.1-2.fc17.noarch
- python-paste-deploy-1.5.0-4.fc17.noarch
- python-paste-script-1.7.5-4.fc17.noarch
- python-pep8-1.0.1-1.fc17.noarch
- python-pip-1.0.2-2.fc17.noarch
- python-pygments-1.4-4.fc17.noarch
- python-qpid-0.18-1.fc17.noarch
- python-routes-1.12.3-3.fc17.noarch
- python-setuptools-0.6.27-2.fc17.noarch
- python-sphinx-1.1.3-1.fc17.noarch
- python-sqlalchemy-0.7.9-1.fc17.x86_64
- python-suds-0.4.1-2.fc17.noarch
- python-tempita-0.5.1-1.fc17.noarch
- python-unittest2-0.5.1-3.fc17.noarch
- python-virtualenv-1.7.1.2-2.fc17.noarch
- python-webob-1.1.1-2.fc17.noarch
- python-wsgiref-0.1.2-8.fc17.noarch
- pyxattr-0.5.1-1.fc17.x86_64
- PyYAML-3.10-3.fc17.x86_64
- qemu-common-1.0.1-2.fc17.x86_64
- qemu-img-1.0.1-2.fc17.x86_64
- qemu-system-x86-1.0.1-2.fc17.x86_64
- qpid-cpp-client-0.18-5.fc17.x86_64
- qpid-cpp-server-0.18-5.fc17.x86_64
- radvd-1.8.5-3.fc17.x86_64
- screen-4.1.0-0.9.20120314git3c2946.fc17.x86_64
- scsi-target-utils-1.0.24-6.fc17.x86_64
- seabios-bin-1.7.1-1.fc17.noarch
- sg3_utils-1.31-2.fc17.x86_64
- sgabios-bin-0-0.20110622SVN.fc17.noarch
- spice-server-0.10.1-5.fc17.x86_64
- sqlite-3.7.11-3.fc17.x86_64
- tcpdump-4.2.1-3.fc17.x86_64
- vgabios-0.6c-4.fc17.noarch
- wget-1.13.4-7.fc17.x86_64
- xen-libs-4.1.3-5.fc17.x86_64
- xen-licenses-4.1.3-5.fc17.x86_64
Python packages installed (with PIP)
These all ended up in /usr/lib/python2.7/site-packages :-(
- WebOp
- amqplib
- boto (splattering over existing boto RPM package with older version)
- cinderclient
- cliff
- cmd2
- compressor
- django_appconf
- django_compressor
- django_openstack_auth
- glance
- horizon
- jsonschema
- keyring
- keystoneclient
- kombu
- lockfile
- nova
- openstack_auth
- pam
- passlib
- prettytable
- pyparsing
- python-cinderclient
- python-glanceclient
- python-novaclient
- python-openstackclient
- python-quantumclient
- python-swiftclient
- pytz
- quantumclient
- suds
- swiftclient
- warlock
- webob
Files changed
- /etc/group (added $USER to ‘libvirtd’ group)
- /etc/gshadow (as above)
- /etc/httpd/conf/httpd.conf (changes Listen 80 to Listen 0.0.0.0:80)
- /usr/lib/python2.7/site-packages/boto (due to overwriting RPM provided boto)
- /usr/bin/cq (as above)
- /usr/bin/elbadmin (as above)
- /usr/bin/list_instances (as above)
- /usr/bin/lss3 (as above)
- /usr/bin/route53 (as above)
- /usr/bin/s3multiput (as above)
- /usr/bin/s3put (as above)
- /usr/bin/sdbadmin (as above)
Files created
- /etc/cinder/*
- /etc/glance/*
- /etc/keystone/*
- /etc/nova/*
- /etc/httpd/conf.d/horizon.conf
- /etc/polkit-1/localauthority/50-local.d/50-libvirt-reomte-access.pkla
- /etc/sudoers.d/50_stack_sh
- /etc/sudoers.d/cinder-rootwrap
- /etc/sudoers.d/nova-rootwrap
- $DEST/cinder/*
- $DEST/data/*
- $DEST/glance/*
- $DEST/horizon/*
- $DEST/keystone/*
- $DEST/noVNC/*
- $DEST/nova/*
- $DEST/python-cinderclient/*
- $DEST/python-glanceclient/*
- $DEST/python-keystoneclient/*
- $DEST/python-novaclient/*
- $DEST/python-openstackclient/*
- /usr/bin/cinder*
- /usr/bin/glance*
- /usr/bin/keystone*
- /usr/bin/nova*
- /usr/bin/openstack
- /usr/bin/quantum
- /usr/bin/swift
- /var/cache/cinder/*
- /var/cache/glance/*
- /var/cache/keystone/*
- /var/cache/nova/*
- /var/lib/mysql/cinder/*
- /var/lib/mysql/glance/*
- /var/lib/mysql/keystone/*
- /var/lib/mysql/mysql/*
- /var/lib/mysql/nova/*
- /var/lib/mysql/performance_schema/*
Thoughts on installation
As we can see from the details above, DevStack does a very significant amount of work as root using sudo. I had fully expected that it was installing RPMs as root, but I had not counted on it adding extra python modules into /usr/lib/python-2.7, nor the addition of files in /etc/, /var or /usr/bin. I had set the $DEST environment variable for DevStack naively assuming that it would cause it to install everything possible under that location. In fact the $DEST variable was only used to control where the GIT checkouts of each openstack component went, along with a few misc files in $DEST/files/
IMHO a development environment setup tool should do as little as humanely possible as root. From the above list of changes, the only things that I believe justify use of sudo privileges are:
- Installation of RPMs from standard YUM repositories
- Installation of /etc/sudoers.d/ config files
- Installation of /etc/polkit file to grant access to libvirtd
Everything else is capable of being 100% isolated from the rest of the OS, under the $DEST directory location. Taking that into account my preferred development setup would be
$DEST
+- nova (GIT checkout)
+- ...etc... (GIT checkout)
+- vroot
+- bin
| +- nova
| +- ...etc...
+- etc
| +- nova
| +- ...etc...
+- lib
| +- python2.7
| +- site-packages
| +- boto
| +- ...etc...
+- var
+- cache
| +- nova
| +- ...etc...
+- lib
+- mysql
This would imply running a private copy of qpid, mysql and httpd, ideally all inside the same screen session as the rest of the OpenStack services, using unprivileged ports. Even if we relied on the system instances of qpid, mysql, httpd and did a little bit more privileged config, 95% of the rest of the stuff DevStack does as root, could still be kept unprivileged. I am also aware that current OpenStack code may not be amenable to installation in locations outside of / by default, but the code is all there to be modified to cope with arbitrary install locations if desired/required.
My other wishlist item for DevStack would be for it to print output that is meaningful to the end user running it. Simply printing a verbose list of every single shell command executed is one of the most unkind things you can do to a user. I’d like to see
# ./devstack.sh
* Cloning GIT repositories
- 1/10 nova
- 2/10 cinder
- 3/10 quantum
* Installing RPMs using YUM
- 1/30 python-pip
- 2/30 libvirt
- 3/30 libvirt-client
* Installing Python packages to $DEST/vroot/lib/python-2.7/site-packages using PIP
- 1/24 WebOb
- 2/24 ampqlib
- 3/24 boto
* Creating database schemas
- 1/10 nova
- 2/10 cinder
- 3/10 quantum
By all means still save the full list of every shell command and their output to a ‘devstack.log’ file for troubleshooting when things go wrong.
When first getting involved in the OpenStack project as a developer, most people will probably recommend use of DevStack. When I first started hacking, I skipped this because it wasn’t reliable on Fedora at that time, but these days it works just fine and there are even basic instructions for DevStack on Fedora. Last week I decided to finally give DevStack a go, since my hand-crafted dev environment was getting kind of nasty. The front page on the DevStack website says it is only supported on Fedora 16, but don’t let that put you off; aside from one bug which does not appear distro specific, it all seemed to work correctly. What follows is an overview of what I did / learnt
Setting up the virtual machine
I don’t like really like letting scripts like DevStack mess around with my primary development environment, particularly when there is little-to-no-documentation about what changes they will be making and they ask for unrestricted sudo (sigh) privileges ! Thus running DevStack inside a virtual machine was the obvious way to go. Yes, this means actual VMs run by Nova will be forced to use plain QEMU emulation (or nested KVM if you are brave), but for dev purposes this is fine, since the VMs don’t need todo anything except boot. My host is Fedora 17, and for simplicity I decided that my guest dev environment will also be Fedora 17. With that decided installing the guest was a simple matter of running virt-install on the host as root
# virt-install --name f17x86_64 --ram 2000 --file /var/lib/libvirt/images/f17x86_64.img --file-size 20 --accelerate --location http://mirror2.hs-esslingen.de/fedora/linux//releases/17/Fedora/x86_64/os/ --os-variant fedora17
I picked the defaults for all installer options, except for reducing the swap file size down to a more sensible 500 MB (rather than the 4 G it suggested). NB if copying this, you probably want to change the URL used to point to your own best mirror location.
Once installation completed, run through the firstboot wizard, creating yourself an unprivileged user account, then login as root. First add the user to the wheel group, to enable it to run sudo commands:
# gpasswd -a YOURUSERNAME wheel
The last step before getting onto DevStack is to install GIT
# yum -y install git
Setting up DevStack
The recommended way to use DevStack, is to simply check it out of GIT and run the latest code available. I like to keep all my source code checkouts in one place, so I’m using $HOME/src/openstack for this project
$ mkdir -p $HOME/src/openstack
$ cd $HOME/src/openstack
$ git clone git://github.com/openstack-dev/devstack.git
Arguably you can now just kick off the stack.sh script at this point, but there are some modifications that are a good idea to do. This involves creating a “localrc” file in the top level directory of the DevStack checkout
$ cd devstack
$ cat > localrc <<EOF
# Stop DevStack polluting /opt/stack
DESTDIR=$HOME/src/openstack
# Switch to use QPid instead of RabbitMQ
disable_service rabbit
enable_service qpid
# Replace with your primary interface name
HOST_IP_IFACE=eth0
PUBLIC_INTERFACE=eth0
VLAN_INTERFACE=eth0
FLAT_INTERFACE=eth0
# Replace with whatever password you wish to use
MYSQL_PASSWORD=badpassword
SERVICE_TOKEN=badpassword
SERVICE_PASSWORD=badpassword
ADMIN_PASSWORD=badpassword
# Pre-populate glance with a minimal image and a Fedora 17 image
IMAGE_URLS="http://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-uec.tar.gz,http://berrange.fedorapeople.org/images/2012-11-15/f17-x86_64-openstack-sda.qcow2"
EOF
With the localrc created, now just kick off the stack.sh script
$ ./stack.sh
At time of writing there is a bug in DevStack which will cause it to fail to complete correctly – it is checking for existence paths before it has created them. Fortunately, just running it for a second time is a simple workaround
$ ./unstack.sh
$ ./stack.sh
From a completely fresh Fedora 17 desktop install, stack.sh will take a while to complete, as it installs a large number of pre-requisite RPMs and downloads the appliance images. Once it has finished it should tell you what URL the Horizon web interface is running on. Point your browser to it and login as “admin” with the password provided in your localrc file earlier.
Because we told DevStack to use $HOME/src/openstack as the base directory, a small permissions tweak is needed to allow QEMU to access disk images that will be created during testing.
$ chmod o+rx $HOME
Note, that SELinux can be left ENFORCING, as it will just “do the right thing” with the VM disk image labelling.
UPDATE: if you want to use the Horizon web interface, then you do in fact need to set SELinux to permissive mode, since Apache won’t be allowed to access your GIT checkout where the Horizon files live.
$ sudo su -
# setenforce 0
# vi /etc/sysconfig/selinux
...change to permissive...
UPDATE:If you want to use Horizon, you must also manually install Node.js from a 3rd party repositoryh, because it is not yet included in Fedora package repositories:
# yum localinstall --nogpgcheck http://nodejs.tchol.org/repocfg/fedora/nodejs-stable-release.noarch.rpm
# yum -y install nodejs nodejs-compat-symlinks
# systemctl restart httpd.service
Testing DevStack
Before going any further, it is a good idea to make sure that things are operating somewhat normally. DevStack has created an file containing the environment variables required to communicate with OpenStack, so load that first
$ . openrc
Now check what images are available in glance. If you used the IMAGE_URLS example above, glance will have been pre-populated
$ glance image-list
+--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
+--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+
| 32b06aae-2dc7-40e9-b42b-551f08e0b3f9 | cirros-0.3.0-x86_64-uec-kernel | aki | aki | 4731440 | active |
| 61942b99-f31c-4155-bd6c-d51971d141d3 | f17-x86_64-openstack-sda | qcow2 | bare | 251985920 | active |
| 9fea8b4c-164b-4f54-8e74-b53966e858a6 | cirros-0.3.0-x86_64-uec-ramdisk | ari | ari | 2254249 | active |
| ec3e9b72-0970-44f2-b442-58d0042448f7 | cirros-0.3.0-x86_64-uec | ami | ami | 25165824 | active |
+--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+
Incidentally the glance sort ordering is less than helpful here – it appears to be sorting based on the UUID strings rather than the image names :-(
Before booting a instance, Nova likes to be given an SSH public key, which it will inject into the guest filesystem to allow admin login
$ nova keypair-add --pub-key $HOME/.ssh/id_rsa.pub mykey
Finally an image can be booted
$ nova boot --key-name mykey --image f17-x86_64-openstack-sda --flavor m1.tiny f17demo1
+------------------------+--------------------------------------+
| Property | Value |
+------------------------+--------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | NsddfbJtR6yy |
| config_drive | |
| created | 2012-11-19T15:00:51Z |
| flavor | m1.tiny |
| hostId | |
| id | 6ee509f9-b612-492b-b55b-a36146e6833e |
| image | f17-x86_64-openstack-sda |
| key_name | mykey |
| metadata | {} |
| name | f17demo1 |
| progress | 0 |
| security_groups | [{u'name': u'default'}] |
| status | BUILD |
| tenant_id | dd3d27564c6043ef87a31404aeb01ac5 |
| updated | 2012-11-19T15:00:55Z |
| user_id | 72ae640f50434d07abe7bb6a8e3aba4e |
+------------------------+--------------------------------------+
Since we’re running QEMU inside a KVM guest, booting the image will take a little while – several minutes or more. Just keep running the ‘nova list’ command to keep an eye on it, until it shows up as ACTIVE
$ nova list
+--------------------------------------+----------+--------+------------------+
| ID | Name | Status | Networks |
+--------------------------------------+----------+--------+------------------+
| 6ee509f9-b612-492b-b55b-a36146e6833e | f17demo1 | ACTIVE | private=10.0.0.2 |
+--------------------------------------+----------+--------+------------------+
Just to prove that it really is working, login to the instance with SSH
$ ssh ec2-user@10.0.0.2
The authenticity of host '10.0.0.2 (10.0.0.2)' can't be established.
RSA key fingerprint is 9a:73:e5:1a:39:e2:f7:a5:10:a7:dd:bc:db:6e:87:f5.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.0.2' (RSA) to the list of known hosts.
[ec2-user@f17demo1 ~]$ sudo su -
[root@f17demo1 ~]#
Working with DevStack
The DevStack setup runs all the python services under a screen session. To stop/start individual services, attach to the screen session with the ‘rejoin-stack.sh’ script. Each service is running under a separate screen “window”. Switch to the window containing the service to be restarted, and just Ctrl-C it and then use bash history to run the same command again.
$ ./rejoin-stack.sh
Sometimes the entire process set will need to be restarted. In this case, just kill the screen session entirely, which causes all the OpenStack services to go away. Then the same ‘rejoin-stack.sh’ script can be used to start them all again.
One annoyance is that unless you have the screen session open, the debug messages from Nova don’t appear to end up anywhere useful. I’ve taken to editing the file “stack-screen” to make each service log to a local file in its checkout. eg I changed
stuff "cd /home/berrange/src/openstack/nova && sg libvirtd /home/berrange/src/openstack/nova/bin/nova-compute"
to
stuff "cd /home/berrange/src/openstack/nova && sg libvirtd /home/berrange/src/openstack/nova/bin/nova-compute 2>&1 | tee nova-compute.log"
