I’m pleased to announce the availability of a new release of gerrymander, version 1.3. Gerrymander provides a python command line tool and APIs for querying information from the gerrit review system, as used in OpenStack and many other projects. You can get it from pypi
# pip install gerrymander
Or straight from GitHub
# git clone git://github.com/berrange/gerrymander.git
If you’re the impatient type, then go to the README file which provides a quick start guide to using the tool.
This release contains a mixture of bug fixes and two new features. When displaying a list of changes, one of the fields that can be shown per-change is the approvals. This is rendered as a list of all the -2/-1/+1/+2 votes made against the current patch set. The text is also coloured to make it easier to tell at a glance what the overall state of the change is. There are two problems with this, first when there were a lot of votes on a change the list gets rather too wide. The bigger problem though has been the high level of false failures in the OpenStack CI testing system. This results in many patches receiving -1’s from testing, which caused gerrymander to colour them in red:
+-------------------------------------+-------------------------------------------------------+----------+-----------------------+
| URL | Subject | Created | Approvals |
+-------------------------------------+-------------------------------------------------------+----------+-----------------------+
| https://review.openstack.org/68942 | Power off commands should give guests a chance to ... | 186 days | w= v=1,1,1,1 c=-2,-1 |
| https://review.openstack.org/77027 | Support image property for config drive | 152 days | w= v=1,-1,-1,-1 c=-1 |
| https://review.openstack.org/82409 | Fixes a Hyper-V list_instances localization issue | 128 days | w= v=1,-1 c=-1 |
| https://review.openstack.org/88067 | Allow deleting instances while uuid lock is held | 104 days | w= v=1,1,1,1 c=2 |
| https://review.openstack.org/108013 | Fixes Hyper-V agent force_hyperv_utils_v1 flag iss... | 12 days | w= v=1,1,1,-1 c=1,1,1 |
My workflow is to focus on things which do not have negative feedback and so I found this was discouraging me from reviewing stuff that was only marked negative due to bogus CI failures. So in this new release, the display is now using separate columns to report test votes, code review votes and workflow votes, each column being separately coloured. Also, instead of showing each individual vote, we only show the so called “casting vote” – ie the one that’s most important, (order is -2, +2, -1, +1)
+-------------------------------------+-------------------------------------------------------+----------+-------+---------+----------+
| URL | Subject | Created | Tests | Reviews | Workflow |
+-------------------------------------+-------------------------------------------------------+----------+-------+---------+----------+
| https://review.openstack.org/68942 | Power off commands should give guests a chance to ... | 186 days | 1 | -2 | |
| https://review.openstack.org/77027 | Support image property for config drive | 152 days | -1 | -1 | |
| https://review.openstack.org/82409 | Fixes a Hyper-V list_instances localization issue | 128 days | -1 | -1 | |
| https://review.openstack.org/88067 | Allow deleting instances while uuid lock is held | 104 days | 1 | 2 | |
| https://review.openstack.org/108013 | Fixes Hyper-V agent force_hyperv_utils_v1 flag iss... | 12 days | -1 | 1 | |
The second new feature is the ‘patchreviewrates’ command which is reports on the review comment activity of people over time. We already have ‘patchreviewstats’ command which gives information about review activity over a fixed window, but this doesn’t let us see long term trends. With the new command we’re reporting on the daily number of review comments per person, averaging over a week, and reported for the last 52 weeks. This lets us see how review activity from contributors goes up and down over the course of a year (or 2 dev cycles). I used this to produced a report which I then imported to LibreOffice to create a graph showing the nova-core team activity over the past two cycles (click image to enlarge)
Nova core team review rates
In summary the changes in version 1.2 of gerrymander are
- Exclude own changes in the todo lists
- Add CSV as an output format for some reports
- Add patchreviewrate report for seeing historica approvals per day
- Replace ‘Approvals’ column with ‘Test’, ‘Review’ and ‘Workflow’ columns in change reports
- Allow todo lists to be filtered per branch
- Reorder sorting of votes to prioritize +2/-2s over +1/-1s
- Avoid exception from unexpected approval vote types
- Avoid creating empty cache file when Ctrl-C’ing ssh client
- Run ssh in batch mode to avoid hang when host key is unknown
Thanks to everyone who contributed patches that went into this new release
I’m pleased to announce the availability of a new release of gerrymander, version 1.2. Gerrymander provides a python command line tool and APIs for querying information from the gerrit review system, as used in OpenStack and many other projects. You can get it from pypi
# pip install gerrymander
Or straight from GitHub
# git clone git://github.com/berrange/gerrymander.git
If you’re the impatient type, then go to the README file which provides a quick start guide to using the tool.
The changes in version 1.2 are
- Don’t drop ‘Restored’ comments in ‘comments’ command
- Change ‘assertEquals’ to ‘assertEqual’
- Print list of options for ‘changes’ command in help message
- Send all output via a pager (eg less or whatever $PAGER says) Can disable via setting GERRYMANDER_PAGER=cat env var.
- Fix ordering of comments when > 10 patch sets are present
- Handle remaining event types
- Stop subprocesses getting signals intended for main process
- Improve colourization of review votes
Thanks to everyone who contributed patches that went into this new release
In the time off over Christmas and the New Year I decided to investigate & learn about the creation of GNOME Shell extensions. As an amateur astronomer, I have an interest in knowing what the “seeing conditions” will be like in the forthcoming nights. There are a number of different websites which provide forecasts, as well as apps for Android and iPhone. I use the Android AstroPanel application quite frequently, but most of the day I’m sitting in front of my laptop and would rather have the data presented there, alongside the regular weather forecast, rather than on my phone. So after finding that extensions are quite simple to create, I decided to create an extension for displaying an astronomical weather forecast for GNOME Shell.
The design of my extension is a cross between the well known Weather extension and the 7Timer/AstroPanel forecast display. Rather than describe it, just look at this screenshot:
AstroWeather GNOME Shell Extension
The actual weather forecast is obtained from the 7Timer REST service using the Soup HTTP client APIs, which conveniently returns JSON data perfect for accessing from JavaScript. The times for sunrise/sunset are calculated locally, using code that I derived from the BSD licensed suncalc project. Finally the locations are pulled from the GNOME GWeather database.
The first release of AstroWeather is available for installation with GNOME 3.10 from the GNOME Shell Extensions website. I didn’t realize that you had to list every single darn GNOME shell version explicitly in the metadata.json file, so while it works fine with GNOME 3.12, the extensions website will seemingly refuse to let you install it :-( I’ll upload a new version which explicitly says it supports 3.12 soon. There’s also a slight bug with sunrise/sunset calculations which cause it to display “Invalid Date” if there is no sunset/sunset time available – in the UK we’re far enough north that you never get any astronomical sunset in June, and I forgot to handle that possibility since I wrote the code when it was nice and dark :-)
The, GPLv2+ licensed, source code is available as a GIT repository on gitorious.org
As well as fixing bugs, in the future I’d like to make it possible to specify arbitrary locations with long+lat since the GWeather location database is very limited in its choices. Using some kind of GeoIP service would make it possible to just “do the right thing” for your current location. I’d also like to have it display the phase of the moon, inform you which planets will be visible overnight and list any ISS overhead passes. Finally I want to integrate with astronomical forecast data sources beyond 7Timer, since their REST API has sadly been having extended periods of downtime. Contributions from any other interested people are of course welcome, either patches via email or merge requests via gitorious.
I recently announced gerrymander, a python client tool and API for extracting information from Gerrit, intended to replace my use of various other ad-hoc tools that people have developed. One of the things I observed with using the previous qgerrit command line tool was that there were a couple of different sets of command line arguments that I used all the time. This was tedious so I ended up creating a number of shell wrapper scripts / aliases to invoke qgerrit with different sets of arguments. With gerrymander though, I wanted to do things a little better by defining this stuff as data rather than code. ie using configuration files, instead of wrapper scripts/aliases.
Taking the ‘changes’ command as an example, it is a generic command for displaying lists of changes in gerrit for a given query. The default behaviour is to output 7 columns: status, url, owner, subject, creation date, last updated date and approvals.
$ gerrymander changes -p openstack/nova-specs
Changes
-------
+-----------+------------------------------------+-----------------------------+-----------------------------------+----------+----------+-------------------------------+
| Status | URL | Owner | Subject | Created | Updated | Approvals |
+-----------+------------------------------------+-----------------------------+-----------------------------------+----------+----------+-------------------------------+
| MERGED | https://review.openstack.org/80785 | russellb | Add README and base directory ... | 68 days | 66 days | w=1 v=2 s=1 c=2,2 |
| NEW | https://review.openstack.org/80865 | mikalstill | Proposed blueprint for libvirt... | 67 days | 1 days | v=-1 c=1,1,-1,-1,-1,-1 |
| ABANDONED | https://review.openstack.org/80866 | mikalstill | Proposed blueprint for live mi... | 67 days | 55 days | v=1 c=-1,-1 |
| MERGED | https://review.openstack.org/81381 | jogo | Add Apache 2 License | 65 days | 63 days | w=1 v=2 s=1 c=2,1,2 |
...snip....
Personally I’m not too interested in the last updated date, status or owner most of the time, and want things sorted by last updated date. I want to give lots of space for the subject, have approvals to be displayed in colour and only display changes that are open and in the master branch. This can be done using command line arguments but it gets very tedious:
$ gerrymander changes -p openstack/nova-specs \
--field url --field subject:80 --field createdOn --field approvals \
--color --branch master --status open --sort createdOn
Changes
-------
+------------------------------------+-------------------------------------------------------------------------------------+----------+--------------------------+
| URL | Subject | Created | Approvals |
+------------------------------------+-------------------------------------------------------------------------------------+----------+--------------------------+
| https://review.openstack.org/80865 | Proposed blueprint for libvirt serial console work in juno. | 67 days | v=-1 c=1,1,-1,-1,-1,-1 |
| https://review.openstack.org/81828 | Scheduler: Adds per-aggregate filters | 63 days | v=1 c=1,1,-1,1 |
| https://review.openstack.org/82456 | Propose: Normalize Weights (adapt weighers) | 60 days | v=-1 c=-1 |
| https://review.openstack.org/82584 | Proposed blueprint for libvirt Sheepdog instances. | 59 days | v=1 c=1,2,1,1 |
...snip....
This output preference can be added to the $HOME/.gerrymander config file directly, avoiding the need to create a wrapper script
[command-changes]
field=url, subject:80, createdOn, approvals
branch=master
sort=createdOn
color=true
With the result that this is now the default
$ gerrymander changes -p openstack/nova-specs
Changes
-------
+------------------------------------+-------------------------------------------------------------------------------------+----------+--------------------------+
| URL | Subject | Created | Approvals |
+------------------------------------+-------------------------------------------------------------------------------------+----------+--------------------------+
| https://review.openstack.org/80865 | Proposed blueprint for libvirt serial console work in juno. | 67 days | v=-1 c=1,1,-1,-1,-1,-1 |
| https://review.openstack.org/81828 | Scheduler: Adds per-aggregate filters | 63 days | v=1 c=1,1,-1,1 |
| https://review.openstack.org/82456 | Propose: Normalize Weights (adapt weighers) | 60 days | v=-1 c=-1 |
| https://review.openstack.org/82584 | Proposed blueprint for libvirt Sheepdog instances. | 59 days | v=1 c=1,2,1,1 |
...snip....
Now the obvious limitation with setting defaults in the config file is that you might have many different sets of defaults you care about. For example when looking at nova-specs designs, I’d prefer the sorting to be done by subject and don’t really care about the author. The gerrymander config file copes with this too, by allowing you to setup command aliases:
[commands]
aliases=nova-specs, cinder-specs
[alias-nova-specs]
basecmd=changes
help=Nova design specs
[alias-cinder-specs]
basecmd=changes
help=Cinder design specs
These new commands are now first-class citizens on a par with any other built-in commands as far as gerrymander is concerned, so you can then define default settings for them independently.
[command-nova-specs]
project = openstack/nova-specs
status = open
field = url, subject:80, createdOn, approvals
sort = subject
color = true
[command-cinder-specs]
project = openstack/cinder-specs
status = open
field = url, subject:80, createdOn, approvals
sort = subject
color = true
Trying out the new nova-specs command…
$ gerrymander nova-specs
Changes
-------
+------------------------------------+-------------------------------------------------------------------------------------+----------+--------------------------+
| URL | Subject | Created | Approvals |
+------------------------------------+-------------------------------------------------------------------------------------+----------+--------------------------+
| https://review.openstack.org/95054 | API: Live Resize | 9 hours | v=1 |
| https://review.openstack.org/85726 | API: Metadata Service Callbacks | 45 days | v=1 c=1 |
| https://review.openstack.org/85673 | API: Proxy neutron configuration to guest instance | 24 days | v=1 c=1 |
| https://review.openstack.org/94918 | Add Barbican wrapper specification | 20 hours | v=1 |
| https://review.openstack.org/94370 | Add LVM ephemeral storage encryption specification | 2 days | v=1 c=1,1,-1 |
...snip....
This blog post has just looked at the ‘changes’ command, but any gerrymander command can be customized in this way – every single command line option is mapped into the configuration file in the same way.
Those who saw my original announcement of gerrymander will know that OpenStack uses the Gerrit tool for code review. One of the efforts made by the Nova (Compute) team during the last release cycle was to increase the automated testing coverage for hypervisors supported by Nova. Previously we only tested Libvirt on KVM, but now there is testing for VMWare, HyperV and XenAPI too. This is great for our code quality, but far less great for the usability of the gerrit web interface. Consider a fairly typical change I needed to look at today, the web UI for gerrit shows many comments. When doing code review of new patch versions though, I mostly don’t care about comments from the “robot” accounts doing automated testing – just the pass / fail status. I’m far more interested in comments made by humans, so I can see whether they’ve been addressed by new patch versions, or whether I agree/disagree with the comments. The gerrit web UI though doesn’t provide any way to separate this info – it just shows a huge list with human & robot comments combined. The ever growing number of robots used by OpenStack for testing mean the human comments are drowned out in the noise
This problem is one of the reasons I created the gerrymander tool. The ‘comments’ command has the ability to display all comments on a change, while filtering out those made by known bot accounts. Now there’s a little setup required, by which I mean you must edit $HOME/.gerrymander to list all the bot accounts. For those using OpenStack though, I’ve uploaded a sample config file, which lists all bot accounts, to the wiki. With this config file in effect I can now look at this same change number 87329
# gerrymander comments --color 87329
Change https://review.openstack.org/87329 (I06c2d9930e0f36a0d7057b6a0f5c9c591caac43f)
libvirt: Use os_command_line when kernel_id is set
Patch 1 (c8d1cf4559174777be4b42b68379cf78ccd8f382)
Jay Pipes: (jaypipes) nova/virt/libvirt/driver.py:3203
You can condense lines 3202-3203 to: guest.os_cmdline =
img_props.get('os_command_line')
Jay Pipes: (jaypipes)
Patch Set 1: I would prefer that you didn't merge this (1 inline comment)
Suggestion for consolidation inline, otherwise, looks good.
Vladan Popovic: (vladan)
Patch Set 1: Thanks for the review Jay.
Patch 2 (58323a7271e0f09689b94bc8632204c179934add)
Daniel Berrange: (berrange) /COMMIT_MSG:18
"AMI images" is a pretty obscure/non-obvious term. This is about images with
an explicit boot kernel set, so please say that explicitly and void the term
"AMI"
Daniel Berrange: (berrange)
Patch Set 2: I would prefer that you didn't merge this (1 inline comment)
Jay Pipes: (jaypipes)
Patch Set 2: recheck bug 1307344
Patch 3 (8f3ced280d80eaee69029f4bf2a193bcba749284)
garyk: (garyk) nova/tests/virt/libvirt/test_libvirt.py:1961
please use self.assertIsNone
Mohammed Naser: (mnaser)
Patch Set 3: Looks good to me, but someone else must approve
garyk: (garyk)
Patch Set 3: I would prefer that you didn't merge this (1 inline comment)
code looks good. one minor nit
Patch 4 (5698a6c7c5195bd1d3ecf01af6fb65c19ae8a990)
No comments
Patch 5 (a87773776a0d39aa5591fde6caa65b92f5b17a6d)
Jay Pipes: (jaypipes)
Patch Set 5: Looks good to me, but someone else must approve ++, thx Vladlan!
Mohammed Naser: (mnaser)
Patch Set 5: Looks good to me, but someone else must approve recheck bug
1307344
Daniel Berrange: (berrange)
Patch Set 5: I would prefer that you didn't merge this Jenkins failures are
genuine bugs
Sreeram Yerrapragada: (syerrapragada)
Patch Set 5: recheck-vmware
Patch 6 (a1eb12f0c8281c0b01dde00b8225d742b8e832e3)
garyk: (garyk)
Patch Set 6: Code-Review+1
The obvious limitation here is that the file comments are not shown inline with the code, but overall I think this is still much more useful than the gerrit web UI display of comments. I can still go look at the web UI if I do need to see the context of certain inline comments. So gerrymander comments report complements the web UI nicely.
