Today Ohloh finished importing the Launchpad source code and produced the first source code analysis report. There seems to be something fishy about the reported line counts (e.g. -3,291 lines of SQL), but the commit counts and contributor list look about right. If you're interested in what sort of effort goes into producing an application like Launchpad, then it is worth a look.

---

Comments:

e - 27 Oct, 2009

Have you seen the perl language?

One of the nice features of Bazaar is the ability to send a bundle of changes to someone via email. If you use a supported mail client, it will even open the composer with the changes attached. If your client isn't supported, then it'll let you compose a message in your editor and then send it to an SMTP server.

GMail is not a supported mail client, but there are a few work arounds listed on the wiki. Those really come down to using an alternative mail client (either the editor or Mutt) and sending the mails through the GMail SMTP server. Neither solution really appealed to me. There doesn't seem to be a programatic way of opening up GMail's compose window and adding an attachment (not too surprising for a web app).

One thing that has been mentioned in the GNOME DVCS debate was that it is as easy to do "git diff" as it is to do "svn diff" so the learning curve issue is moot. I'd have to disagree here.

Traditional Centralised Version Control

With traditional version control systems (e.g. CVS and Subversion) as used by Free Software projects like GNOME, there are effectively two classes of users that I will refer to as "committers" and "patch contributors":

Yesterday, a BoF was scheduled for discussion of distributed version control systems with GNOME. The BoF session did not end up really discussing the issues of what GNOME needs out of a revision control system, and some of the examples Federico used were a bit snarky.

We had a more productive meeting in the session afterwards where we went over some of the concrete goals for the system. The list from the blackboard was:

It has been a while coming, but MySQL has announced their move to Bazaar for version control. This has been a long time coming, and it is great to finally see it announced publicly.

The published Bazaar branches include 8 years of history going back to MySQL 3.23.22, imported from the BitKeeper repositories. So you can see a lot more than just the history since the switch: you can use all the normal Bazaar tools to see where the code came from and how it evolved. Giuseppe Maxia has posted some instructions on how to check out the code for those who are interested.

One of the features I recently discovered in Bazaar is the --author option for "bzr commit". This lets you make commits to a Bazaar branch on behalf of another person. When used, the new revision credits two people: you as the committer and the other person as the author.

While Bazaar does make it easy for non-core contributors to send changes in a form that correctly attributes them (e.g. by publishing a branch or sending a bundle), I doubt we'll ever see the end of pure patches. Some cases include:

Last week we moved psycopg from Subversion to Bazaar. I did the migration using Gustavo Niemeyer's svn2bzr tool with a few tweaks to map the old Subversion committer IDs to the email address form conventionally used by Bazaar.

The tool does a good job of following tree copies and create related Bazaar branches. It doesn't have any special handling for stuff in the tags/ directory (it produces new branches, as it does for other tree copies). To get real Bazaar tags, I wrote a simple post-processing script to calculate the heads of all the branches in a tags/ directory and set them as tags in another branch (provided those revisions occur in its ancestry). This worked pretty well except for a few revisions synthesised by a previous cvs2svn migration. As these tags were from pretty old psycopg 1 releases I don't know how much it matters.

While doing a bit of work on Storm, I decided to try out the loom plugin for Bazaar. The loom plugin is designed to help maintain a stack of changes to a base branch (similar to quilt). Some use cases where this sort of tool are useful include:

1. Maintaining a long-running diff to a base branch. Distribution packaging is one such example.
2. While developing a new feature, the underlying code may require some refactoring. A loom could be used to keep the refactoring separate from the feature work so that it can be merged ahead of the feature.
3. For complex features, code reviewers often prefer to changes to be broken down into a sequence of simpler changes. A loom can help maintain the stack of changes in a coherent fashion.

A loom branch helps to manage these different threads in a coherent manner. Each thread in the loom contains all the changes from the threads below it, so the revision graph ends up looking something like this:

When working on my bzr-avahi plugin, Robert asked me about how it should fit in with his bzr-dbus plugin. The two plugins offer complementary features, and could share a fair bit of infrastructure code. Furthermore, by not cooperating, there is a risk that the two plugins could break when both installed together.

Given the dependencies of the two packages, it made more sense to put common infrastructure in bzr-dbus and have bzr-avahi depend on it. That said, bzr-dbus is a bit more difficult to install than bzr-avahi, since it requires installation of a D-Bus service activation file. After looking at the code, it seemed that there was room to simplify how bzr-dbus worked and improve its reliability at the same time.

At Canonical, one of the approaches taken to accelerate development is to hold coding sprints (otherwise known as hackathons, hackfests or similar). Certain things get done a lot quicker face to face compared to mailing lists, IRC or VoIP.

When collaborating with someone at one of these sprints the usual way to let others look at my work would be to commit the changes so that they could be pulled or merged by others. With legacy version control systems like CVS or Subversion, this would generally result in me uploading all my changes to a server in another country only for them to be downloaded back to the sprint location by others.

Elijah has been writing an interesting series of articles comparing different version control systems. While the previous articles have been very informative, I think the latest one was a bit muddled. What follows is an expanded version of my comment on that article.

Elijah starts by making an analogy between text editors and version control systems, which I think is quite a useful analogy. When working with a text editor, there is a base version of the file on disk, and the version you are currently working on which will become the next saved version.

One useful feature of Bazaar is the ability to cryptographically sign revisions. I was discussing this with Ryan on IRC, and thought I'd write up some of the details as they might be useful to others.

Anyone who remembers the past security of GNOME and Debian servers should be able to understand the benefits of being able to verify the integrity of a source code repository after such an incident. Rather than requiring all revisions made since the last known safe backup to be examined, much of the verification could be done mechanically.

In my previous article, I outlined Bazaar's bundle feature. This article describes how the Bazaar developers use bundles as part of their development and code review process.

Proposed changes to Bazaar are generally posted as patches or bundles to the development mailing list. Each change is discussed on the mailing list (often going through a number of iterations), and ultimately approved or rejected by the core developers. To aide in managing these patches Aaron Bentley (one of the developers wrote a tool called Bundle Buggy.

This article follows on from the series of tutorials on using Bazaar that I have neglected for a while. This article is about the bundle feature of Bazaar. Bundles are to Bazaar branches what patches are to tarballs or plain source trees.

Context/unified diffs and the patch utility are arguably one of most important inventions that enable distributed development:

• The patch is a self contained text file, making it easy to send as an email attachment or attach to a bug report.
• The size of the patch is proportional to the size of the changes rather than the size of the source tree. So submitting a one line fix to the Linux kernel is as easy as a one line fix for a small one person project.
• Even if the destination source tree has moved forward since the patch was created, the patch utility does a decent job of applying the changes using heuristics to match the surrounding context. Human intervention is only needed if the edits are to the same section of code.
• As patches are human readable text files, they are a convenient form to review the code changes.

Of course, patches do have their limitations:

Previously, I posted about the FM radio plugin I was working on. I just posted the code to bug 168735. A few notes about the implementation:

• The code only supports Video4Linux 2 radio tuners (since that’s the interface my device supports, and the V4L1 compatibility layer doesn’t work for it). It should be possible to port it support both protocols if someone is interested.
• It does not pass the audio through the GStreamer pipeline. Instead, you need to configure your mixer settings to pass the audio through (e.g. unmute the Line-in source and set the volume appropriately). It plugs in a GStreamer source that generates silence to work with the rest of the Rhythmbox infrastructure. This does mean that the volume control and visualisations won’t work
• No properties dialog yet. If you want to set titles on the stations, you’ll need to edit rhythmdb.xml directly at the moment.
• The code assumes that the radio device is /dev/radio0.

Other than that, it all works quite well (I've been using it for the last few weeks).

When at conferences and sprints, I often want to see what someone else is working on, or to let other people see what I am working on. Usually we end up pushing up to a shared server and using that as a way to exchange branches. However, this can be quite frustrating when competing for outside bandwidth when at a conference.

It is possible to share the branch from a local web server, but that still means you need to work out the addressing issues.

While browsing the log of one of my Bazaar branches, I noticed that the commit messages were being recorded as occurring in the +0800 time zone even though WA switched over to daylight savings.

Bazaar stores commit dates as a standard UNIX seconds since epoch value and a time zone offset in seconds. So the problem was with the way that time zone offset was recorded. The code in bzrlib that calculates the offset looks like this:

In my previous entry, I mentioned that Andrew was actually publishing the contents of all his Bazaar branches with his rsync script, even though he was only advertising a single branch. Yesterday I had a need to actually do this, so I thought I'd detail how to do it.

As a refresher, a Bazaar repository stores the revision graph for the ancestry of all the branches stored inside it. A branch is essentially just a pointer to the head revision of a particular line of development. So if the branch has been deleted but the data is still in the repository, recovering it is a simple matter of discovering the identifier for the head revision.

This article responds to some of the points in Andrew's post about Pushing a bzr branch with rsync.

bzr rspush and shared repositories

First of all, to understand why bzr rspush refuses to operate on a non-standalone branch, it is worth looking at what it does:

1. Download the revision history of the remote branch, and check to see that the remote head revision is an ancestor of the local head revision. If it is not, error out.
2. If it is an ancestor, use rsync to copy the local branch and repository information to the remote location.

Now if you bring shared repositories into the mix, and there is a different set of branches in the local and remote repositories, then step (2) is liable to delete revision information needed by those branches that don't exist locally. This is not a theoretical concern if you do development from multiple machines (e.g. a desktop and a laptop) and publish to the same repository.

One of the things we've been working on for Launchpad is good integration with Bazaar. Launchpad provides a way to register or host Bazaar branches, and nominate a Bazaar branch as representing a particular product series.

For each registered branch, there is a branch information page. This leads to a bit of confusion since Bazaar uses URLs to identify branches, so people try running bzr branch on a branch information page. We also get people trying to branch the product or product series pages.

When outlining the use of team branches on Launchpad previously, I used the --create-prefix option when pushing the branch to sftp://bazaar.launchpad.net. This was to make sure the initial push would succeed, even if the /\~username/product directory the branch would be created in didn't exist.

To simplify things for users, we made a change to the SFTP server in the latest release, so that --create-prefix is no longer necessary. This does not affect the allowed branch directories though: the structure is used to associate the branches with products, and decide who can write to the branches.

Over the weekend, I released gnome-gpg 0.5.0. The main features in this release is support for running without gnome-keyring-daemon (of course, you can't save the passphrase in this mode), and to use the same keyring item name for the passphrase as Seahorse. The release can be downloaded here:

http://download.gnome.org/sources/gnome-gpg/0.5/

I also switched over from Arch to Bazaar. The conversion was fairly painless using bzr baz-import-branch, and means that I have both my revisions and Colins revisions in a single tree. The branch can be pulled from:

Earlier, David Allouche described how to host Bazaar branches on Launchpad. At the end, he alluded to the ability to create branches that can be committed to by anyone on a team. I'll describe how this works here.

Launchpad Teams

Launchpad allows people to organise themseleves into teams. Most of the things people can do in Launchpad can also be done by teams, including owning branches.

You can create a new team at the following page:

Have you wanted to play around with bzr but had nowhere to share your branches? You can now publish them through Launchpad. David Allouche provides the details.

In short, you can upload branches to sftp://bazaar.launchpad.net/, and they will be published on http://bazaar.launchpad.net/.

I've been doing most JHBuild development in my bzr branch recently. If you have bzr 0.8rc1 installed, you can grab it here:

bzr branch http://www.gnome.org/~jamesh/bzr/jhbuild/jhbuild.dev

I've been keeping a regular CVS import going at http://www.gnome.org/~jamesh/bzr/jhbuild/jhbuild.cvs using Tailor, so changes people make to module sets in CVS make there way into the bzr branch. I've used a small hack so that merges back into CVS get recorded correctly in the jhbuild.cvs branch:

One of the new features in the soon to be released bzr 0.8 is the new "knit" storage format.

When comparing the size of the repository data for jhbuild with "knit" and "metadir" formats (metadir is just the old storage format with repository, branch and checkout bookkeeping separated), I see the following:

                metadir   knit

---

Size 9.9MB 5.5MB Number of files 1267 307

The reason for the smaller number of files is that information about all revisions in the repository is now stored together rather than in separate files. So the file count comes out at a constant plus 2 times the number of tracked files (a knit index file plus the knit data file). For comparison, the CVS repository I imported this from was 4.4MB, and comprised 143 files.

One of the new features comming up in the next release of bzr is support for shared repositories. This provides a way to reduce disk space needed to store multiple related branches. To understand how repositories work, it helps to know a bit about how branches are stored by bzr.

There are three concepts that make up a bzr branch:

1. A checkout or working tree. This is the source files you are working with. It represents the state of the source code at some recorded revision plus any local changes you've made. In the diagram on the right, it is represented as the red node.
2. The branch, consisting of a linear sequence of revisions. This is represented by the blue nodes in the diagram. Note that there may be multiple paths from the first revision to the current revision due to branching and merging. The branch revision history indicates the path that was taken by this particular branch.
3. The repository, being a store of the text of all the revisions in the ancestry of the branch, plus metadata about those revisions. This essentially stores information about every node and edge in the diagram.

In previous versions of bzr, this information was not clearly separated. However with the new default branch format in bzr 0.8 they are separated, and a particular directory need not contain all three parts, which is what makes the space savings and performance improvements possible.

In my previous post, I mentioned using Tailor to import jhbuild into a Bazaar-NG branch. In case anyone else is interested in doing the same, here are the steps I used:

1. Install the tools

First create a working directory to perform the import, and set up tailor. I currently use the nightly snapshots of bzr, which did not work with Tailor, so I also grabbed bzr-0.7:

$ wget http://darcs.arstecnica.it/tailor-0.9.20.tar.gz
$ wget http://www.bazaar-ng.org/pkg/bzr-0.7.tar.gz
$ tar xzf tailor-0.9.20.tar.gz
$ tar xzf bzr-0.7.tar.gz
$ ln -s ../bzr-0.7/bzrlib tailor-0.9.20/bzrlib

2. Prepare a local CVS Repository to import from

As most people probably know, the Gnome project is planning a migration to Subversion. In contrast, I've decided to move development of jhbuild over to bzr. This decision is a bit easier for me than for other Gnome modules because:

• No need to coordinate with GDP or GTP, since I maintain the docs and there is no translations.
• Outside of the moduleset definitions, the large majority of development and commits are done by me.
• There aren't really any interesting branches other than the mainline.

I plan to leave the Gnome module set definitions in CVS/Subversion though, since many people help in keeping them up to date, so leaving them there has some value.

I updated my bzr openssh plugin to be a proper patch against bzr.dev, and got it merged. So if you have bzr-openssh-sftp.py in your ~/.bazaar/plugins directory, you should remove it when upgrading.

Unfortunately there was a small problem resolving a conflict when merging it, which causes the path to get mangled a little inside _sftp_connect(). Once this is resolved, the mainline bzr should fully follow settings in ~/.ssh/config, because it will be running the same ssh binary as you normally use.

One of the transports available in bzr is sftp. This is implemented using the Paramiko SSH and SFTP library. Unfortunately there are a few issues I experienced with the code:

• Since it is an independent implementation of SSH, none of my OpenSSH settings in ~/.ssh/config were recognised. The particular options I rely on include:
  1. User: when the remote username doesn't match my local one. One less thing to remember when connecting to a remote machine.
  2. IdentityFile: use different keys to access different machines.
  3. ProxyCommand: access work machines that are behind the firewall.
• Paramiko does not currently support SSH compression. This is a real pain for larger trees.

The easiest way to fix all these problems would be to use OpenSSH directly, so wrote a small plugin to do so. I decided to follow the model used to do this in gnome-vfs and Bazaar 1.x: communicate with an ssh subprocess via pipes and implement the SFTP protocol internally.

When a project grows to a certain size, it will probably need a way to share code between multiple software packages they release. In the context of Gnome, one example is the sharing of the libbackground code between Nautilus and gnome-control-center. The simplest way to do this is to just copy over the files in question and manually synchronise them. This is a pain to do, and can lead to problems if changes are made to both copies, so you'd want to avoid it if possible. So most version control systems provide some way to share code in this way. As with the previous articles, I'll focus on Bazaar, CVS and Subversion

The Python developers have been discussing a migration off CVS on the python-dev mailing list. During the discussion, Bazaar-NG was mentioned. A few posts of note:

• Mark Shuttleworth provides some information on the Bazaar roadmap. Importantly, Bazaar-NG will become Bazaar 2.0.
• Steve Alexander describes how we use Bazaar to develop Launchpad. This includes a description of the branch review process we use to integrate changes into the mainline.

I'm going to have to play around with bzr a bit more, but it looks very nice (and should require less typing than baz ...)

Havoc: we are looking at ways to better integrate version control in Launchpad. There are many areas that could benefit from better use of version control, but I'll focus on bug tracking since you mentioned it.

Take the attachment handling in Bugzilla, for instance. In non-ancient versions, you can attach statuses to attachments such as "obsolete" (which has some special handling in the UI — striking out obsolete attachments and making it easy to mark attachments as obsolete when uploading a new attachment). This makes it easy to track and manage a sequence of patches as a fix for a bug is developed (bug 118372 is a metacity bug with such a chain of patches).

Bryan: there are a number of steps you can skip in your little tutorial:

1. You don't need to set my-default-archive. If you often work with multiple archives, you can treat working copies for all archives pretty much the same. If you are currently inside a working copy, any branch names you use will be relative to your current one, so you can still use short branch names in almost all cases (this is similar to the reason I don't set $CVSROOT when working with CVS).

This posting follows on from my previous postings about Bazaar, but is a bit more advanced. In most cases you don’t need to worry about this, since the tools should just work. However if problems occur (or if you’re just curious about how things work), it can be useful to know a bit about what’s going on inside.

Changesets vs. Tree Snapshots

A lot of the tutorials for Arch list “changeset orientation” as one of its benefits over other systems such as Subversion, which were said to be based on “tree snapshots”. At first this puzzled me, since from my mathematical background the relationship between these two concepts seemed the same as the relationship between integrals and derivatives:

I got a few responses to the comparison between CVS, Subversion and Bazaar command line interfaces I posted earlier from Elijah, Mikael and David. As I stated in that post, I was looking at areas where the three systems could be compared. Of course, most people would choose Arch because of the things it can do with it that Subversion and CVS can't. Below I'll discuss two of those things: disconnected development and distributed development. I'll follow on from the examples in the previous post.

With the current discussion on gnome-hackers about whether to switch Gnome over to Subversion, it has been brought up a number of times that people can switch from CVS to Subversion without thinking about it (the implication being that this is not true for Arch). Given the improvements in Bazaar, it isn't clear that Subversion is the only system that can claim this benefit.

For the sake of comparison, I'm considering the case of a shared repository accessed by multiple developers over SSH. While this doesn't exploit all the benefits of Arch, it gives a better comparison of the usability of the different tools.

Travels

I've put some of the photos from my trip to Mataró, and the short stop over in Japan on the way back. The Mataró set includes a fair number taken around La Sagrida Familia, and the Japan set is mostly of things around the Naritasan temple (I didn't have enough time to get into Tokyo).

Multi-head

A few months back, I got a second monitor for my computer and configured it in a Xinerama-style setup (I'm actually using the MergedFB feature of the radeon driver, but it looks like Xinerama to X clients). Overall it has been pretty nice, but there are a few things that Gnome could do a bit nicer in the setup:

Mataró

The conference has been great so far. The PyGTK BoF on the weekend was very productive, and I got to meet Anthony Baxter (who as well as being the Python release manager, wrote a cool VoiP application called Shtoom). There was an announcement of some of the other things Canonical have been working on, which has been reported on in LWN (currently subscriber only) among other places.

Over the weekend, I had a little time to do some tourist-type things in Barcelona. I went to La Sagrada Família. It was a great place to visit, and there was an amazing level of detail in the architecture. You can walk almost to the very top of the cathedral, and see out over the Barcelona skyline (and see various bits of the cathedral not visible from the ground). I'll have to put my photos up online.

Libtool

When looking into the libtool problem I mentioned earlier, I decided to take a look at the libtool-2.0 betas. Overall, it looks pretty good. I've updated the gnome-common autogen.sh script to support it. So if a package uses the LT_INIT macro, it will call libtoolize for you.

One of the new features in these versions of libtool is that if you have a AC_CONFIG_MACRO_DIR(directory) call in your configure.ac file, it will copy the libtool M4 macros to that directory. If you then call aclocal with the correct -I flag, autoconf will use that version of the macro.