bzr

I went to Jonathan Corbet's yearly update of the status of the Linux kernel. He talked about the various big changes that went into the kernel over the last year as well as the development processes. The Linux kernel is probably one of the largest open source projects, and very healthy - there are a lot of individuals and companies contributing to it. With this size
comes a few interesting challenges coping with the flow of changes into Linus' tree. Their current processes seem to deal with this quite well, and don't seem to need a lot of major changes at the moment.

His talk also included the obligatory list of features that landed in the last year. The only one that really matters to me is the Nouveau driver, which I'm looking forward to trying out.

The second talk I went to in the morning was Selena Deckelmann's overview of the Open Source database landscape. She mentioned there's new projects started daily, but it was still a bit disappointing not to see TDB up there.

After lunch Rob gave a talk about Subunit, introducing to the ideas behind the Subunit protocol as well as presenting an overview of the tools that are available for it and the projects that have Subunitized as of yet. It's exciting to see the Subunit universe slowly growing, I wasn't aware of some of the projects that are using it. The recently announced
testrepository also looks interesting, even though it is still very rudimentary at the moment.

In the evening Tridge, Rusty, Andrew, Jeremy,AJ and I participated in the hackoff as the "Samba Team".

The hackoff was a lot of fun, and consisted of 6 problems, each of which involved somehow decoding the data file for the problem and extracting a short token from it in one way or another, which was required to retrieve the next problem. We managed to solve 4 problems in the hour that the organizers had allocated, and ended first because we were a bit quicker in solving the 4th problem than the runner-ups. No doubt the fact that we were the largest team had something to do with this.

I hung out with some of the awesome Git and Github developers in the Malthouse in the evening, and talked about Dulwich, Bazaar and Launchpad ("No *really*, I am not aware of any plans to add Git support to Launchpad.").

So far I'm very much enjoying my first DebCamp / DebConf. It's nice to finally meet a lot of people in person that I have worked together with or talked to on IRC in the last few years. Cáceres is a relatively small town with a nice old city center.

I arrived early for DebCamp and spent the first few days here working on fixing bugs in the Bazaar and Samba packages as well as discussing the integration between Samba 4 and Kerberos with Sam (both in general and on Debian specifically). In trying to set up a Samba 4 domain we found a number of bugs in the provisioning script, most of which seem to be fixed now.

In the last few days I've mostly worked on getting Samba 4 and OpenChange ready to go into Sid (they're in experimental only at the moment) and have discussed bzr-builddeb and related Bazaar issues with James.

My identi.ca feed is now also being forwarded to twitter here: http://twitter.com/ctrlsoft.

cp: Pixies - Velouria

At the moment I'm in Barcelona, attending the Bazaar sprint, which happens to be cohosted with the Ubuntu Developer Summit. I attended half a day of the UDS in Sydney a couple of years ago, so I had some idea of what I was in for already. It's a lot of fun, and having the two events at the same place at the same time was really worthwhile :-) I finally got to meet some of the Ubuntu server people in person.

I flew here a couple of days early so I could do some sightseeing, never having been in Barcelona (or Spain) before. The city was much nicer than I had expected, and we had some great weather. The timing was great, too: we we were there during the Cycling Tour of Catalonia and F.C.B. won both the national title and the European champions league (I hope I got that right, the important bit was the large amount of people partying outside :-).

During the first day of the summit I gave a short plenary talk on the Samba packaging in Ubuntu. Other than that I spent most of the time during the day switching back and forth between the Bazaar sprint room and the various break-out sessions. The main topics in the Bazaar world were the 2.0 release, and what had to be done to get it out of the door.

% bzr branch deb:line6-usb-source debian
Retrieving Vcs locating from line6-usb-source Debian version 0.7.4-1
Branched 354 revision(s).
% bzr merge-upstream https://line6linux.svn.sourceforge.net/svnroot/line6linux/driver/trunk
All changes applied successfully.
Using version string 0.7.4+svn511 for upstream branch.
The new upstream version has been imported. You should now update the changelog (try dch -v 0.7.4+svn511-1 "New upstream snapshot."), resolve any conflicts, and then commit.
% dch -v 0.7.4+svn511-1 "New upstream snapshot.
% bzr builddeb
Building using working tree
Preparing the build area: ../build-area
Purging the build dir: ../build-area/line6-usb-0.7.4+svn511
[...]
Placing result in /home/jelmer/bzr/line6-usb/result
% ls ../result
line6-usb_0.7.4+svn511-1_amd64.changes line6-usb_0.7.4+svn511-1.dsc
line6-usb-source_0.7.4+svn511-1_all.deb
line6-usb_0.7.4+svn511-1.diff.gz line6-usb_0.7.4+svn511.orig.tar.gz

cp: Phideaux - Microdeath Softstar

Ever since bzr-svn started supporting "true push", people have been complaining about the extra file properties it sets.

The key thing about "true" push is that it preserves the exact revisions that were present in Subversion. This lets bzr behave on Subversion branches transparently using the same UI you also use for "native" Bazaar branches.

In other words, if I push to a Subversion branch from my machine, then that branch in Subversion contains enough information for somebody else to reconstruct the exact bzr branch I had.

Since some Bazaar metadata can not be represented in Subversion, it is stored in Bazaar-specific Subversion properties. Unfortunately, these file properties show up in email commit notifications and trac and so they tend to annoy people.

There are two ways around this:

• Revision properties

Bazaar-specific metadata can be stored in in custom Subversion revision properties (these don't show up in commit notifications). Unfortunately, this requires Subversion 1.5 or newer to run on the server.

I hope to start setting revision properties instead of file properties when possible as of the next bzr-svn release.

• less strict push

It's also possible to throw away any data that can not be represented in Subversion. Since this means that the remote branch won't end up an exact same copy of the local revisions, this isn't true push. The two branches will have diverged (no matter how slightly) after such a push so it is necessary to rebase on the remote branch after pushing.

This is similar to the way git-svn pushes data into Subversion - it calls it "dcommit".

Since this uses rebase it has the usual disadvantages of rebases, which I won't get into right now.

As of a couple of days ago, bzr-svn now also supports this mode of pushing using the "dpush" command, by popular demand.

cp: Brandi Carlile - The Story

bzr-svn has always been using the standard Python bindings that were provided with Subversion itself. Unfortunately, I had to fix some issues in these bindings since they were incomplete or broken and thus bzr-svn has always depended on a development snapshot of Subversion.

As of today, bzr-svn is using its own Python bindings for Subversion.

There were several reasons for switching to our own bindings:

• There are no requirements for backwards compatibility within bzr-svn. This means the API can be made sane without worrying about the mess it was in the past and users who still rely on that.
• Deployment. It took 2 years for my fixes to the Subversion Python bindings to be part of a release. It'll be even longer before Subversion 1.5 makes it into most available distributions. That makes it very hard to just download and install bzr-svn.
• They're in plain C, not SWIG. SWIG has a big advantage for the Subversion folks since it can generate python, ruby, java or tcl bindings all at once without a lot of overhead per language. However, it has issues as well that make it a bad choice for bzr-svn.
  • It generates inefficient code - it generates proxy classes that add more layers in the stack
  • Bindings tend to be very much like the C API rather than "Pythonic". To make them more Pythonic, you need tons of typemaps. For example, the Python bindings in bzr-svn provide an iterator when browsing the revision history rather than a callback as C and the SWIG bindings do.
  • Hard to write - personally at least, I write bindings in C faster than in SWIG
  • Adds an extra dependency to the build process. Several people had trouble building Subversion on their Mac machines because they didn't have the right version of SWIG available.

Since all of the patches that bzr-svn depended on previously were in the Python bindings for Subversion, it is now possible to use bzr-svn with any version of Subversion newer than 1.4.0. Of course, you do need to have the development headers installed as well.

cp: Kathleen Edwards - Independent Thief

I was happy to see that John Carr has set up a Bazaar Mirror of all projects in GNOME Subversion, all created using bzr-svn. There's also a quick introduction to using Bazaar for GNOME developers on the GNOME wiki.

Wouter, long time Bazaar user and GNOME dude, recently blogged about pushing Bazaar branches into GNOME Subversion, working around the restrictions imposed by the pre-commit hooks in GNOME Subversion.

The problems John ran into with memory usage in the Python Subversion bindings encouraged me to continue the work on bzr-svn's own Python bindings, thus avoiding any dependency on unreleased versions of Subversion and several other issues.

My relationship with git is still one of love and hate. It cuts corners to increase performance in a couple of places and that can be really bloody annoying.

For example, jerry renamed one of the top-level directories in Samba 3 (revision 9f672c26d63955f613088489c6efbdc08b5b2d14). Git will skip rename detection in this revision because of the number of files it affects, thus causing the output of "git log <path>" of this particular directory to be useless.

I'm the first to admit "bzr log" on directories and files in large history projects is painfully slow, but at least it gets the output right.

cp: Brandi Carlile - The Story

This slightly evil hack to bzr-svn allows using bzr-builddeb as a drop-in replacement for svn-buildpackage, making it recognize the "mergeWithUpstream" property svn-buildpackage uses.

cp: Jeff Healey - Mess O' Blues

The London bzr sprint is over again for this year. It was really good to meet everybody in person again and also to meet some of the folks who hadn't been to a sprint before.

Last years sprint was mainly about improving performance; this year, we discussed adoption blockers and how to remove them. A short summary of the brainstorming is on the wiki.

Martin's blog has some pictures.

The Mars Volta concert we went to last night in Tilburg was absolutely brilliant. Very energetic and definitely one of the best acts I've ever seen live. We were standing in the back of a completely packed venue for 3 hours, but it was very much worth it.

cp: Soft Machine - Teeth

Hooray, Bazaar (including the plugins I have installed here) has reached the 10000 unit tests! :-)

This is something that's been lingering in the back of my head for the last year or so. I think I am missing something in the sequence of [Branch, Repository, WorkingTree]. Here are some of the reasons why I think this is the case:

• Tags should ideally be shared amongst a set of related branches. This has come up often during discussions about where tags belong.
• Management of sets of bzr branches is hard
  • It seems to make sense for the configuration of several plugins to be project-specific:
    • bzr-pqm-submit's pqm address
    • bzr-email target address
    • bzr-cia's project setting
    • ...
  • may be useful to override whoami
  • having a way to group branches allows mass-pushes/pulls
• I often find that the public_location I set is almost the same for related branches, with only the last part of the url differing and containing the branch nick
• It would be nice if "bzr register-branch" could automatically determine what product to register as

I'm not looking for repositories:

• repositories can contain data from multiple totally unrelated branches. a tag "1.0" could conflict because there are multiple unrelated projects that have it.
• repositories are a storage optimization and I like them that way

although other projects (mercurial, git) seem to be using repositories to allow talking about a group of related branches.

I'm not looking for "just" directories:

• There's no place in a directory to store settings or tags
• Having a long list of settings in ~/bazaar/locations.conf doesn't scale and the settings won't be able to propagate

Having another semantic object (``Product''?) on which options/tags can be set would help. Perhaps based on the root id (where available) ?

cp: Symphony X - The Odyssey

After a short but very nice holiday in Ireland and a long journey back over Brussels, I'm back in the Netherlands. Renting a car turned out to be a very good idea, although it took some time to find a rental company that would rent a car to people under 25.

We spent the first two days sightseeing Dublin in traditional Irish weather (short bursts of rain every half hour or so) and made two small road trips in the week following in the best weather imaginable.

The next major release of bzr-svn, 0.4, has now been released. The main change in this release is that the behaviour of push is now intuitive. The big hack that allowed push to somewhat work in the previous release has been replaced by proper push which behaves the same way as it would against a Bazaar branch.

It's now also possible to branch from non-standard branch locations such as /foo in a repository and not necessarily standard locations like /trunk or /trunk/foo. See the release announcement for a list of other changes.

It's interesting to see what other people are saying about bzr-svn on their blogs:

• Migrating to bzr using bzr-svn
• Missing support for svn:externals
• Issues with branching schemes
• using bzr-svn for rhythmbox
• bzr-svn on Gentoo
• issues with Windows support

Note that bzr-svn 0.4 has been tested on Windows and that branching schemes are now more flexible.

Ohloh is a nice web 2.0 site that contains stats on various Free Software projects. At the moment, they only support Subversion, CVS and Git. They're open to feature requests though. If enough people ask for it, hopefully they'll support Bazaar at some point.

One of the things that I've always missed in DVCS is the ability to refuse commits in a branch that's shared by multiple people based on a test suite run. Sure, it's possible to have a pre-commit hook - but that would mean that you'd have to wait for the full test suite to run until the commit finishes.
With the time it takes to run the Samba testsuite, this is not really an option.

One of the things that would work is to have everybody work in a separate branch and then have some sort of tool that merges those revisions from everybody's personal branches that worked ok. However, to my knowledge, there is no such tool for Subversion.

Bazaar uses a tool called PQM (Patch Queue Manager). PQM usually controls the main branch (for example for Bazaar, it controls bzr.dev), and waits for GPG-signed requests to merge a specific revision into that main branch. Before accepting such a revision, it will try to run the testsuite to make sure it passes. This guarantees that the main branch never contains broken code (as far as can be indicated by the testsuite).

Now that bzr-svn supports true push, it is possible to actually use a PQM with a Subversion branch. I've tried it on a smaller branch last week, and am now looking into using this for my Samba work.

I'm currently doing a bit of sightseeing in London, after attending the
Bazaar sprint at the Canonical office. It was a good sprint, and a bit
different from the previous ones in that there was only a limit amount of
actual coding involved. The view from the Canonical office is magnificent, so we were even able to do some sightseeing while working; (-:

Bazaars' focus has previously mainly been on correctness and features. The
first has always been one of our strengths, and we're in pretty good shape
regarding the second. Performance has been one of the main complaints from
users about Bazaar and so we've recently tried to get better in that area.

Since 0.12, we have already tried to optimise some of the common code paths and
some people have been working on a high performance smart server (to speed up
remote operations).

During the first two-and-a-half days of the sprint, we have analysed 20
of the most common use cases with Bazaar and determined what complexity they
should ideally require to be able to work. After this analysis, we looked at
ways to change our data structures to reach these goals.

I've been mainly a spectator during the latter parts of these discussion, but
they were interesting to follow.

One of the things I worked on were support for true push in bzr-svn. This was
one of the bugs that bit a lot of users of bzr-svn. The upcoming bzr-svn 0.4
now supports true push as well as commits in heavyweight checkouts. I hope to
release 0.4 after adding nested tree and ignores support so that I don't
have to upgrade the mappings again.

Ah well, time for some more sightseeing and getting back to the reason I'm working on all this: Samba!

As of 0.15, Bazaar will have initial support for nested trees. Nested trees basically allow you to add a 'magic' directory to your Bazaar branch that in itself is another Bazaar branch. A very good use case of this is
if you have a project for which you need to include some library (maybe popt?). Nested trees allow you to add a reference to the upstream popt branch, avoiding the need to synchronize every time upstream fixes a bug.

Subversion has similar functionality, in their terminology called 'externals'. Externals are very easy to use - all you need to do is set the svn:externals property.

I've done some early work on supporting externals in bzr-svn and mapping them to nested trees. At the moment, bzr-svn is able to track nested trees that are being added. Removes and changes to existing nested tree locations are still on my todo list, but will definitely be supported in bzr-svn 0.4.0.

Since I've upgraded serendipity to version 1.1, it seems to keep losing my draft blog entries (either that, or hiding them somewhere I can't find them). For that reason, this blog post is a bit short, as I'm too lazy to rewrite the original one I had drafted up.

Bazaar 0.15 is promising to be a really great release. Several new features are going to land: dirstate (fast working tree access), tags and initial nested tree support.

I miss travelling.

cp: Smashing Pumpkins - Ava Adore

I much enjoyed the Bazaar sprint, though it was very intensive. The last and seventh day, I only got a handful of things done, pondering about other things throughout the rest of the day. Sleep caught up with me very early in the evening. Some of the main topics I worked on included shallow branches, bzr-gtk and several bzr-svn related things.

Robert, Vincent and Wouter got a lot of work done on the high-performance smart server code. I'm interested in having a look at that in the near future as well.

cp: Anouk - Nobody's Wife