Jan 27, 2010

Code opera: using Gource to watch a story of Perforce depot

Gource is a neat open-source tool which allows to visualize dynamics of source code repositories, using a planar, spring-based tree layout ("balloon tree").

Showing P4 depot
Why being interested in Perforce in particular - a proprietary, non-free (and rather expensive) software?.. For a couple of reasons:
* It's quite commonly used in a big, corporate software shops (including the one I am working at), and still theoretically can be used in open software projects, for free (as in a beer)
* Gource does not seem to have an official support for it yet (it supports Git/Mercurial out of the box, and SVN/CVS via contrib Python scripts)

As an example we can check Adobe Source Libraries:
export P4PORT=stlab.adobe.com:10666
export P4USER=guest
echo "AdobeGuest" | p4 login
perl gource-p4.pl //adobe_source_libraries/ > gil.log
gource --log-format custom gil.log
(it may take a little while to gather the log file).

Another example is Jam (open-source build system) depot. This time in Windows command line:
p4 set P4PORT=public.perforce.com:1666
p4 set P4USER=anon
p4 login
perl gource-p4.pl //public/jam/src/ > jam.log
gource --log-format custom jam.log
The gource-p4.pl script looks like this:
#!/usr/bin/perl

$root = $ARGV[0];
map { my($change, $user) = $_ =~ /Change (.*) on.*by (.*?)@.*/;
    map { my($file, $act, $time) = 
            $_ =~ /$root(.*)\n.+headAction (.*)\n.+\n.+headTime (.*)\n.*/; 
        $act =~ s/edit|integrate|branch/M/; 
        $act =~ s/delete/D/; $act =~ s/add/A/; 
        $line = $time."|".$user."|".$act."|".$file."|\n";
        print $line if $file;
    } split(/\.+ depotFile /, `p4 fstat -e $change $root...`);
} reverse split(/\n+/, `p4 changes $root...`);


Granted, both examples of open-source Perforce depots visualizations do not look very exciting, because neither of them seems to be a primary development depot, and most of the commits are done by a few maintenance people.

However, I've also tried it at the company where I work, watching one-year's worth evolution of the main code branch, and that was quite insightful.

Gource also allows to supply a folder with images to be used as user portraits, which adds personality to the show, so I used that as well.

Stories in code
And I was quite amazed with what I've seen: the code tells stories.

To an outsider it might look like just a boring pictures flying around and zapping with lasers some colored circles tagged by the file names.
And generally that's the way how it is - plain boring. Boring when observed out of the context, that is.

And even if one of those flying pictures was myself, I was quite under-impressed at first.

But then small stories began to pop in my head:

"Oh yes, I remember that refactoring when part of public API was renamed and it triggered the chain reaction..."
"Here, ... Bill integrates his branch to ours"
"Oh, he found the reason for that bug everyone was puzzled about... see, three seemingly unrelated folders were changed to fix it"
"We have quite an activity of people working at the same on that clump of files, maybe we should improve modularity there?.. and then some people have to touch too many different files at once, it seems... could it be bad cohesiveness of code?"
"Yeah... there, Peter makes that commit with funny comments we laughed at all together"
"Right... Christmas. No commits, obviously... And then, see - what a rush! People fly around like crazy, apparently having a few days off was a good idea"
"And here Mark and Frank are working on that new API together... it's nice to see an activity in the unit tests' folders"
"Oh, see, John have just committed his new system there, in that new tidy bunch of files... cool stuff, he knows his job... surgical changes"
"Ah, that... such a pity we had to revert that... And now - ha-ha, see, Brandon started to fly around like crazy, such a productivity again, huh?.. What was that? A new girlfriend or something?"

And so on.

That really reminds me the topic of emergent storytelling in games, when players create their own stories and enjoy them:
Human beings like stories. Our brains have a natural affinity not only for enjoying narratives and learning from them, but also for creating them. In the same way that your mind sees an abstract pattern and resolves it into a face, your imagination sees a pattern and resolves it into a story.
Games have always had a close affinity with story-making. Adding a few lines of description to a video game or a background and artwork to an abstract board game gives dramatic context and an added sense of depth, allowing the player to create an internal narrative as the game progresses.
(from "Second Person: Role-Playing and Story in Games and Playable Media")

Here it's not a game, but rather a meta-game of a kind (it's quite ironic that this is rather a process of making a game in this particular case, because I work in a game development company). There are still quite a few parallels, it seems.

Quite frankly, Gource provides only the most basic display. It does not tell too much about the code structure and how well different areas of work are factored between different contributors (even though quite often well-though folder structure is a sign of overall good code organization).

But I can imagine even more narrative devices added, for example:
* Fetching the information from the continuous integration system and display some crazy particle explosions when build was broken
* Doing similar, but even more intense display when automatic regression tests were broken
* Integrate with bug tracker, and give some visual cues about the nature of work being done (bugs, regressions, crashes, new features)
* Visually tracking the physical code dependencies as well (like #include graphs in C++) and their evolution

I believe all this is can be useful in a practical sense as well.
It seems that entertaining things generally have bigger cognitive potential. Andy Hunt tells in "Pragmatic Thinking&Learning":
...In fact, additional studies have shown exactly that: positive emotions are essential to learning and creative thinking. Being "happy" broadens your thought processes and brings more of the brain's hardware online.
Aesthetics make a difference, whether it's an user interface, the layout of your code and comments, the choices of variable names, the arrangement of your desktop or whatever.
Nice things matter, especially if they provide a useful view, yet another mental projection upon the code base, which allows to extend one's mental models of this code in a fun, aesthetically pleasing way.

Work should not necessary be boring.

So what?..
From the practical perspective, one quite satisfying thing to me was that such generally useless one evening exercise gave me quite a bit of awareness about several (seemingly unrelated) things:

* Reading the source code of Gource (C++), compiling and running it both on Linux and Windows
* Making up somewhat artificial problem in this context (how to use it with Perforce) and figuring out what steps can be taken to solve it
* Getting to know a little bit of Perl (yes, this is my first Perl script ever... it must be quite obvious from the code, anyway), while solving a real-life problem
* Trying to do it in a (somewhat) functional programming style
* Improving my regular expressions skills
* Getting to know a bit of p4 command line interface, again while solving a real-life problem
* Getting some insight about the Adobe GIL and Perforce Jam open source code bases
* Using Gource to visualize evolution of the code base at the company where I work.
* Reflecting on it

I am certainly not sure if regular solving of such small artificial problems (and learning a few random things while doing that) make you a better developer or something.
Perhaps it does. But if it does not - who cares?..
It was fun, after all.

1 comment :

RCL said...

Something is wrong with the way you match file, act and time... I don't know perl well enough to say what is wrong, maybe it's because of Windows line endings (I use p4 on Windows)?