Tags: Java
A couple weeks ago I did a little experiment in community processes. Supposedly community driven processes are better quality because there's more eyeballs. That's an interesting claim, and I wanted to test it.
The Register has an article along the same lines as my test:
Wikipedia founder admits to serious quality problems
My test is written up here:
An experiment in community process
Basically, was reading a book about collaborative development processes and remembered an article I saw several months ago about a test of the wikipedia ability to fix their encyclopedic project through their own community process. In that test someone posted a bogus article expecting the wikipedia community to notice it and do something about it. When they didn't notice it after a week the guy went "hmmm...".
In my test I posted a bogus article, specifically a randomly generated computer science paper. I wanted to test that community to see how responsive it is. This time the wikipedia community came through and within 18 hours my article was gone.
It's kind of mixed results. In some cases bogus articles stay in place, and in others they get deleted. However the Register article weighs in with their analysis that there's a lot of problems with the community process on the wikipedia, with the evidence being the large number of questionable articles.
Well, I haven't made a deep study of the wikipedia, but the articles I've looked at were generally good. That is, except for the ones related to a peculiar niche interest I have in "energy healing" (See
peaceguide.com or reiki.7gen.com for some resources that demonstrate what energy healing is). The related wikipedia entries are rather spotty, but it's such an out-of-mainstream niche one would expect poorly informed articles.
One thing the Register article talks about is the tortured quality of the writing in general. I suppose that's a direct result of the communal process, since you've got hundreds of volunteer editors running around each tweaking what other people have already tweaked. That's surely a recipe for strangely written prose if I've ever heard of one.
Looking at it from my perspective in the Java team, I can't help but think about different software development models. You've got open source projects and their collaborative development model, and you've got the Java team with it's long-standing largely closed development model that we're searching for a way to open to the larger community. The open source advocates claim the collaborative model is better repeating mantras like "with enough eyeballs, all bugs are shallow".
I'm interested in actually testing this claim rather than accepting it on blind faith. The result with the wikipedia quality gives me reason to doubt that a collaborative process is going to always result in high quality.
As another test I'm looking a little at the Hibernate project, and using the Findbugs tool. I'm really getting familiar with using findbugs as part of a normal software development process, but I thought to also run it on several versions of the Hibernate build. I've only just begun this but I did find something very interesting.
Namely ... in Hibernate 3.0 there are several very shallow bugs ... which all the eyeballs in that project did not find. For example code like
if (obj == null) { throw new SomeSpecificException("unknown object " + obj.toString()); }
Clearly that code is going to throw a NullPointerException rather than the specific excpetion the author of that code expected it to throw.
Clearly no software development model is perfect, and bugs slip through the cracks of every software development model.
Source:
weblogs.java.net
Comments
NullPointerException, very funny. ;-)
Realistically, there is no comparison between a wiki, in particular Wikipedia, and community based software development.
In a wiki, anybody can post any type of position. Whereas in a community software project; each patch, or feature submisssion, is reviewed in detail, by the community, and approved by trusted lieutenants, before inclusion into the codebase. This is precisely what helps make these bugs so shallow.
Correspondingly, and completely unlike wikis: There are not the post / delete / repost / re-delete wars.
John
Posted by: cajo on October 19, 2005 at 08:36 PM
While I strongly beleive in the open source development model, I have to question John assertions.
Yes, there's more of a gatekeeping function in OSS than wiki's, but that should make bugfixing easier in wikis, not harder. (It also makes bug introduction easier in wikis, but that's not the point the author was trying to make).
The assertion that every code submission is reviewed is on it's face, wrong. The NullPointerException example posted above is one good example of that not happening.
And there are certainly post/delete/repost wars in OSS - they're just handled differently, usually via flamewars on mailling lists, especially in communities without strong decision mechanisms.
Posted by: driscoll on October 20, 2005 at 10:18 AM
