Tags: Java
Today I spent a few hours looking at the presence of quality organizations in different open source projects. Typically you think of an open source project has being some software developers scratching an itch, and so the processes circle around the software developers. So I expected to find little presence of quality organizations within the various open source projects.
The results of my search surprised me. In some instances anyway.
Most of the testing in quality organizations seems to be unit tests written by the developers. While unit tests is better than no tests at all, unit tests have a limited focus. How do you know the whole thing is gonna work well if you only test the parts?
One project that stood out is netbeans. They've not only got a full time QA team, all their test plans are right there on their web site.
Some projects (e.g. Eclipse) had zero presence of testing on the web site. I'm sure these projects have tests and test activities, and I'm talking about the visibility of the quality process.
Why is this important? Quality is important! If the open source community were to shirk on quality, the perception of the software will suffer as will acceptance. I remember reading assertions that in some open source projects there's many eyeballs looking at and using the software daily, and that means daily adhoc testing.
Well, okay, that sounds reasonable, but is that what you want to rely on? In commercial products you have organized test plans, dedicated test teams, regular testing, etc. One truism widely noted is that it's a lot cheaper for the developer to find a bug the day after they write it, than it is for the bug to be found out in the field. First, the bug found in the field diminishes the perception of the product, and secondly the bug found in the field takes a long process to raise the attention of a developer so s/he can fix the problem. The next most cheapest place to find the bug is in the weekly testing by the dedicated QA team.
And when the organization is being run on a shoestring because there's no money flowing through the organization - cost is very important.
In any case that (whether individual open source projects are having strong QA teams or not) isn't the axe I'm looking to grind. Instead what I'm looking at is what openness might mean to the Java SE quality team. Today's activity was a kind of literature review, looking to see what models I could find in the open source projects.
Oh, and before you readers take this to mean I'm working on open-sourcing Java, no, that's not it ... there is a process working its way through the Java SE team to create more openness. You'll have seen earlier results on this such as the stuff we publish at
http://jdk.dev.java.net/.
All that's going on is I'm putting together ideas. I'm interested in hearing yours.
Source:
weblogs.java.net
Comments
Open source and quality assurance tend to not go together because the people who know how to do QA don't do open source. At least I never found any who do. When Netbeans has QA; its because Sun pays people to do so.
In the end open source invents another methodology to keep quality high and keep bugs out of public releases. The release early and release often surely helps here. (more eyeballs and all) Having a good release coordinator that refuses to release a final version until certain bugs are fixed is another. One thing that corporate people tend to forget is that many (successfull) open source projects have always had a more structured release schedule; with feature freezes and beta releases and all. Corporate software needs QA much more since they have a lot less beta testers and a shorter beta period to begin with. There is a growing awareness of usability people that Open source needs and accepts their help. I'm hoping that someday the same will start to be true for QA people.
Posted by: zander on August 14, 2005 at 10:08 AM
on Eclipse QA: Huh... have you looked
at the download page of, for instance, Eclipse 3.1?
The test information is right at the top of the page (links to test results from the build,...) and starting with the Eclipse 3.1 process (July 2004) the team monitored the performance of the product compared to the Eclipse 3.0 release (see the Performance Results link) which is very useful to see if recent changes have caused problems. The CVS repositories also contain the .test modules for the various Eclipse elements/plugins, AND they are available from the products download site (the link I gave above) in the section titled "JUnit Plugin Tests and Automated Testing Framework" with the testing framework that is required to run them.
And of course there the Bugzilla system for Eclipse for all your other QA questions.
Posted by: murphee on August 15, 2005 at 06:51 AM
This just looks like Netbeans propaganda. Of course QA is important, but next time you want to give the impression that Netbeans is superior to Eclipse, please do this in a more subtle manner, here it is too visible.
Posted by: jpz on August 16, 2005 at 12:56 AM
Murphee, thank you for being helpful and pointing me in the right direction. I'll add that to the information I am collecting. However, I did spend a long time hunting around the eclipse site looking for pages that described the testing/quality procedures or team. Bugzilla is an issue tracking system, not quality procedures.
It looks like my earlier comment didn't get posted. Anyway, what I'm attempting to do is understand how the quality team works in typical open source projects. I work for the Java SE Quality team, and I'm looking to see how the openness initiative would apply to my team. Hence, I'm looking for models of how the various open source projects have done it to see what we might be able to do.
There's nothing nefarious here. I really don't know whether it's better to have a dedicated QA team like netbeans does. For example, besides netbeans I could have (should have) mentioned GNOME and KDE, both of which have significant a presence for their quality organizations. However, what's different there is the tone on the GNOME and KDE web sites that it's the public that owns the quality of GNOME and KDE. This is different than how netbeans approaches it where netbeans has dedicated Sun staff doing a lot of the work. Posted by: robogeek on August 16, 2005 at 02:02 PM
Hi David,
I just found your blog post about QA in open source. I'm doing a similar search. Can we talk offline?
Thanks John
Posted by: johndaddamio on December 21, 2006 at 02:52 PM
