Open, community developed, test suite for Java

; Date: Mon Apr 23 2007

Tags: OpenJDK

Robert Burrell Donkin ( JCP: Time For An OpenTCK) and Steve Loughran (Open tests for open standards) discuss their thoughts about an "Open TCK". These are interesting thoughts .. ones that make it tempting to conclude that there must be an open TCK for Java. I seem to remember however that in some standards bodies, even ones where the standards are "open" (for some definition of "open") that access to the standards and tests require ponying up money. I sure remember that was the case for the OSI network protocol standards "back in the day" when I worked on implementing an X.400 email system. So an "open standard" (the phrase they use) doesn't always have "open" tests, but there's a difficulty with that word "open". One meaning is as in "open source" where there are specific definitions of "open source". But if you think back a few years ago, the "open software foundation" was an important force in computer systems and there's no way you could call their software suites "open source", but they were "open", right? And, OSI, the 'O' stood for Open, right?

I cannot directly address the part of their postings related to the open letter from the Harmony project .. there are other people than myself working on plans regarding the JCK (Java's platform TCK).

What I can talk about is an idea we are developing to sponsor an open source test suite. Robert Burrell Donkin mentioned the value and attraction to: Pooling functional and integration tests would allow a more comprehensive test suite to be created with less total cost than each implementor creating their own.

I would go even further to say that much of the knowledge about Java quality and compatibility problems are in the minds of application developers. We do a lot of analysis of potential bugs, developing test scenarios, what we think typical API usage patterns might be, etc, but clearly given the bugs that exist we're not catching every problem. Not that any quality team for any software project catches every problem, but wouldn't it be cool if there were a way for us to know about more usage scenarios? Especially the ones that regularly trip up application developers?

I've been discussing (on this blog, and at the FOSDEM meeting) recently some plans we have for a quality community for the OpenJDK project. The quality community will offer ways to contribute to testing and test suite development of the OpenJDK. We are still planning what the test suite development will look like, and we would like to know your thoughts about this. Please use the comment box below.

Remember that for Java there are at least four different classifications of test suites that exist:

1. The TCK's and the JCK are specifically focused on conformance with the specifications.
2. The Unit and Regression suite is developed by the Java SE development team.
3. The performance tests, benchmarks etc, some of which are standard open benchmarks and some of which are in-house benchmarks.
4. The functional tests developed by the software quality engineering team (SQE).

Again I am talking about the fourth of those, the functional test suites.

UPDATE: I found this blog posting by Mark J. Wielaard very interesting for some historical perspective.

UPDATE: Sam Ruby, quoting from this blog entry, asked " Question: why aren't we talking about #1?". To make it clearer ... my job and role in the Java SE team and the OpenJDK project is to build a quality team for the OpenJDK. I and my team are specifically focused on the fourth of those test suite classifications. There are other people in the organization who deal with the TCK/JCK issues. I am not the proper person to comment on TCK/JCK issues, instead they are, and it is their role to make (or not make) those comments. By saying "I am talking about the fourth of those" does not mean that "you" (all of "you") cannot talk about the JCK/TCK, just that I am not going to do so. You people can talk about the JCK/TCK all you want.

Source: weblogs.java.net