Tags: OpenJDK
I want to put out to the listeners a question which we've been pondering in the nascent Open JDK Quality Team. This team is still a work in progress, and we are due to become public as part of the "full launch" of the OpenJDK project (due to be released 'soon'.. stay tuned.. the marketing team has plans for another big announcement). Tom Marble posted the slides
I presented at FOSDEM in February which contains an overview of the OpenJDK Quality Team.
We intend to publish some of our tests as open source. "We" in this case are the quality team.. so to back up for a moment and explain the context. In the Java SE team there are approximately four piles of tests and quality activities (we embed Quality in everything we do). The JCK is the conformance test suite which many know about as one of the major hurdles to achieving the right to use the Java logo and other niceties. As a test suite it's very centric on testing assertions embedded in the Java platform specification. The development team writes Unit and Regression tests as they implement features or fix bugs. There are Performance tests which largely consist of benchmarks in many forms, some of which are already publicly available.
My team, the "we" mentioned above, is the Software Quality Engineering organization for Java SE. "We" write Functional tests for the Java SE platform. As new features are developed we are writing tests of those features. Our approach is a little different from the other three models listed above, in that we are expected to think more like application developers.
When I say "we intend to publish some of our tests", I am talking about the functional tests. The development team is planning to open some of their unit and regression tests, and the contribution policy requires supplying unit test(s) along with every code contribution. "We", the software quality engineering team, as we move toward creating the OpenJDK Quality team will offer an opportunity to have a different involvement with the OpenJDK project than writing code and modifying the JDK.
It's well understood in open source that it takes a village to grow an open source project, and that not everybody in the village is interested in coding the core platform. Some get their kicks in other ways, such as finding bugs. Finding bugs in software is potentially a very seriously interesting thing to do, and for something like the Java platform it usually involves writing code.
With all that out of the way, let me get to the meat of this blog posting.
We, the SQE team, have a number of in-house-developed tools that probably differ from the tools that are widely used. Such as, rather than use JUnit or TestNG test frameworks we have a different model and our own frameworks we've developed.
One concern we have is, will our different tools get in the way of collaborating with y'all. That is first assuming we are able to gain some interest and find volunteers willing to work with us on openly testing the OpenJDK, will the differing tools get in the way?
Should we instead retool our suites to adopt TestNG or JUnit? (TestNG seems preferable..)
Additionally we prefer having test suite execution reports in an XML format so that we can more easily put the execution results into a database and make a wide range of reports. We've defined a rather comprehensive XML format which we've heavily used for several years now. We are willing to publish the specification for this format and work with the community to improve it. We feel it would be positive for the community to adopt this XML format, because it could improve test results reporting across the board.
What's different about the test tools? Well, I don't know JUnit or TestNG very well, so let me describe a little about the tools we use.
First we've separated the test harness from the test framework. But I think these terms may not exactly be understood everywhere the way I understand them.
What I call test harness is a software application that robustly executes a series of test cases. It ensures the tests are executed and collects results even in the face of failures and crashes. It doesn't necessarily know anything about the innards of the tests or the implementation, it just knows it can execute test cases. In fact our preferred harness, which we call Tonga (it's an internal name, not vett'd for trademarks) acts like a Bourne Shell interpreter and each test case is executed almost directly like you would with a shell script.
What I call a test framework would be a software library which provides methods useful across the board. It might be seen as a skeleton test, or it might offer common classes and methods used in all the tests in a suite. Oh, and a test suite, to me, is a group of tests all centric on a given topic or functional section of the Java platform.
What I've read of JUnit and TestNG leads me to think those frameworks meld together the test framework and test harness concepts.
So there it is... we have dozens of questions embedded in what I've just written. Please offer your opinions in the comment box below. Or you can email me directly using the first-name dot last-name at sun dot com convention which we use here at Sun.
Source:
weblogs.java.net
