Tags: OpenJDK
Last week I wrote a bit about
the OpenJDK quality team we plan to launch, and about one of our primary concerns. We are pretty sure there are many people who would like to contribute to Open JDK quality, because there are many people who see the vision of Java and want to see it live. This concern we have is about test harness choice and whether the choice of test harness could interfere with our hope to collaborate with the public on test suite development.
Last week I discussed the different modes of tests we have for Java SE: the JCK, the developers unit and regression tests, the performance tests, and the SQE team's functional tests. And, to remind you, by "we" I'm referring to the SQE team and our functional tests.
The JCK is executed using Java Test (a.k.a. JTHarness), as is required by the licensing of the JCK. Java Test has a long history and is the official harness of the JCK. The JTHarness project is an open source release of java test.
The developers unit and regression tests are executed using jtreg, which is a layer over Java Test. A binary release of jtreg is being made through the OpenJDK project.
The SQE team uses javatest for JCK execution, uses jtreg for some tests, but we tend to prefer using "Tonga" for our test suites. Tonga is an internal name, so if it violates anybodies trademark please forgive me. It also has a long history in our team and we find it to have features which make it worth our while to have developed this harness and continue maintaining it. There's also a practical issue that we have over 30,000 tests written using Tonga and it would be costly to convert them to a different harness.
We have a number of questions, and few answers. But before I get into them I want to address a question inspired by my previous blog entry: I'm curious how Tonga differs from xUnit/TestNG
It might be best to go through the functionality of Tonga. The essential difference is that xUnit and TestNG both have the test extend a superclass which provides the test framework. Tonga by contrast runs tests from a main method, and Tonga can also test command lines because it provides a shell script interpreter.
Tests are listed in a test list: The testlist is a simple text file which lists the names of test cases along with a few parameters.
Supports multiple test types: One of the parameters in the testlist is the test type. We use tonga to execute jtreg tests, for example. We believe Tonga can support xUnit or TestNG test types. The native Tonga test format is a directory hierarchy, not package hierarchy, which contains test directories.
Testlist entries are tagged with keywords: And the user can select subsets of tests by specifying the tags.
Configuration through bourne-shell workalike thingy: This makes configuration of a test suite pretty straightforward, given that shell scripting is common knowledge, and also very flexible, due to shell scripting. It also makes for easy automation of command line tests, since you're dealing with shell scripts and hence shell commands. This gives you tons of flexibility in test implementation, giving data to tests, reusing the same test multiple times with different datasets, etc.
Tests are each executed in a fresh VM: Each test execution gets its own JVM instance. If a test pollutes the JVM, following tests know they aren't affected by the previous tests, because each has a fresh (clean) VM to execute in. Tonga can execute tests in "same VM" mode, but this is rarely used.
Distributed test execution: Some complex tests might set up several server processes and run test cases that depend on those servers. For example our networking and security tests regularly use this to set up the required servers, so our tests can make LDAP etc queries against a known environment. This makes it easy to set up and tear down complex test configurations.
XML results format: Tonga can produce its execution results in a well documented and understood XML format. We funnel the results into a results reporting database and web application. The XML file itself is pretty useful and we sometimes XSLT it into standalone web pages. The file contains extensive information about the test environment, JDK version, etc, plus the execution results of each test.
There's more but this should give a flavor of what Tonga offers, and those who know xUnit/TestNG better than I do will see some differences.
My questions:
Does it seem strange for us to consider publishing another test harness? Will you guys be confused over which test harness to use?
Or is that worry overblown? I think it might be... for example I understand most or all of the subteams of the Mozilla project use different harnesses. So it seems like the nature of software engineers to want specific features in test harnesses and sometimes to build new ones when the existing ones don't work the way they want. I read an article recently that software source reuse is often not done in practice, while most people do clearly agree it's a worth thing to try and do.
Do you think the XML results format is attractive? We hope to ask the public to do test execution and send in reports, and would like the reports to be in an XML format. We think the format we've developed is pretty good and think it might be attractive to other test harnesses as well.
Source:
weblogs.java.net
