Tags: Java
The other day I wrote about a
patent some of us in the quality team received over a test execution scheduling tool we developed. While the patent and the software is interesting, the thing that's really interesting is what that software enables us to do.
Namely ... because of DTF we are able to schedule execution of a tremendous amount of testing on a wide set of platform combinations. Without DTF we would get lost with the test execution schedule the java quality team faces. I thought it would be good to outline just what that is.
First thing is the number of tests. I don't know where to find the total, but I am sure it's over 100,000 tests. This isn't just the JCK but a whole slew of "functional" tests the quality team writes. The JCK tests have a strictly defined charter of measuring conformance with the standard, and there's a whole lotta testing done beyond that for functional behavior.
Obviously we don't run all the tests everytime we do testing. I understand it takes 2-4 weeks (or more?) to go through all the tests on all the platforms we test on. We therefore shift and juggle what gets tested each week based on what changes were made, and just to rotate through the tests and platforms.
The next thing to consider is the number of platforms on which we test Java. I can't publish the exact platform list, but obviously it's going to be a lot more than "Windows", "Linux" and "Solaris". The platform matrix considers many more factors such as the operating system version, the CPU, graphics cards, locales, web browsers and more. For example an i18n test needs to run in every locale, while a plugin test needs to run on every web browser. Graphics tests need to run on every graphics card.
Clearly if we did a straight permutation of every operating system version, on every CPU, on every locale, on every web browser, on every graphics card ... that is the direction we go with the platform matrix, but nobody could afford to test every combination. Hence we make a platform matrix that includes several combinations of those attributes, and we base it on judgement and experience.
What this gives us is each week is a group of tests to execute, and a group of platforms on which to execute those tests.
Here's where DTF comes in. In case you didn't read my posting the other day, the capsule summary of what DTF does is this: Using JINI technology it dynamically constructs a network of systems on which we can schedule test executions. It has a "controller" which matches test execution job requests with test systems on which to execute the tests. It uses JINI requests to tell systems what tests to run when, and it collects results and sends out notifications.
So then, handling the test execution load is moderately simple ... one just creates a set of "job files" describing the tests to execute and the test platforms, and hands those job files to DTF. DTF does all the scheduling working at it all night long etc.
Source:
weblogs.java.net
Comments
David, thanks for your nice articles. Was there a specific reason why you chose to build your own tool, rather than using a decentralized continuous intergation approach a la tinderbox/xenofarm ? cheers, dalibor topic
Posted by: robilad on December 01, 2005 at 11:30 AM
I covered that some in the 'patent' posting. First, we developed the DTF in around 1998(?) and surveyed what was available then. I think Netscape was still closed source then so tinderbox wasn't available, and I don't know what xenofarm is. Several available tools did pop up and we considered them. The basic task is a specialization of batch job submission and control after all. The issue is that we need to cover Windows, Solaris and Linux, and at the time nothing covered all three. Today Sun's Grid Engine does cover all three except it doesn't support 64-bit Windows.
Posted by: robogeek on December 01, 2005 at 03:39 PM
hi david, thanks for your reply. Xenofarm is a distributed continous integration tool used by Pike developers (another nice programming language). cheers, dalibor topic
Posted by: robilad on December 03, 2005 at 04:45 PM
