December 22, 2007

If it hurts, automate it

In many projects, it's painful to ensure that all unit tests are run on every patch before committing. Add to this some other basic checks:

• no new javac compiler warnings
• no new Findbugs warnings
• zero javadoc warnings
• zero @author attributions

and committing patches becomes very painful and time consuming indeed.

Something that painful, of course, is begging to be automated -- and that's exactly what we did with our patching process for the Hadoop project.

Every patch for Hadoop must be attached to a Jira issue. When a Jira issue is moved into the Patch Available state by a developer, the Hadoop continuous integration server automatically picks up the issue's latest patch, applies it to a fresh checkout of trunk, builds the software, runs all the unit tests, and verifies all the other items listed above. Once the build and testing is complete, a comment like this is automatically added to the Jira issue:

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12368715/1185_20071030b.patch
against trunk revision r590273.

  @author +1. The patch does not contain any @author tags.
  javadoc +1. The javadoc tool did not generate any warning messages.
  javac +1. The applied patch does not generate any new compiler warnings.
  findbugs +1. The patch does not introduce any new Findbugs warnings.
  core tests -1. The patch failed core unit tests.
  contrib tests -1. The patch failed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1033/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1033/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1033/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1033/console

This message is automatically generated.

This automated system diminishes a Hadoop committer's burden to simply ensuring these two comments exist on the Jira issue:

• a "+1" comment from the automated patch build, and
• a "+1" comment from a code reviewer.

No more committing pain!

Nigel Daley
Grid Computing QA Lead

Want a fun job testing open source? Passionate about software quality? I'm hiring! Talk to me at: ndaley at yahoo-inc dot com. Testing and coding experience required.

Posted by ndaley at 8:22 AM | Comments (2) | TrackBack | Permalink

December 3, 2007

Getting Paid to Test Open Source Software

To my mind, there's really nothing better. Working on open source software with a community of programmers passionate about what they're building, and getting paid to do it. Perhaps this is becoming more common for developers, but it is certainly a rare occurrence for a quality engineer. Very few companies that I know of dedicate QA resources to open source projects.

My employer asks that I contribute my testing expertise to the Apache Hadoop project and the Apache Pig project, among others.

Tools are a big part of testing any project. Finding the right test tools, that will add value, can require careful research and analysis. This problem is somewhat constrained for open source projects, since the tools themselves should also be open sourced and (hopefully) distributable with the project. A great list of open source testing tools can be found at http://opensourcetesting.org/. In addition, some companies (and another and another) allow open source projects to use their tools for free, but not redistribute them.

On the Hadoop project, for instance, we use a number of tools that fit the above models:

• Hudson - open source continuous integration server
• Findbugs - static analyzer of Java code
• JUnit - unit test framework
• Checkstyle - coding style adherence tool
• Clover - commercial code coverage tool, donated to Apache
• Jira - commercial issue tracking tool, donated to Apache

There are many other tools that I'd like to evaluate and integrate as time permits, including PMD, Chord, MultithreadedTC. Have experience using any of these tools on open source projects? Have other tool suggestions? I'd love to hear your comments!

Want a fun job testing open source? Passionate about software quality? I'm hiring! Talk to me at: ndaley at yahoo-inc dot com. Testing and coding experience required.

Nigel Daley
Grid Computing QA Lead

Posted by ndaley at 7:41 AM | Comments (0) | TrackBack | Permalink