December 22, 2007

If it hurts, automate it

In many projects, it's painful to ensure that all unit tests are run on every patch before committing. Add to this some other basic checks:

• no new javac compiler warnings
• no new Findbugs warnings
• zero javadoc warnings
• zero @author attributions

and committing patches becomes very painful and time consuming indeed.

Something that painful, of course, is begging to be automated -- and that's exactly what we did with our patching process for the Hadoop project.

Every patch for Hadoop must be attached to a Jira issue. When a Jira issue is moved into the Patch Available state by a developer, the Hadoop continuous integration server automatically picks up the issue's latest patch, applies it to a fresh checkout of trunk, builds the software, runs all the unit tests, and verifies all the other items listed above. Once the build and testing is complete, a comment like this is automatically added to the Jira issue:

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12368715/1185_20071030b.patch
against trunk revision r590273.

  @author +1. The patch does not contain any @author tags.
  javadoc +1. The javadoc tool did not generate any warning messages.
  javac +1. The applied patch does not generate any new compiler warnings.
  findbugs +1. The patch does not introduce any new Findbugs warnings.
  core tests -1. The patch failed core unit tests.
  contrib tests -1. The patch failed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1033/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1033/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1033/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1033/console

This message is automatically generated.

This automated system diminishes a Hadoop committer's burden to simply ensuring these two comments exist on the Jira issue:

• a "+1" comment from the automated patch build, and
• a "+1" comment from a code reviewer.

No more committing pain!

Nigel Daley
Grid Computing QA Lead

Want a fun job testing open source? Passionate about software quality? I'm hiring! Talk to me at: ndaley at yahoo-inc dot com. Testing and coding experience required.

Posted at December 22, 2007 8:22 AM

Comments

Posted by: Ned Collyer at February 4, 2008 3:59 PM | Permalink

Posted by: Nigel Daley at February 6, 2008 2:52 PM | Permalink