Nick Dimiduk

n10k.com: blog et al.

How to Contribute to HBase and Hadoop2

In case you haven’t heard, Hadoop2 is on the way! There are loads more new features than I can begin to enumerate, including lots of interesting enhancements to HDFS for online applications like HBase. One of the most anticipated new features is YARN, an entirely new way to think about deploying applications across your Hadoop cluster. It’s easy to think of YARN as the infrastructure necessary to turn Hadoop into a cloud-like runtime for deploying and scaling data-centric applications. Early examples of such applications are rare, but two noteworthy examples are Knitting Boar and Storm on YARN. Hadoop2 will also ship a MapReduce implementation built on top of YARN that is binary compatible with applications written for MapReduce on Hadoop-1.x.

The HBase project is rearing to get onto this new platform as well. Hadoop2 will be a fully supported deployment environment for HBase 0.96 release. There are still lots of bugs to squish and the build lights aren’t green yet. That’s where you come in!

Squash your first HBase + Hadoop2 bug

Start by grabbing the HBase source if you don’t have it already.

$ git clone git://git.apache.org/hbase.git
$ cd hbase

To build it, you’ll also need a modern version of maven installed. Installation is platform dependent.

To build HBase against Hadoop2, you’ll need to enable the appropriate profile. Use the -Dhadoop.profile=2.0 argument to enable it. The first time you build HBase will take a while as you’ll need to download a bunch of dependencies. Maven takes care of this for you.

$ mvn -Dhadoop.profile=2.0 clean package -DskipTests
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

If everything went well, you now have a working build of HBase vs Hadoop2. Now go back to Jenkins and pick your favorite regression. TestImportTsv looks interesting. HBase uses the Surefire plugin to handle tests, which makes it easy to run just a single test.

$ mvn -Dhadoop.profile=2.0 test -Dtest=org.apache.hadoop.hbase.mapreduce.TestImportTsv#testBulkOutputWithAnExistingTable
...
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.apache.hadoop.hbase.mapreduce.TestImportTsv
...
*boom*

Now that you’ve reproduced the bug, dive into the source, hack up a solution, and rebuild and rerun the test. All the usual maven build phases apply, just be sure you always include the -Dhadoop.profile=2.0 in your build commands. Once you have a working solution, find the relevant JIRA (ie, HBASE-8454), generate a patch, and post your fix.

Congratulations! You’re now an HBase contributor :D

What’s that? You’re only just now getting to read this post and TestImportTsv has already been fixed? Don’t worry, there’s plenty more issues to tackle.

Comments