MRUNIT hadoop逐步調試工具！

MRUNIT hadoop MapReduce逐步調試工具！

MRUnit簡介
MRUnit是一款由Couldera公司開發的專門針對Hadoop中編寫MapReduce單元測試的框架。能夠用MapDriver單獨測試
Map，用ReduceDriver單獨測試Reduce，用MapReduceDriver測試MapReduce做業。
實戰
咱們將利用MRUnit對本系列上篇文章MapReduce基本編程中的字數統計功能進行單元測試。
加入MRUnit依賴<dependency>
<groupId>com.cloudera.hadoop</groupId>
<artifactId>hadoop-mrunit</artifactId>
<version>0.20.2-320</version>
<scope>test</scope>
</dependency>
單獨測試Map
public class WordCountMapperTest {
private Mapper mapper;
private MapDriver driver;
@Before
public void init(){
mapper = new WordCountMapper();
driver = new MapDriver(mapper);
}
@Test
public void test() throws IOException{
String line = "Taobao is a great website";
driver.withInput(null,new Text(line))
.withOutput(new Text("Taobao"),new IntWritable(1))
.withOutput(new Text("is"), new IntWritable(1))
.withOutput(new Text("a"), new IntWritable(1))
.withOutput(new Text("great"), new IntWritable(1))
.withOutput(new Text("website"), new IntWritable(1))
.runTest();
}
}
上面的例子經過MapDriver的withInput和withOutput組織map函數的輸入鍵值和期待的輸出鍵值，經過runTest方法運行
做業，測試Map函數。測試運行經過。
單獨測試Reduce
public class WordCountReducerTest {
private Reducer reducer;
private ReduceDriver driver;
@Before
public void init(){
reducer = new WordCountReducer();
driver = new ReduceDriver(reducer);
}
@Test
public void test() throws IOException{
String key = "taobao";
List values = new ArrayList();

values.add(new IntWritable(2));
values.add(new IntWritable(3));
driver.withInput(new Text("taobao"), values)
.withOutput(new Text("taobao"), new IntWritable(5))
.runTest();
}
}
上面的例子的測試Map函數的寫法相似，測試reduce函數，
由於reduce函數實現相加功能，所以咱們假設輸入爲<taobao,[2,3]>，
則期待結果應該爲<taobao,5>.測試運行經過。
測試MapReduce
public class WordCountTest {
private Mapper mapper;
private Reducer reducer;
private MapReduceDriver driver;
@Before
public void init(){
mapper = new WordCountMapper();
reducer = new WordCountReducer();
driver = new MapReduceDriver(mapper,reducer);
}
@Test
public void test() throws RuntimeException, IOException{
String line = "Taobao is a great website, is it not?";
driver.withInput("",new Text(line))
.withOutput(new Text("Taobao"),new IntWritable(1))
.withOutput(new Text("a"),new IntWritable(1))
.withOutput(new Text("great"),new IntWritable(1))
.withOutput(new Text("is"),new IntWritable(2))
.withOutput(new Text("it"),new IntWritable(1))
.withOutput(new Text("not"),new IntWritable(1))
.withOutput(new Text("website"),new IntWritable(1))
.runTest();
}
}
此次咱們測試MapReduce的做業，經過MapReduceDriver的withInput構造map函數的輸入鍵值，經過withOutput構造
reduce函數的輸出鍵值。來測試這個字數統計功能，此次運行測試時拋出了異常，測試沒有經過但沒有詳細junit異常信
息，在控制檯顯示