三.hadoop mapreduce之WordCount例子

目錄：

目錄見文章1

 

這個案列完成對單詞的計數，重寫map,與reduce方法，完成對mapreduce的理解。 

Mapreduce初析

　　Mapreduce是一個計算框架，既然是作計算的框架，那麼表現形式就是有個輸入（input），mapreduce操做這個輸入（input），經過自己定義好的計算模型，獲得一個輸出（output），這個輸出就是咱們所須要的結果。

　　咱們要學習的就是這個計算模型的運行規則。在運行一個mapreduce計算任務時候，任務過程被分爲兩個階段：map階段和reduce階段，每一個階段都是用鍵值對（key/value）做爲輸入（input）和輸出（output）。而程序員要作的就是定義好這兩個階段的函數：map函數和reduce函數。

 

 1.準備 w.txt 文件,用於當測試數據

yaojiale hahaha 
yaojiale llllll  

 

2.構建maven項目，將WordCount類打包成mrtest.jar.丟到hadoop所在服務器上

 pom.xml

<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.7.3</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-mapreduce-client-core -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-core</artifactId>
            <version>2.7.3</version>
        </dependency>
        <!-- 加上這個就不報本地某錯了 Cannot initialize Cluster 
  https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-mapreduce-client-common -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-common</artifactId>
            <version>2.6.4</version>
        </dependency>

 

WordCount.java 代碼：

package org.apache.hadoop.examples; import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.util.GenericOptionsParser; public class WordCount { //WordCOuntMap方法接收LongWritable，Text的參數，返回<Text， IntWriatable>鍵值對。
  public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{ private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); } } } public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values,Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs(); if (otherArgs.length != 2) { System.err.println("Usage: wordcount <in> <out>"); System.exit(2); } Job job = new Job(conf, "word count"); job.setJarByClass(WordCount.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(otherArgs[0])); FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } }

 2.將w.txt放到hdfs下（folder下有w.txt文件）

bin/hdfs dfs -put /usr/software/folder input

而後查看

root@ubuntu:/usr/software/hadoop# bin/hdfs dfs -ls
Found 1 items
drwxr-xr-x   - root supergroup          0 2018-07-16 21:50 input //內有w.txt文件

3.運行程序統計WordCount

bin/hadoop jar /usr/software/mrtest2.jar input output

而後查看可得

root@ubuntu:/usr/software/hadoop# bin/hdfs dfs -ls
Found 2 items
drwxr-xr-x - root supergroup 0 2018-07-16 21:50 input
drwxr-xr-x - root supergroup 0 2018-07-16 22:18 output

root@ubuntu:/usr/software/hadoop# bin/hdfs dfs -cat output/*
hahaha 1
llllll 1
yaojiale 2

完畢。

附錄：附上一個hadoop自帶的例子：

計算圓周率

 bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.6.jar pi 4 1000


result:

　Estimated value of Pi is 3.14000000000000000000

 

 

 

 參考文章： 

Hadoop之MapReduce的HelloWorld（七）

代碼詳細解釋