轉自：http://my.oschina.net/leejun2005/blog/276891?utm_source=tuicool&utm_medium=referralhtml
在許多狀況下，一個用戶須要瞭解待分析的數據，儘管這並不是所要執行的分析任務 的核心內容。以統計數據集中無效記錄數目的任務爲例，若是發現無效記錄的比例 至關高，那麼就須要認真思考爲什麼存在如此多無效記錄。是所採用的檢測程序存在 缺陷，仍是數據集質量確實很低，包含大量無效記錄？若是肯定是數據集的質量問 題，則可能須要擴大數據集的規模，以增大有效記錄的比例，從而進行有意義的 分析。
計數器是一種收集做業統計信息的有效手段，用於質量控制或應用級統計。計數器 還可輔助診斷系統故障。若是須要將日誌信息傳輸到map或reduce任務，更好的 方法一般是嘗試傳輸計數器值以監測某一特定事件是否發生。對於大型分佈式做業 而言，使用計數器更爲方便。首先，獲取計數器值比輸出日誌更方便，其次，根據 計數器值統計特定事件的發生次數要比分析一堆日誌文件容易得多。 java
Hadoop爲每一個做業維護若干內置計數器, 以描述該做業的各項指標。例如，某些計數器記錄已處理的字節數和記錄數，使用戶可監控已處理的輸入數據量和已產生的輸出數據量，並以此對 job 作適當的優化。apache
14/06/08 15:13:35 INFO mapreduce.Job: Counters: 46 File System Counters FILE: Number of bytes read=159 FILE: Number of bytes written=159447 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=198 HDFS: Number of bytes written=35 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Rack-local map tasks=1 Total time spent by all maps in occupied slots (ms)=3896 Total time spent by all reduces in occupied slots (ms)=9006 Map-Reduce Framework Map input records=3 Map output records=12 Map output bytes=129 Map output materialized bytes=159 Input split bytes=117 Combine input records=0 Combine output records=0 Reduce input groups=4 Reduce shuffle bytes=159 Reduce input records=12 Reduce output records=4 Spilled Records=24 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=13 CPU time spent (ms)=3830 Physical memory (bytes) snapshot=537718784 Virtual memory (bytes) snapshot=7365263360 Total committed heap usage (bytes)=2022309888 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=81 File Output Format Counters Bytes Written=35
計數器由其關聯任務維護，並按期傳到tasktracker，再由tasktracker傳給 jobtracker.所以，計數器可以被全局地彙集。詳見第 hadoop 權威指南第170頁的「進度和狀態的更新」小節。與其餘計數器（包括用戶定義的計數器）不一樣，內置的做業計數器實際上 由jobtracker維護，沒必要在整個網絡中發送。
一個任務的計數器值每次都是完整傳輸的，而非自上次傳輸以後再繼續數未完成的傳輸，以免因爲消息丟失而引起的錯誤。另外，若是一個任務在做業執行期間失 敗，則相關計數器值會減少。僅當一個做業執行成功以後，計數器的值纔是完整可 靠的。 網絡
MapReduce容許用戶編寫程序來定義計數器，計數器的值可在mapper或reducer 中增長。多個計數器由一個Java枚舉(enum)類型來定義，以便對計數器分組。一 個做業能夠定義的枚舉類型數量不限，各個枚舉類型所包含的字段數量也不限。枚 舉類型的名稱即爲組的名稱，枚舉類型的字段就是計數器名稱。計數器是全局的。 換言之，MapReduce框架將跨全部map和reduce彙集這些計數器，並在做業結束 時產生一個最終結果。app
Note1： 須要說明的是，不一樣的 hadoop 版本定義的方式會有些許差別。框架
（1）在0.20.x版本中使用counter很簡單,直接定義便可，如無此counter，hadoop會自動添加此counter.
Counter ct = context.getCounter("INPUT_WORDS", "count");
ct.increment(1);
（2）在0.19.x版本中,須要定義enum
enum MyCounter {INPUT_WORDS };
reporter.incrCounter(MyCounter.INPUT_WORDS, 1);
RunningJob job = JobClient.runJob(conf);
Counters c = job.getCounters();
long cnt = c.getCounter(MyCounter.INPUT_WORDS);分佈式
Notice2： 使用計數器須要清楚的是它們都存儲在jobTracker的內存裏。Mapper/Reducer 任務序列化它們，連同更新狀態被髮送。爲了運行正常且jobTracker不會出問題，計數器的數量應該在10-100個，計數器不只僅只用來聚合MapReduce job的統計值。新版本的hadoop限制了計數器的數量，以防給jobTracker帶來損害。你最不想看到的事情就是因爲定義上百個計數器而使jobTracker宕機。
下面我們來看一個計數器的實例（如下代碼請運行在 0.20.1 版本以上）： ide
3.1 測試數據：函數
hello world 2013 mapreduce hello world 2013 mapreduce hello world 2013 mapreduce
3.2 代碼：oop
/**
* Project Name:CDHJobs
* File Name:MapredCounter.java
* Package Name:tmp
* Date:2014-6-8下午2:12:48
* Copyright (c) 2014, decli#qq.com All Rights Reserved.
*
*/
package tmp; import java.io.IOException; import java.util.StringTokenizer; import org.apache.commons.lang3.StringUtils; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Counter; import org.apache.hadoop.mapreduce.CounterGroup; import org.apache.hadoop.mapreduce.Counters; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class WordCountWithCounter { static enum WordsNature { STARTS_WITH_DIGIT, STARTS_WITH_LETTER, ALL } /** * The map class of WordCount. */ public static class TokenCounterMapper extends Mapper<Object, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); } } } /** * The reducer class of WordCount */ public static class TokenCounterReducer extends Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int sum = 0; String token = key.toString(); if (StringUtils.isNumeric(token)) { context.getCounter(WordsNature.STARTS_WITH_DIGIT).increment(1); } else if (StringUtils.isAlpha(token)) { context.getCounter(WordsNature.STARTS_WITH_LETTER).increment(1); } context.getCounter(WordsNature.ALL).increment(1); for (IntWritable value : values) { sum += value.get(); } context.write(key, new IntWritable(sum)); } } /** * The main entry point. */ public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = new Job(conf, "WordCountWithCounter"); job.setJarByClass(WordCountWithCounter.class); job.setMapperClass(TokenCounterMapper.class); job.setReducerClass(TokenCounterReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path("/tmp/dsap/rawdata/june/a.txt")); FileOutputFormat.setOutputPath(job, new Path("/tmp/dsap/rawdata/june/a_result")); int exitCode = job.waitForCompletion(true) ? 0 : 1; Counters counters = job.getCounters(); Counter c1 = counters.findCounter(WordsNature.STARTS_WITH_DIGIT); System.out.println("-------------->>>>: " + c1.getDisplayName() + ": " + c1.getValue()); // The below example shows how to get built-in counter groups that Hadoop provides basically. for (CounterGroup group : counters) { System.out.println("=========================================================="); System.out.println("* Counter Group: " + group.getDisplayName() + " (" + group.getName() + ")"); System.out.println(" number of counters in this group: " + group.size()); for (Counter counter : group) { System.out.println(" ++++ " + counter.getDisplayName() + ": " + counter.getName() + ": " + counter.getValue()); } }