hadoop運行wordcount實例，hdfs簡單操做

時間 2019-12-12

標籤 hadoop 運行 wordcount 實例 hdfs 簡單欄目 Hadoop 简体版

原文原文鏈接

1.查看hadoop版本node

[hadoop@ltt1 sbin]$ hadoop version
Hadoop 2.6.0-cdh5.12.0
Subversion http://github.com/cloudera/hadoop -r dba647c5a8bc5e09b572d76a8d29481c78d1a0dd
Compiled by jenkins on 2017-06-29T11:33Z
Compiled with protoc 2.5.0
From source with checksum 7c45ae7a4592ce5af86bc4598c5b4
This command was run using /home/hadoop/hadoop260/share/hadoop/common/hadoop-common-2.6.0-cdh5.12.0.jar

2.經過hadoop自帶的jar文件，能夠簡單測試一些功能。git

提君博客原創github

查看hadoop-mapreduce-examples-2.6.0-cdh5.12.0.jar文件所支持的MapReduce功能列表app

[hadoop@ltt1 sbin]$ hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.12.0.jar
An example program must be given as the first argument.
Valid program names are:
aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
dbcount: An example job that count the pageview counts from a database.
distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
grep: A map/reduce program that counts the matches of a regex in the input.
join: A job that effects a join over sorted, equally partitioned datasets
multifilewc: A job that counts words from several files.
pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
randomwriter: A map/reduce program that writes 10GB of random data per node.
secondarysort: An example defining a secondary sort to the reduce.
sort: A map/reduce program that sorts the data written by the random writer.
sudoku: A sudoku solver.
teragen: Generate data for the terasort
terasort: Run the terasort
teravalidate: Checking results of terasort
wordcount: A map/reduce program that counts the words in the input files.
wordmean: A map/reduce program that counts the average length of the words in the input files.
wordmedian: A map/reduce program that counts the median length of the words in the input files.
wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files.

3.在hdfs上建立文件夾dom

hadoop fs -mkdir /input

4.查看hdfs的更目錄列表ide

[hadoop@ltt1 ~]$ hadoop fs -ls /
Found 2 items
drwxr-xr-x - hadoop supergroup 0 2017-09-17 08:11 /input
drwx------ - hadoop supergroup 0 2017-09-17 08:07 /tmpoop

5.上傳本地文件到hdfs測試

hadoop fs -put $HADOOP_HOME/*.txt /input

6.查看hdfs上input目錄下文件url

[hadoop@ltt1 ~]$ hadoop fs -ls /input
Found 3 items
-rw-r--r--   2 hadoop supergroup      85063 2017-09-17 08:15 /input/LICENSE.txt
-rw-r--r--   2 hadoop supergroup      14978 2017-09-17 08:15 /input/NOTICE.txt
-rw-r--r--   2 hadoop supergroup       1366 2017-09-17 08:15 /input/README.txt

7.wordcount簡單測試。spa

提君博客原創

[hadoop@ltt1 ~]$ hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.12.0.jar wordcount /input /output
17/09/17 08:19:12 INFO input.FileInputFormat: Total input paths to process : 3
17/09/17 08:19:13 INFO mapreduce.JobSubmitter: number of splits:3
17/09/17 08:19:13 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1505605169997_0002
17/09/17 08:19:14 INFO impl.YarnClientImpl: Submitted application application_1505605169997_0002
17/09/17 08:19:14 INFO mapreduce.Job: The url to track the job: http://ltt1.bg.cn:9180/proxy/application_1505605169997_0002/
17/09/17 08:19:14 INFO mapreduce.Job: Running job: job_1505605169997_0002
17/09/17 08:19:27 INFO mapreduce.Job: Job job_1505605169997_0002 running in uber mode : false
17/09/17 08:19:27 INFO mapreduce.Job:  map 0% reduce 0%
17/09/17 08:19:39 INFO mapreduce.Job:  map 33% reduce 0%
17/09/17 08:19:48 INFO mapreduce.Job:  map 100% reduce 0%
17/09/17 08:19:50 INFO mapreduce.Job:  map 100% reduce 100%
17/09/17 08:19:50 INFO mapreduce.Job: Job job_1505605169997_0002 completed successfully
17/09/17 08:19:50 INFO mapreduce.Job: Counters: 50
>>提君博客原創  http://www.cnblogs.com/tijun/  <<

    File System Counters
        FILE: Number of bytes read=42705
        FILE: Number of bytes written=588235
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=101699
        HDFS: Number of bytes written=30167
        HDFS: Number of read operations=12
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=3
        Launched reduce tasks=1
        Data-local map tasks=2
        Rack-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=47617
        Total time spent by all reduces in occupied slots (ms)=8244
        Total time spent by all map tasks (ms)=47617
        Total time spent by all reduce tasks (ms)=8244
        Total vcore-milliseconds taken by all map tasks=47617
        Total vcore-milliseconds taken by all reduce tasks=8244
        Total megabyte-milliseconds taken by all map tasks=48759808
        Total megabyte-milliseconds taken by all reduce tasks=8441856
    Map-Reduce Framework
        Map input records=2035
        Map output records=14239
        Map output bytes=155828
        Map output materialized bytes=42717
        Input split bytes=292
        Combine input records=14239
        Combine output records=2653
        Reduce input groups=2402
        Reduce shuffle bytes=42717
        Reduce input records=2653
        Reduce output records=2402
        Spilled Records=5306
        Shuffled Maps =3
        Failed Shuffles=0
        Merged Map outputs=3
        GC time elapsed (ms)=881
        CPU time spent (ms)=22320
        Physical memory (bytes) snapshot=690192384
        Virtual memory (bytes) snapshot=10862809088
        Total committed heap usage (bytes)=380243968
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=101407
    File Output Format Counters 
        Bytes Written=30167

8.查看wordcount運行結果（因爲結果太長，只舉出了部分結果）

[hadoop@ltt1 ~]$ hadoop fs -cat /output/*
worldwide,    4
would    1
writing    2
writing,    4
written    19
xmlenc    1
year    1
you    12
your    5
zlib    1
 252.227-7014(a)(1))    1
§    1
「AS    1
「Contributor    1
「Contributor」    1
「Covered    1
「Executable」    1
「Initial    1
「Larger    1
「Licensable」    1
「License」    1
「Modifications」    1
「Original    1
「Participant」)    1
「Patent    1
「Source    1
「Your」)    1
「You」    2
「commercial    3
「control」    1

>>提君博客原創 http://www.cnblogs.com/tijun/ <<

至此，經過一個wordcount的一個小栗子，簡介實踐了一下hdfs的建立文件夾，上傳文件，查看目錄，運行wordcount實例。

提君博客原創

>>提君博客原創 http://www.cnblogs.com/tijun/ <<