【hadoop】hadoop3.2.0的安裝並測試

前言：前段時間將hadoop01的虛擬機弄的崩潰掉了，也沒有備份，從新從hadoop02虛擬上克隆過來的，結果hadoop-eclipse插件同樣的編譯，竟然用不起了，找了3天的緣由，最後仍是沒有解決，只能用hadoop shell 命令去測試了，反正影響不大，只不過用着不方便而已。

心累中...........

正文：

解壓安裝Hadoop

[hadoop@hadoop01 ~]$ cp /home/hadoop/Resources/hadoop-3.2.0.tar.gz ~/
[hadoop@hadoop01 ~]$ tar -zxvf ~/hadoop-3.2.0.tar.gz
[hadoop@hadoop01 ~]$ cd hadoop-3.2.0
[hadoop@hadoop01 hadoop-3.2.0]$ ls -l
total 184
drwxr-xr-x. 2 hadoop hadoop    203 Jan  8  2019 bin
drwxr-xr-x. 3 hadoop hadoop     20 Jan  8  2019 etc
drwxr-xr-x. 2 hadoop hadoop    106 Jan  8  2019 include
drwxr-xr-x. 3 hadoop hadoop     20 Jan  8  2019 lib
drwxr-xr-x. 4 hadoop hadoop   4096 Jan  8  2019 libexec
-rw-rw-r--. 1 hadoop hadoop 150569 Oct 19  2018 LICENSE.txt
-rw-rw-r--. 1 hadoop hadoop  22125 Oct 19  2018 NOTICE.txt
-rw-rw-r--. 1 hadoop hadoop   1361 Oct 19  2018 README.txt
drwxr-xr-x. 3 hadoop hadoop   4096 Jan  8  2019 sbin
drwxr-xr-x. 4 hadoop hadoop     31 Jan  8  2019 share

配置Hadoop環境變量

[hadoop@hadoop01 hadoop-3.2.0]$ gedit /home/hadoop/hadoop-3.2.0/etc/hadoop/hadoop-env.sh
編輯文件並保存：
# The java implementation to use. By default, this environment
# variable is REQUIRED on ALL platforms except OS X!
# export JAVA_HOME=
export JAVA_HOME=/usr/java/jdk1.8.0_11/

配置YARN環境變量

[hadoop@hadoop01 hadoop-3.2.0]$ gedit /home/hadoop/hadoop-3.2.0/etc/hadoop/yarn-env.sh
編輯文件並保存：
export JAVA_HOME=/usr/java/jdk1.8.0_11/

 

配置核心組件文件（core-site.xml）

[hadoop@hadoop01 hadoop-3.2.0]$ gedit /home/hadoop/hadoop-3.2.0/etc/hadoop/core-site.xml
編輯文件並保存：
<configuration>
<!--hdfs 的默認地址、端口 訪問地址-->
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop01:9802</value>
</property>
<!--hdfs臨時路徑-->
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hadoopdata</value>
</property>
</configuration>

配置文件系統（hdfs-site.xml）

[hadoop@hadoop01 hadoop-3.2.0]$ gedit /home/hadoop/hadoop-3.2.0/etc/hadoop/hdfs-site.xml
編輯文件並保存：
<configuration>
<!--hdfs web的地址 -->
<property>
    <name>dfs.namenode.http-address</name>
    <value>hadoop01:50070</value>
</property>
<!-- 副本數-->
<property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
<!-- 是否啓用hdfs權限檢查 false 關閉 -->
       <property>
        <name>dfs.permissions.enabled</name>
        <value>false</value>
    </property>
<!-- 塊大小,默認字節, 可以使用 k m g t p e-->
       <property>
        <name>dfs.blocksize</name>
    <!--128m-->
        <value>134217728</value>
    </property>
    <!--hadoop的name和data目錄路徑-->
       <property>
         <name>dfs.namenode.name.dir</name>
         <value>file:/home/hadoop/hdfs/name</value>
   </property>
   <property>
         <name>dfs.datanode.data.dir</name>
         <value>file:/home/hadoop/hdfs/data</value>
   </property>
</configuration>

配置yarn-site.xml文件

[hadoop@hadoop01 hadoop-3.2.0]$ gedit /home/hadoop/hadoop-3.2.0/etc/hadoop/yarn-site.xml
編輯文件並保存：
<configuration>
<!-- Site specific YARN configuration properties -->
<!--集羣master，-->
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hadoop01</value>
</property>

<!-- NodeManager上運行的附屬服務-->
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<!--容器可能會覆蓋的環境變量，而不是使用NodeManager的默認值-->
<property>
        <name>yarn.nodemanager.env-whitelist</name>
    <value> JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ</value>
</property>
<!-- 關閉內存檢測，虛擬機須要，不配會報錯-->
<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>
</configuration>

 

配置MapReduce計算框架文件
[hadoop@hadoop01 hadoop-3.2.0]$ gedit /home/hadoop/hadoop-3.2.0/etc/hadoop/mapred-site.xml
編輯文件並保存：

<configuration>
<!--local表示本地運行，classic表示經典mapreduce框架，yarn表示新的框架-->
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>
<!--若是map和reduce任務訪問本地庫（壓縮等），則必須保留原始值
當此值爲空時，設置執行環境的命令將取決於操做系統：
Linux：LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native.
windows：PATH =%PATH%;%HADOOP_COMMON_HOME%\\bin.
-->
<property>
        <name>mapreduce.admin.user.env</name>
        <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<!--
能夠設置AM【AppMaster】端的環境變量
若是上面缺乏配置，可能會形成mapreduce失敗
-->
<property>
        <name>yarn.app.mapreduce.am.env</name>
        <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
</configuration>

 

【選】配置slaves文件（hadoop2.x修改slaves）
[hadoop@hadoop01 hadoop-3.2.0]$ gedit /home/hadoop/hadoop-3.2.0/etc/hadoop/slaves
編輯文件並保存：

hadoop02
hadoop03

【選】配置workers文件（hadoop3.x修改workers）
[hadoop@hadoop01 hadoop-3.2.0]$ gedit /home/hadoop/hadoop-3.2.0/etc/hadoop/workers
編輯文件並保存：

hadoop01
hadoop02
hadoop03

複製hadoop01上的Hadoop到hadoop02和hadoop03節點上
scp -r /home/hadoop/hadoop-3.2.0 hadoop@hadoop02:~/
scp -r /home/hadoop/hadoop-3.2.0 hadoop@hadoop03:~/

配置操做系統環境變量（須要在全部節點上進行，且使用通常用戶權限）
gedit ~/.bash_profile
source ~/.bash_profile
編輯文件並保存：

#如下是新添加入代碼
export JAVA_HOME=/usr/java/jdk1.8.0_11/
export PATH=$JAVA_HOME/bin:$PATH
#hadoop
export HADOOP_HOME=/home/hadoop/hadoop-3.2.0
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

建立Hadoop數據目錄（全部節點操做）
mkdir /home/hadoop/hadoopdata

格式化文件系統（主端進行）
hdfs namenode -format

啓動和關閉Hadoop
cd ~/hadoop-3.2.0
sbin/start-all.sh
stop-all.sh

啓動成功結果：

[hadoop@hadoop01 hadoop-3.2.0]$ jps
20848 DataNode
21808 Jps
21076 SecondaryNameNode
21322 ResourceManager
20668 NameNode
21468 NodeManager
[hadoop@hadoop01 hadoop-3.2.0]$ 

【最後測試】在Hadoop集羣中運行程序
將計算圓周率pi的Java程序包投入運行

[hadoop@hadoop01 hadoop-3.2.0]$ cd ~/hadoop-3.2.0/share/hadoop/mapreduce
[hadoop@hadoop01 mapreduce]$ ls
hadoop-mapreduce-client-app-3.2.0.jar     hadoop-mapreduce-client-hs-plugins-3.2.0.jar       hadoop-mapreduce-client-shuffle-3.2.0.jar   lib
hadoop-mapreduce-client-common-3.2.0.jar  hadoop-mapreduce-client-jobclient-3.2.0.jar        hadoop-mapreduce-client-uploader-3.2.0.jar  lib-examples
hadoop-mapreduce-client-core-3.2.0.jar    hadoop-mapreduce-client-jobclient-3.2.0-tests.jar  hadoop-mapreduce-examples-3.2.0.jar         sources
hadoop-mapreduce-client-hs-3.2.0.jar      hadoop-mapreduce-client-nativetask-3.2.0.jar       jdiff
[hadoop@hadoop01 mapreduce]$ hadoop jar hadoop-mapreduce-examples-3.2.0.jar
An example program must be given as the first argument.
Valid program names are:
  aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
  aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
  bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
  dbcount: An example job that count the pageview counts from a database.
  distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
  grep: A map/reduce program that counts the matches of a regex in the input.
  join: A job that effects a join over sorted, equally partitioned datasets
  multifilewc: A job that counts words from several files.
  pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
  pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
  randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
  randomwriter: A map/reduce program that writes 10GB of random data per node.
  secondarysort: An example defining a secondary sort to the reduce.
  sort: A map/reduce program that sorts the data written by the random writer.
  sudoku: A sudoku solver.
  teragen: Generate data for the terasort
  terasort: Run the terasort
  teravalidate: Checking results of terasort
  wordcount: A map/reduce program that counts the words in the input files.
  wordmean: A map/reduce program that counts the average length of the words in the input files.
  wordmedian: A map/reduce program that counts the median length of the words in the input files.
  wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files.
[hadoop@hadoop01 mapreduce]$ hadoop jar hadoop-mapreduce-examples-3.2.0.jar pi 10 10
Number of Maps  = 10
Samples per Map = 10
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
2019-08-27 13:47:11,866 INFO client.RMProxy: Connecting to ResourceManager at hadoop01/192.168.1.100:8032
2019-08-27 13:47:12,179 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1566884685380_0001
2019-08-27 13:47:12,285 INFO input.FileInputFormat: Total input files to process : 10
2019-08-27 13:47:12,341 INFO mapreduce.JobSubmitter: number of splits:10
2019-08-27 13:47:12,372 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2019-08-27 13:47:12,479 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1566884685380_0001
2019-08-27 13:47:12,480 INFO mapreduce.JobSubmitter: Executing with tokens: []
2019-08-27 13:47:12,645 INFO conf.Configuration: resource-types.xml not found
2019-08-27 13:47:12,645 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2019-08-27 13:47:13,018 INFO impl.YarnClientImpl: Submitted application application_1566884685380_0001
2019-08-27 13:47:13,099 INFO mapreduce.Job: The url to track the job: http://hadoop01:8088/proxy/application_1566884685380_0001/
2019-08-27 13:47:13,099 INFO mapreduce.Job: Running job: job_1566884685380_0001
2019-08-27 13:47:20,205 INFO mapreduce.Job: Job job_1566884685380_0001 running in uber mode : false
2019-08-27 13:47:20,209 INFO mapreduce.Job:  map 0% reduce 0%
2019-08-27 13:47:27,371 INFO mapreduce.Job:  map 20% reduce 0%
2019-08-27 13:47:46,535 INFO mapreduce.Job:  map 20% reduce 7%
2019-08-27 13:47:50,559 INFO mapreduce.Job:  map 40% reduce 7%
2019-08-27 13:47:51,570 INFO mapreduce.Job:  map 50% reduce 7%
2019-08-27 13:47:53,586 INFO mapreduce.Job:  map 60% reduce 7%
2019-08-27 13:47:58,631 INFO mapreduce.Job:  map 60% reduce 20%
2019-08-27 13:47:59,641 INFO mapreduce.Job:  map 80% reduce 20%
2019-08-27 13:48:00,665 INFO mapreduce.Job:  map 100% reduce 20%
2019-08-27 13:48:01,682 INFO mapreduce.Job:  map 100% reduce 100%
2019-08-27 13:48:01,708 INFO mapreduce.Job: Job job_1566884685380_0001 completed successfully
2019-08-27 13:48:01,780 INFO mapreduce.Job: Counters: 54
    File System Counters
        FILE: Number of bytes read=226
        FILE: Number of bytes written=2443397
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=2640
        HDFS: Number of bytes written=215
        HDFS: Number of read operations=45
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=3
        HDFS: Number of bytes read erasure-coded=0
    Job Counters
        Launched map tasks=10
        Launched reduce tasks=1
        Data-local map tasks=10
        Total time spent by all maps in occupied slots (ms)=270199
        Total time spent by all reduces in occupied slots (ms)=31653
        Total time spent by all map tasks (ms)=270199
        Total time spent by all reduce tasks (ms)=31653
        Total vcore-milliseconds taken by all map tasks=270199
        Total vcore-milliseconds taken by all reduce tasks=31653
        Total megabyte-milliseconds taken by all map tasks=276683776
        Total megabyte-milliseconds taken by all reduce tasks=32412672
    Map-Reduce Framework
        Map input records=10
        Map output records=20
        Map output bytes=180
        Map output materialized bytes=280
        Input split bytes=1460
        Combine input records=0
        Combine output records=0
        Reduce input groups=2
        Reduce shuffle bytes=280
        Reduce input records=20
        Reduce output records=0
        Spilled Records=40
        Shuffled Maps =10
        Failed Shuffles=0
        Merged Map outputs=10
        GC time elapsed (ms)=67681
        CPU time spent (ms)=63700
        Physical memory (bytes) snapshot=2417147904
        Virtual memory (bytes) snapshot=30882955264
        Total committed heap usage (bytes)=2966421504
        Peak Map Physical memory (bytes)=382750720
        Peak Map Virtual memory (bytes)=2810384384
        Peak Reduce Physical memory (bytes)=181923840
        Peak Reduce Virtual memory (bytes)=2815541248
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters
        Bytes Read=1180
    File Output Format Counters
        Bytes Written=97
Job Finished in 49.977 seconds
Estimated value of Pi is 3.20000000000000000000