hadoop2.5.2 安裝部署

時間 2019-11-16

原文原文鏈接

0x00 平臺環境

OS: CentOS-6.5-x86_64
JDK: jdk-8u111-linux-x64
Hadoop: hadoop-2.6.5html

0x01 操做系統基本設置

1.1 網絡配置

1.1.1 修改主機名

//查看當前主機名
# hostname
//修改當前主機名
# vim /etc/sysconfig/network

NETWORKING 是否利用網絡
GATEWAY 默認網關
IPGATEWAYDEV 默認網關的接口名
HOSTNAME 主機名
DOMAIN 域名

1.1.2 配置靜態IP

# vim /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE 接口名（設備,網卡）
BOOTPROTO IP的配置方法（static:固定IP， dhcpHCP， none:手動）
HWADDR MAC地址
ONBOOT 系統啓動的時候網絡接口是否有效（yes/no）
TYPE 網絡類型（一般是Ethemet）
NETMASK 網絡掩碼
IPADDR IP地址
IPV6INIT IPV6是否有效（yes/no）
GATEWAY 默認網關IP地址
DNS1
DNS2

個人配置以下：java

DEVICE=eth0
HWADDR=00:0C:29:D3:53:77
TYPE=Ethernet
UUID=84d51ff5-228e-44ae-812d-7e59aa190715
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
IPADDR=192.168.1.10
GATEWAY=192.168.1.1
//虛擬機下NAT網絡模式這兩項不用配置
DNS1=202.204.65.5
DNS2=202.204.65.6

1.1.3 配置hosts文件

# vim /etc/hosts

192.168.1.10 master
192.168.1.11 slave1
192.168.1.12 slave2

1.2 關閉防火牆和SELinux

1.2.1 關閉防火牆

//臨時關閉
# service iptables stop
//永久關閉
# chkconfig iptables off
# service ip6tables stop
# chkconfig ip6tables off

1.2.2 關閉SELinux

# vim /etc/sysconfig/selinux
SELINUX=enforcing
//更改成以下配置
SELINUX=disable

接着執行以下命令node

# setenforce 0
# getenforce

1.3 創建通常用戶hadoop

若是隻有root用戶或者沒有hadoop用戶的狀況下：linux

//新增用戶
# useradd hadoop
//設置密碼
# passwd hadoop
//根據提示輸入兩次密碼

0x02 配置master免密鑰登陸slave

2.1 生成密鑰

在全部節點執行一直按回車就能夠了。git

$ su hadoop
$ ssh-keygen -t rsa

2.2 追加authorized_keys

將msater的id_rsa.pub追加到受權key中（只須要將master節點的公鑰追加到authorized_keys）web

$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keysshell

更改authorized_keys的權限，分別在全部節點操做apache

chomd 600 authorized_keysvim

2.3 複製authorized_keys

將authorized_keys複製到全部slave節點segmentfault

$ scp ~/.ssh/authorized_keys hadoop@192.168.1.11:~/.ssh/
$ scp ~/.ssh/authorized_keys hadoop@192.168.1.12:~/.ssh/

2.4 測試

master免密鑰登錄全部slave節點

$ ssh slave1
$ ssh slave2

0x03 hadoop2.5.2安裝

3.1 解壓

$ tar -zvxf hadoop-2.6.5.tar.gz
$ mv hadoop-2.6.5 ~/cloud/
$ ln -s /home/hadoop/cloud/hadoop-2.6.5 /home/hadoop/cloud/hadoop

3.2 配置環境變量

在尾部追加

# vim  /etc/profile
# set hadoop environment
export HADOOP_HOME=/home/hadoop/cloud/hadoop
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export CLASSPATH=.:$JAVA_HOME/lib:$HADOOP_HOME/lib:$CLASSPATH
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

使環境變量當即生效注意在哪一個用戶下執行該命令，環境變量在那個用戶下生效

# su hadoop
$ source /etc/profile

0x04 配置hadoop文件

4.1 core-site.xml

注意：hadoop_tmp文件夾必定要配置在存儲空間比較大的位置，不然會報錯
可能出現的問題：
(1)Unhealthy Nodes 問題
http://blog.csdn.net/korder/a...
(2)local-dirs turned bad
(3)Hadoop運行任務時一直卡在: INFO mapreduce.Job: Running job
http://www.bkjia.com/yjs/1030...

<configuration>
     <property>
            <name>fs.defaultFS</name>
            <value>hdfs://master:9000</value>
     </property>
     <property>
            <name>hadoop.tmp.dir</name>
            <value>file:/home/hadoop/cloud/hadoop/hadoop_tmp</value>
            <!--須要本身建立hadoop_tmp文件夾-->
     </property>
     <property>
            <name>io.file.buffer.size</name>
            <value>131072</value>
     </property>
     <property>
            <name>hbase.rootdir</name>
            <value>hdfs://master:9000/hbase</value>
     </property>
</configuration>

4.2 hdfs-site.xml

<configuration>
       <property>
               <name>dfs.replication</name>
               <value>2</value>
       </property>
       <property>
               <name>dfs.namenode.secondary.http-address</name>
               <value>master:9001</value>
       </property>
       <property>
              <name>dfs.namenode.name.dir</name>
              <value>file:/home/hadoop/cloud/hadoop/dfs/name</value>
              <description>namenode上存儲hdfs元數據</description>
       </property>
       <property>
               <name>dfs.datanode.data.dir</name>
               <value>file:/home/hadoop/cloud/hadoop/dfs/data</value>
               <description>datanode上數據塊物理存儲位置</description>
       </property>
       <property>
               <name>dfs.webhdfs.enabled</name>
                <value>true</value>
       </property>
</configuration>

注：訪問namenode的 webhdfs 使用50070端口，訪問datanode的webhdfs使用50075端口。要想不區分端口，直接使用namenode的IP和端口進行全部webhdfs操做，就須要在全部
datanode上都設置hdfs-site.xml中dfs.webhdfs.enabled爲true。

4.3 mapred-site.xml

<configuration>
       <property>
             <name>mapreduce.framework.name</name>
             <value>yarn</value>
      </property>
      <property>
             <name>mapreduce.jobhistory.address</name>
             <value>master:10020</value>
      </property>
      <property>
             <name>mapreduce.jobhistory.webapp.address</name>
             <value>master:19888</value>
      </property>
      <property>
             <name>mapreduce.jobtracker.http.address</name>
             <value>NameNode:50030</value>
      </property>
</configuration>

jobhistory是Hadoop自帶一個歷史服務器，記錄Mapreduce歷史做業。默認狀況下，jobhistory沒有啓動，可用如下命令啓動：

$ sbin/mr-jobhistory-daemon.sh start historyserver

4.4 yarn-site.xml

<configuration>
       <property>
              <name>yarn.nodemanager.aux-services</name>
              <value>mapreduce_shuffle</value>
       </property>
       <property>
               <name>yarn.resourcemanager.address</name>
               <value>master:8032</value>
       </property>
       <property>
              <name>yarn.resourcemanager.scheduler.address</name>
              <value>master:8030</value>
       </property>
       <property>
              <name>yarn.resourcemanager.resource-tracker.address</name>
              <value>master:8031</value>
       </property>
       <property>
               <name>yarn.resourcemanager.admin.address</name>
               <value>master:8033</value>
       </property>
       <property>
               <name>yarn.resourcemanager.webapp.address</name>
               <value>master:8088</value>
       </property>
       <property>
               <name>yarn.resourcemanager.zk-address</name>
               <value>master:2181,slave1L2181,slave2:2181</value>
       </property>
       <property>
               <name>yarn.log-aggregation-enable</name>
               <value>true</value>
       </property>
</configuration>

4.5 slaves

修改slaves文件，添加datanode節點hostname到slaves文件中

slave1
slave2

4.6 hadoop-env.sh

vim /home/hadoop/cloud/hadoop/etc/hadoop/hadoop-env.sh
export JAVA_HOME=${JAVA_HOME} -> export JAVA_HOME=/usr/java
export HADOOP_COMMON_LIB_NATIVE_DIR=/home/hadoop/hadoop/lib/native

4.7 複製到slave節點

最後，將整個/home/hadoop/cloud/hadoop-2.6.5文件夾及其子文件夾使用scp複製到Slave相同目錄中：

$ scp -r /home/hadoop/cloud/hadoop-2.6.5 hadoop@slave1:/home/hadoop/cloud/
$ scp -r /home/hadoop/cloud/hadoop-2.6.5 hadoop@slave2:/home/hadoop/cloud/

0x05 運行hadoop

5.1 格式化

確保配置文件中各文件夾已經建立

$ hdfs namenode –format

成功後顯示信息

************************************************************/
17/09/09 04:27:03 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
17/09/09 04:27:03 INFO namenode.NameNode: createNameNode [-format]
17/09/09 04:27:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Formatting using clusterid: CID-243cecfb-c003-4213-8112-b5f227616e39
17/09/09 04:27:04 INFO namenode.FSNamesystem: No KeyProvider found.
17/09/09 04:27:04 INFO namenode.FSNamesystem: fsLock is fair:true
17/09/09 04:27:04 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
17/09/09 04:27:04 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
17/09/09 04:27:04 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
17/09/09 04:27:04 INFO blockmanagement.BlockManager: The block deletion will start around 2017 Sep 09 04:27:04
17/09/09 04:27:04 INFO util.GSet: Computing capacity for map BlocksMap
17/09/09 04:27:04 INFO util.GSet: VM type       = 64-bit
17/09/09 04:27:04 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB
17/09/09 04:27:04 INFO util.GSet: capacity      = 2^21 = 2097152 entries
17/09/09 04:27:04 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
17/09/09 04:27:04 INFO blockmanagement.BlockManager: defaultReplication         = 2
17/09/09 04:27:04 INFO blockmanagement.BlockManager: maxReplication             = 512
17/09/09 04:27:04 INFO blockmanagement.BlockManager: minReplication             = 1
17/09/09 04:27:04 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
17/09/09 04:27:04 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
17/09/09 04:27:04 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
17/09/09 04:27:04 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
17/09/09 04:27:04 INFO namenode.FSNamesystem: fsOwner             = hadoop (auth:SIMPLE)
17/09/09 04:27:04 INFO namenode.FSNamesystem: supergroup          = supergroup
17/09/09 04:27:04 INFO namenode.FSNamesystem: isPermissionEnabled = false
17/09/09 04:27:04 INFO namenode.FSNamesystem: HA Enabled: false
17/09/09 04:27:04 INFO namenode.FSNamesystem: Append Enabled: true
17/09/09 04:27:05 INFO util.GSet: Computing capacity for map INodeMap
17/09/09 04:27:05 INFO util.GSet: VM type       = 64-bit
17/09/09 04:27:05 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB
17/09/09 04:27:05 INFO util.GSet: capacity      = 2^20 = 1048576 entries
17/09/09 04:27:05 INFO namenode.NameNode: Caching file names occuring more than 10 times
17/09/09 04:27:05 INFO util.GSet: Computing capacity for map cachedBlocks
17/09/09 04:27:05 INFO util.GSet: VM type       = 64-bit
17/09/09 04:27:05 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB
17/09/09 04:27:05 INFO util.GSet: capacity      = 2^18 = 262144 entries
17/09/09 04:27:05 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
17/09/09 04:27:05 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
17/09/09 04:27:05 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
17/09/09 04:27:05 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
17/09/09 04:27:05 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
17/09/09 04:27:05 INFO util.GSet: Computing capacity for map NameNodeRetryCache
17/09/09 04:27:05 INFO util.GSet: VM type       = 64-bit
17/09/09 04:27:05 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
17/09/09 04:27:05 INFO util.GSet: capacity      = 2^15 = 32768 entries
17/09/09 04:27:05 INFO namenode.NNConf: ACLs enabled? false
17/09/09 04:27:05 INFO namenode.NNConf: XAttrs enabled? true
17/09/09 04:27:05 INFO namenode.NNConf: Maximum size of an xattr: 16384
17/09/09 04:27:05 INFO namenode.FSImage: Allocated new BlockPoolId: BP-706635769-192.168.32.100-1504902425219
17/09/09 04:27:05 INFO common.Storage: Storage directory /home/hadoop/cloud/hadoop/dfs/name has been successfully formatted.
17/09/09 04:27:05 INFO namenode.FSImageFormatProtobuf: Saving image file /home/hadoop/cloud/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
17/09/09 04:27:05 INFO namenode.FSImageFormatProtobuf: Image file /home/hadoop/cloud/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 323 bytes saved in 0 seconds.
17/09/09 04:27:05 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
17/09/09 04:27:05 INFO util.ExitUtil: Exiting with status 0
17/09/09 04:27:05 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/192.168.32.100
************************************************************/

5.2 啓動hadoop

$ start-dfs.sh
$ start-yarn.sh
//能夠用一條命令來代替：
$ start-all.sh

5.3 `jps`查看進程

(1)master主節點進程：

8193 Jps
7943 ResourceManager
7624 NameNode
7802 SecondaryNameNode

(2)slave數據節點進程：

1413 DataNode
1512 NodeManager
1626 Jps

5.4 經過瀏覽器查看集羣運行狀態

概覽：http://172.16.1.156:50070/
集羣：http://172.16.1.156:8088/
JobHistory：http://172.16.1.156:19888

jobhistory是Hadoop自帶一個歷史服務器，記錄Mapreduce歷史做業。默認狀況下，jobhistory沒有啓動，可用如下命令啓動：

$ sbin/mr-jobhistory-daemon.sh start historyserver

0x06 測試hadoop

運行wordcount

6.1 創建文件

$ vi wordcount.txt
hello you
hello me
hello everyone

6.2 在HDFS上創建目錄

$ hadoop fs -mkdir /data/wordcount
$ hadoop fs –mkdir /output/

目錄/data/wordcount用來存放Hadoop自帶WordCount例子的數據文件，運行這個MapReduce任務結果輸出到/output/wordcount目錄中。

6.3 上傳文件

$ hadoop fs -put wordcount.txt/data/wordcount/

6.4 執行wordcount程序

$ hadoop jar /home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1.jar wordcount /data/wordcount /output/wordcount/

6.5 查看結果

# hadoop fs -text /output/wordcount/part-r-00000
everyone  1 
hello  3  
me    1  
you   1

0x07 搭建中遇到的問題

7.1 命令不能識別問題

在配置環境變量過程可能遇到輸入命令ls命令不能識別問題：ls -bash: ls: command not found

緣由：在設置環境變量時，編輯profile文件沒有寫正確，將export PATH=$JAVA_HOME/bin:$PATH中冒號誤寫成分號 ,致使在命令行下ls等命令不可以識別。解決方案：export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin

7.2 nodemanager進程自殺

在主機上啓動hadoop集羣，而後使用jps查看主從機上進程狀態，可以看到主機上的resourcemanager和各個從機上的nodemanager，可是過一段時間後，從機上的nodemanager就沒有了，主機上的resourcemanager還在。

緣由是防火牆處於開啓狀態：
注：nodemanager啓動後要經過心跳機制按期與RM通訊，不然RM會認爲NM死掉，會中止NM服務。

7.3 SSH鏈接慢的問題

sshd服務中設置了UseDNS yes，當配置的DNS服務器出現沒法訪問的問題，可能會形成鏈接該服務器須要等待10到30秒的時間。因爲使用UseDNS，sshd服務器會反向解析鏈接客戶端的ip，即便是在局域網中也會。
當平時鏈接都是很快，忽然變的異常的慢，多是sshd服務的服務器上配置的DNS失效，例如DNS配置的是外網的，而此時外面故障斷開。終極解決方案是不要使用UseDNS，在配置文件/etc/sshd_config(有些linux發行版在/etc/ssh/sshd_config)中找到UseDNS 設置其值爲 no，若是前面有#號，須要去掉，重啓sshd服務器便可。

vim /etc/ssh/sshd_config
UseDNS no

7.4 從新格式化HDFS報錯

FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join java.io.IOException: There appears to be a gap in the edit log. We expected txid 176531929, but got txid 176533587.

緣由：是由於namenode和datenode數據不一致引發的
解決辦法：刪除master slave節點data和name文件夾下的內容，便可解決。缺點是數據不可恢復。
另外一種解決辦法：http://blog.csdn.net/amber_am...
參考連接：
https://yq.aliyun.com/article...
https://taoistwar.gitbooks.io...

7.5 Unable to load native-hadoop library

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

I assume you're running Hadoop on 64bit CentOS. The reason you saw that warning is the native Hadoop library $HADOOP_HOME/lib/native/libhadoop.so.1.0.0 was actually compiled on 32 bit.Anyway, it's just a warning, and won't impact Hadoop's functionalities.

http://stackoverflow.com/ques...

(1)簡便的解決方法是：(後來我發現這兩步都要作)
下載64位的庫，解壓到hadoop-2.7.0/lib/native/，不在有警告
下載地址：http://dl.bintray.com/sequenc...
(2)修改hadoop-env.sh

export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=/usr/local/hadoop/lib/native"
export HADOOP_COMMON_LIB_NATIVE_DIR="/usr/local/hadoop/lib/native/"

7.6 hadoop提交jar包卡死

hadoop提交jar包卡住不會往下執行的解決方案,卡在此處：
INFO mapreduce.Job: Running job: job_1474517485267_0001
這裏咱們在集羣的yarn-site.xml中添加配置

<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>4096</value>
</property>
<property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>2048</value>
</property>
<property>
    <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>2.1</value>
</property>

從新啓動集羣，運行jar包便可

可是，並無解決個人問題，個人問題是Unhealthy Nodes，最後才發現！！可能不添加上述配置原來配置也是對的。
http://www.voidcn.com/blog/ga...

2017年1月22日, 星期日

update: 2017-06-02
增長操做系統基本設置部分
修改部分配置文件內容

update:2017.10.11遷移到segmentfault