hadoop的HA機制+zookeeper

 關於hadoop的HA配置以及wordcount測試

一，簡單環境配置

1，查看centos版本位數：

$>getconf LONG_BIT,

2，桌面模式和文本模式之間進行切換：

1），在終端命令行進行設置時只能暫時改變模式，

     $>init 3    表示切換到文本模式

     $>init 5    表示切換到桌面模式

2），永久改變模式須要修改配置文件，進入到etc目錄下

      $>sudo nano inittab   修改該文件最後一行

     若須要文本模式則改成 id:3:initdefault

     若須要桌面模式則改成：id:5:initdefault

注意：改成桌面模式時最好加大內存，改成4G左右

3，配置靜態ip,修改主機名，配置hosts文件

    1），進入桌面模式下進行配置，

2），查看配置的靜態ip

     cd /etc/sysconfig/networking/devices/ifcfg-eth2

3）,修改主機名

          nano /etc/sysconfig/network  本機的主機名

4，域名映射的配置：

      nano /etc/hosts  集羣中的主機域名映射表

5，檢驗配置是否生效

ping hostname觀察網絡配置是否生效

6，關閉防火牆

          1）查看防火牆狀態

                   service iptables status

          2）關閉防火牆

             service iptables stop

          3）查看防火牆開機啓動狀態

                   chkconfig iptables --list

          4）關閉防火牆開機啓動

                   chkconfig iptables off

7，配置ssh免登錄

           1）生成ssh免登錄密鑰

              進入到個人home目錄

                 cd ~/.ssh

ssh-keygen -t rsa （四個回車）

           2）執行完這個命令後，會生成兩個文件id_rsa（私鑰）、id_rsa.pub（公鑰）

              將公鑰拷貝到要免登錄的機器上

                 ssh-copy-id localhost

        

 

二，配置HA

1,讓兩個NN節點在某個時間只能有一個節點正常響應客戶端請求，響應請求的必須爲ACTIVE狀態的那一臺

2，standby狀態的節點（第二個namenode）必須可以無縫的切換爲ACTIVE狀態，意味着兩個NN必須時時刻刻保持一致

3，管理edits，寫了一個分佈式應用qjournal:它依賴於zookeeper實現

4，如何避免狀態切換時發生brain split現象

   fencing: ssh發送kill指令

   執行自定義shell腳本

  

5，部署方式：

3臺機器：

第一臺                 第二臺                              第三臺

nn1                    nn2

zkfc1                  zkfc2                                   dn1                                                                   

zk1,                   zk2,                                    zk3

jn1,                   jn2,                                     jn3

RM                     RM                                    NM

 

7臺機器：

一、        namenode                    zkfc

2,         namenode                    zkfc

3,         resourcemanager

4,         resourcemanager

5,         zookeeper  journalnode  datanode  nodemanager

6,         zookeeper  journalnode  datanode  nodemanager

7,         zookeeper  journalnode  datanode  nodemanager

hadoop2.0已經發布了穩定版本了，增長了不少特性，好比HDFS HA、YARN等。最新的hadoop-2.4.1又增長了YARN HA

前期準備就不詳細說了，我下面的HA配置選擇的是3個節點

1.修改Linux主機名

2.修改IP

3.修改主機名和IP的映射關係

4.關閉防火牆

5.ssh免登錄

6.安裝JDK，配置環境變量等

集羣規劃：

         主機名              IP                       安裝的軟件                                          運行的進程

         s104               192.168.43.104  jdk、hadoop,zookeeper1            NN1,zkfc1,RM1,QPM,JN1 

         s106               192.168.43.106     jdk、hadoop,zookeeper2      NN2,zkfc2,RM2,QPM,JN2 

         s108               192.168.43.108     jdk、hadoop、zookeeper3        DN,NM,QPM,JN3     

說明：

1.在hadoop2.0中一般由兩個NameNode組成，一個處於active狀態，另外一個處於standby狀態。Active NameNode對外提供服務，而Standby NameNode則不對外提供服務，僅同步active namenode的狀態，以便可以在它失敗時快速進行切換。

         hadoop2.0官方提供了兩種HDFS HA的解決方案，一種是NFS，另外一種是QJM。這裏咱們使用簡單的QJM。在該方案中，主備NameNode之間經過一組JournalNode同步元數據信息，一條數據只要成功寫入多數JournalNode即認爲寫入成功。一般配置奇數個JournalNode

         這裏還配置了一個zookeeper集羣，用於ZKFC（DFSZKFailoverController）故障轉移，當Active NameNode掛掉了，會自動切換Standby NameNode爲standby狀態

2.hadoop-2.2.0中依然存在一個問題，就是ResourceManager只有一個，存在單點故障，hadoop-2.4.1解決了這個問題，有兩個ResourceManager，一個是Active，一個是Standby，狀態由zookeeper進行協調

安裝步驟：

1.安裝配置zooekeeper集羣（在s104上）

1).上傳zk安裝包到~/目錄下

2).解壓 tar -zxvf zookeeper-3.4.5.tar.gz -C app

3),進入zookeeper-3.4.5，而後進入到文件夾conf進行配置（先在一臺節點上配置）

3.1添加一個zoo.cfg配置文件

                  $>zookeeper-3.4.5/conf

                  $>mv zoo_sample.cfg zoo.cfg

3.2修改配置文件（zoo.cfg）

                        dataDir=/home/hadoop/app/zookeeper-3.4.5/data

                         server.1=itcast05:2888:3888

                         server.2=itcast06:2888:3888

                         server.3=itcast07:2888:3888

3.3在（dataDir=/home/hadoop/app/zookeeper-3.4.5/data）建立一個myid文件，裏面內容是server.N中的N（server.2裏面內容爲2）

                       echo "1" > myid

3.4將配置好的zk拷貝到其餘節點

                       scp -r /home/hadoop/app/zookeeper-3.4.5/ s106:/home/hadoop/app

                       scp -r /home/hadoop/app/zookeeper-3.4.5/ s108:/home/hadoop/app

3.5注意：在其餘節點上必定要修改myid的內容

                      在s106應該講myid的內容改成2 （echo "2" > myid）

                      在s108應該講myid的內容改成3 （echo "3" > myid）

4).啓動集羣(後面配置完成進行集羣測試時再關閉，而後按步驟啓動)

                分別啓動zk,進入到zookeeper的bin目錄下

       ./zkServer.sh start

           默認端口號查看：netstat -nltp | grep 2181

           查看運行狀態： ./zkServer.sh status

5),設置成功後，進入命令行客戶端，鏈接到集羣訪問數據，作測試

           cd zookeeper-3.4.5/bin          ./zkCli.sh

6)，zookeeper管理客戶存放的數據採用的是相似於文件樹的結構，每個節點叫作node。

           注意：zookeeper集羣可以開啓，開啓的節點數必須超過配置的一半    

2.安裝配置hadoop集羣（在s104上操做）

2.1解壓

                            tar -zxvf hadoop-2.4.1.tar.gz -C app

2.2配置HDFS（hadoop2.0全部的配置文件都在$HADOOP_HOME/etc/hadoop目錄下）

                     #將hadoop添加到環境變量中

                            vim /etc/profile

                            export JAVA_HOME=/home/hadoop/app/jdk1.7.0_65

                            export HADOOP_HOME=/home/hadoop/app/hadoop-2.4.1

                            export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin

                     #hadoop2.0的配置文件所有在$HADOOP_HOME/etc/hadoop下

                            cd /home/hadoop/app/hadoop-2.4.1/etc/hadoop

2.2.1修改hadoop-env.sh

                                     export JAVA_HOME=/home/hadoop/app/jdk1.7.0_65

2.2.2修改core-site.xml

<configuration>

                             <!-- 指定hdfs的nameservice爲ns1 -->

                                               <property>

                                                        <name>fs.defaultFS</name>

                                                        <value>hdfs://ns1/</value>

                                               </property>

                                               <!-- 指定hadoop臨時目錄 -->

                                               <property>

                                                        <name>hadoop.tmp.dir</name>

                                                        <value>/home/hadoop/app/hadoop-2.4.1/tmp</value>

                                               </property>

                                               <!-- 指定zookeeper地址 -->

                                               <property>

                                                        <name>ha.zookeeper.quorum</name>

                                                        <value>s104:2181,s106:2181,s108:2181</value>

                                               </property>

                                     </configuration>

2.2.3修改hdfs-site.xml

                                     <configuration>

                                     <!--指定hdfs的nameservice爲ns1，須要和core-site.xml中的保持一致 -->

                                               <property>

                                                        <name>dfs.nameservices</name>

                                                        <value>ns1</value>

                                               </property>

                                     <!-- ns1下面有兩個NameNode，分別是nn1，nn2 -->

                                               <property>

                                                        <name>dfs.ha.namenodes.ns1</name>

                                                        <value>nn1,nn2</value>

                                               </property>

                                               <!-- nn1的RPC通訊地址 -->

                                               <property>

                                                        <name>dfs.namenode.rpc-address.ns1.nn1</name>

                                                        <value>s104:9000</value>

                                               </property>

                                               <!-- nn1的http通訊地址 -->

                                               <property>

                                                        <name>dfs.namenode.http-address.ns1.nn1</name>

                                                        <value>s104:50070</value>

                                               </property>

                                              <!-- nn2的RPC通訊地址 -->

                                               <property>

                                                        <name>dfs.namenode.rpc-address.ns1.nn2</name>

                                                        <value>s106:9000</value>

                                               </property>

                                               <!-- nn2的http通訊地址 -->

                                               <property>

                                                        <name>dfs.namenode.http-address.ns1.nn2</name>

                                                        <value>s106:50070</value>

                                               </property>

                                               <!-- 指定NameNode的元數據在JournalNode上的存放位置 -->

                                               <property>

                                                        <name>dfs.namenode.shared.edits.dir</name>

                                                        <value>qjournal://s104:8485;s106:8485;s108:8485/ns1</value>

                                               </property>

                                               <!-- 指定JournalNode在本地磁盤存放數據的位置 -->

                                               <property>

                                                        <name>dfs.journalnode.edits.dir</name>

                                                        <value>/home/hadoop/app/hadoop-2.4.1/journaldata</value>

                                               </property>

                                               <!-- 開啓NameNode失敗自動切換 -->

                                               <property>

                                                        <name>dfs.ha.automatic-failover.enabled</name>

                                                        <value>true</value>

                                               </property>

                                               <!-- 配置失敗自動切換實現方式 -->

                                               <property>

<name>dfs.client.failover.proxy.provider.ns1</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<!-- 配置隔離機制方法，多個機制用換行分割，即每一個機制暫用一行-->

                                               <property>

                                                        <name>dfs.ha.fencing.methods</name>

                                                        <value>

                                                                 sshfence

                                                                 shell(/bin/true)

                                                        </value>

                                               </property>

         <!-- 使用sshfence隔離機制時須要ssh免登錄 -->

                                               <property>

                                                        <name>dfs.ha.fencing.ssh.private-key-files</name>

                                                        <value>/home/hadoop/.ssh/id_rsa</value>

                                               </property>

                                               <!-- 配置sshfence隔離機制超時時間 -->

                                               <property>

                                                        <name>dfs.ha.fencing.ssh.connect-timeout</name>

                                                        <value>30000</value>

                                               </property>

                                     </configuration>

                           

2.2.4修改mapred-site.xml

                                     <configuration>

                                               <!-- 指定mr框架爲yarn方式 -->

                                               <property>

                                                        <name>mapreduce.framework.name</name>

                                                        <value>yarn</value>

                                               </property>

                                     </configuration>     

                           

2.2.5修改yarn-site.xml

                                     <configuration>

                                                        <!-- 開啓RM高可用 -->

                                                        <property>

                                                           <name>yarn.resourcemanager.ha.enabled</name>

                                                           <value>true</value>

                                                        </property>

                                                        <!-- 指定RM的cluster id -->

                                                        <property>

                                                           <name>yarn.resourcemanager.cluster-id</name>

                                                           <value>yrc</value>

                                                        </property>

                                                        <!-- 指定RM的名字 -->

                                                        <property>

                                                           <name>yarn.resourcemanager.ha.rm-ids</name>

                                                           <value>rm1,rm2</value>

                                                        </property>

                                                        <!-- 分別指定RM的地址 -->

                                                        <property>

                                                           <name>yarn.resourcemanager.hostname.rm1</name>

                                                           <value>s104</value>

                                                        </property>

                                                        <property>

                                                           <name>yarn.resourcemanager.hostname.rm2</name>

                                                           <value>s106</value>

                                                        </property>

                                                        <!-- 指定zk集羣地址 -->

                                                        <property>

                                                           <name>yarn.resourcemanager.zk-address</name>

                                                           <value>s104:2181,s106:2181,s108:2181</value>

                                                        </property>

                                                        <property>

                                                           <name>yarn.nodemanager.aux-services</name>

                                                           <value>mapreduce_shuffle</value>

                                                        </property>

                                     </configuration>     

                                    

2.2.6修改slaves(slaves是指定子節點的位置，由於要在s104上啓動HDFS、在s104又啓動yarn，因此s104上的slaves文件既指定的是datanode的位置，又指定的是nodemanager的位置)

                                     s108

2.2.7配置免密碼登錄

                   #首先要配置s104到s106,s108的免密碼登錄

                   #在s104上生產一對鑰匙

                      ssh-keygen -t rsa

                   #將公鑰拷貝到其餘節點，包括本身

                      ssh-coyp-id s104

                     ssh-coyp-id s106

                      ssh-coyp-id s108

                   #配置s106到s104,s108的免密碼登錄(兩個namenode之間要配置ssh免密碼登錄，別忘了配置s106到s104的免登錄)

                   #在s106上生產一對鑰匙

                      ssh-keygen -t rsa

                   #將公鑰拷貝到其餘節點

                      ssh-coyp-id s104

                      ssh-coyp-id s106

                      ssh-coyp-id s108                                    

2.4將配置好的hadoop拷貝到其餘節點

                            scp -r /home/hadoop/app/hadoop-2.4.1/ hadoop@s106:/home/hadoop/app/

                            scp -r /home/hadoop/app/hadoop-2.4.1/ hadoop@s108:/home/hadoop/app/

###注意：嚴格按照下面的步驟

2.5啓動zookeeper集羣（分別在s104,s105,s106上啓動zk）

                            cd /weekend/zookeeper-3.4.5/bin/

                            ./zkServer.sh start

                            #查看狀態：一個leader，兩個follower

                            ./zkServer.sh status        

2.6啓動journalnode（分別在在s104,s105,s106上執行）

                            cd /home/hadoop/app/hadoop-2.4.1

                            sbin/hadoop-daemon.sh start journalnode

                            #運行jps命令檢驗，s104,s105,s106上多了JournalNode進程

2.7格式化HDFS

                            #在s104上執行命令:

                            hdfs namenode -format

                          #格式化後會在根據core-site.xml中的hadoop.tmp.dir配置生成個文件，這裏我配置的是/home/hadoop/app/hadoop-2.4.1/tmp，將其拷貝到s106 的/home/hadoop/app/hadoop-2.4.1/下。

                           scp -r tmp/ s106:/home/hadoop/app/hadoop-2.4.1/

                            ##也能夠這樣，建議hdfs namenode –bootstrapStandby

2.8格式化ZKFC(在s104上執行便可)

                            hdfs zkfc –formatZK

2.9啓動HDFS(在s104上執行)

                            sbin/start-dfs.sh     

2.10，我配置的是3個節點的HA配置，因此namenode和resourcemanager放在同一個節點上，我配置的兩個resourcemanager放在s104,s106

          在s104上啓動 sbin/start-yarn.sh

              到此，hadoop-2.4.1配置完畢，能夠統計瀏覽器訪問:

                       http://192.168.43.104:50070

                       NameNode 's104:9000' (active)

                       http://192.168.43.106:50070

                       NameNode 's106:9000' (standby)

        

                  

        

         驗證YARN：

                   運行一下hadoop提供的demo中的WordCount程序：

                   hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar wordcount /user/input /user/output

          OK，大功告成！！！     

測試集羣工做狀態的一些指令：

bin/hdfs dfsadmin -report        查看hdfs的各節點狀態信息

bin/hdfs haadmin -getServiceState nn1           獲取一個namenode節點的HA狀態

sbin/hadoop-daemon.sh start namenode  單獨啓動一個namenode進程

./hadoop-daemon.sh start zkfc   單獨啓動一個zkfc進程

                           

 

HA配置成功，實驗測試過程以下：

HA架構：

                       

 S104節點

 

S106節點

 

 

S108節點

 

開始active節點爲s104

 

 

 

 

 

當s104被咱們kill掉時，active節點自動變爲s106,以下圖

手動啓動那個掛掉的NameNode

                   sbin/hadoop-daemon.sh start namenode

                   經過瀏覽器訪問：http://192.168.43.104:50070，以下圖，該節點變爲standby

         datanode節點一直是s108         

 

驗證yarn, 運行一下hadoop提供的demo中的WordCount程序：

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar wordcount  /profile  /out

        

         OK，大功告成！！！

 

 

 

最後wordcount的結果以下圖所示

 

網頁中生成的結果如圖所示