032 HDFS中高可用性HA的講解

時間 2019-11-19

標籤 hdfs 用性講解欄目 Hadoop 简体版

原文原文鏈接

HDFS Using QJMjava

　　HA使用的是分佈式的日誌管理方式node

一：概述linux

1.背景web

　　若是namenode出現問題，整個HDFS集羣將不能使用。shell

　　是否是能夠有兩個namenode呢apache

　　　　一個爲對外服務->activejson

　　　　一個處於待機->standbyapi

　　　　他們的之間共享的元數據叫 nameservice安全

2.HDFS HA的幾大中重點session

　　1）保證兩個namenode裏面的內存中存儲的文件的元數據同步

　　　　->namenode啓動時，會讀鏡像文件

　　2）變化的記錄信息同步

　　3）日誌文件的安全性

　　　　->分佈式的存儲日誌文件（cloudera公司提出來的）

　　　　　　->2n+1個，使用副本數保證安全性

　　　　->使用zookeeper監控

　　　　　　->監控兩個namenode，當一個出現了問題，能夠達到自動故障轉移。

　　　　　　->若是出現了問題，不會影響整個集羣

　　　　　　->zookeeper對時間同步要求比較高。

　　4）客戶端如何知道訪問哪個namenode

　　　　->使用proxy代理

　　　　->隔離機制

　　　　->使用的是sshfence

　　　　->兩個namenode之間無密碼登陸

　　5）namenode是哪個是active

　　　　->zookeeper經過選舉選出zookeeper。

　　　　->而後zookeeper開始監控，若是出現文件，自動故障轉移。

3.架構

　　注意兩個namenode之間的話。

4.使用QJM

　　JournalNode：日誌節點，進程管理。

二：準備

3.規劃集羣

　　namenode　　　　namenode

　　journalnode　　 journalnode　　　　journalnode -->日誌的分佈，這是日誌節點，考慮的是日誌的安全性。

　　datanode　　　　 datanode　　　　　 datanode

4.關閉全部的進程

5.保存分佈式的源數據，複製一份用來HDFS HA的檢測。

　　先是第一臺，先將分佈式的etc/hadoop，保存爲dist-hadoop，保存源數據。

　　同時，新建tmp。

　　至於第二臺以及第三臺，在分發之間再進行配置。

三：配置文件

6.將core-site.xml中的文件系統刪除，並添加新的文件系統

　　之前的是使用是配置一臺，如今配置多態namenode，使用的方式是nameservices的名稱的方式。

　　添加配置

7.配置hdfs-site.xml

8.繼續配置hdfs-site.xml

　　dfs.nameservices的配置

　　dfs.ha.namenodes.[nameservice ID]的配置

　　包括rpc，http的namenodde地址。

　　dfs.namenode.shared.edits.dir的配置

　　這是journalnode的地址

　　dfs.journalnode.edits.dir 的配置

　　這是journalnode的日誌存儲的目錄

　　先新建目錄：

　　dfs.client.failover.proxy.provider的配置（客戶端）

　　dfs.ha.fencing.methods的配置

　　使用的方式爲ssh攔截

9.配置完成，在分發以前先進行的是目錄的規劃

10.分發

四：啓動

1.啓動三臺zookeeper

　　如今有些奇怪，這三臺zookeeper爲啥要啓動，好像沒用到。

　　看視頻上啓動的，在這裏暫時不啓動了，由於沒有用到，在自動故障轉移的時候纔去使用。

2.啓動三臺的日誌節點

3.格式化第一臺虛擬機

　　由於是共享數據，因此格式化一臺虛擬機便可。

　　由於管理的是同一個元數據，因此另一個不須要格式化。

4.啓動namenode（第一臺）

5.緊接着，同步元數據（在第二臺上寫命令）

　　最好是bin/hdfs namenode -help查看

6.啓動namenode（第兩臺虛擬機）

7.啓動三臺的datanode

8.觀看兩臺的啓動狀態

　　關於這個問題，剛纔在官網上看了一下，的確是啓動後會是兩個standby。

　　官網以下：

9.強制切換狀態

　　1）、查找幫助命令，屬於bin/hdfs haadmin

　　2）、具體命令

10.結果

　　1）、

　　或者：

　　這個查看第一臺的狀態，也能夠在第二臺虛擬機上查看節點一，例如：

　　2）、

五：再次測試

1.在HDFS上新建目錄並上傳文件

2.殺死第一臺的namenode，進行測試

3.將avtive的狀態切換到第二臺

　　這個命令應該在第二臺虛擬機上進行操做。

　　經過實驗發現，若是不進行強制轉換成active，這個命令是不會成功的，因此須要添加上--forceactive。

4.看第一臺是否能夠觀看hdfs中的文件

　　若是能夠，說明，HA發揮做用了。

　　由於這時proxy提供的接口變成nn2.

六：自動故障轉移

　　前提：關閉全部的進程。

　　依賴：zookeeper的監控，組件爲：ZKFC。

　　啓動之後都是standby，選舉一個active。

　　規劃：

　　namenode　　　　namenode

　　ZKFC 　　　ZKFC

　　journalnode　　 journalnode　　　　journalnode

　　datanode　　　　 datanode　　　　　 datanode

1.看看官網的說法

　　這個地方關於zookeeper的使用不是太理解，須要到官網上研究一下。

　　還配置了一個zookeeper集羣，用於ZKFC（DFSZKFailoverCont roller）故障轉移。

　　文章最後作一下說明。對ZKFC的原理。

2.配置core-site.xml

　　添加zookeeper的服務，包括主機名和端口號。

3.配置hdfs-site.xml

　　添加自動故障轉移的使能。

4.分發

5.肯定關閉全部的進程

　　這一步是開始的基礎。

　　官網上的說法：

　　Before you begin configuring automatic failover, you should shut down your cluster. It is not currently possible to transition from a manual failover setup to an automatic failover setup while the cluster is running.

6.開啓三臺的zookeeper服務

　　先開啓監控，而後方便下面進行初始化ZK，產生一個ZNode目錄。

7.初始化HA在zookeeper中的狀態bin/hdfs zkfc -formatZK

　　在zookepper上建立znode節點，用於存儲數據，方便自動實現故障轉移。

　　關於這個ZKFC的初始化須要在哪臺虛擬機上進行，經過思考與實踐，在能夠進行HA的那臺機器上便可，並且作一次就好了。

　　在下面有說明狀況。

8.觀察成功與否

　　進入zookeeper目錄

　　命令：bin/zkCli.sh

9.啓動sbin/start-dfs.sh

　　前兩臺虛擬機會出現DFZKFailoverController。

　　若是沒有開啓DFZKFailoverController，能夠手動開啓，命令是sbin/hadoop-daemon.sh start zkfc。

　　查看官網的說法：

七：簡單檢測

1.展現前兩臺的虛擬機狀態

2.殺死第二臺的虛擬機

3.結果

　　這時，第一臺虛擬機變成active。

八：HA流程

1.說明

　　ZKGZ經過RPC協議獲取namenode的狀態

　　ZKFC鏈接zookeeper，完成選舉。

　　stabdby節點ZKFC監控本機的namenode。

　　ZKFC在監控週期內檢測到失效的namenode。

　　ZKFC記錄失敗的active namenode信息到zookeeper。

　　zookeeper發送一個通知給stabdby的節點ZKFC

　　standby節點ZKFC進入切換的流程，隔離剛纔的主機。

　　ZKFC進行選舉，確保新的active節點。

九：ZKFC原理說明

1.基本原理

　　zk的基本特性：
　　(1) 可靠存儲小量數據且提供強一致性
　　(2) ephemeral node, 在建立它的客戶端關閉後，能夠自動刪除
　　(3) 對於node狀態的變化，能夠提供異步的通知(watcher)

　　zk在zkfc中能夠提供的功能：
　　(1) Failure detector: 及時發現出故障的NN，並通知zkfc
　　(2) Active node locator: 幫助客戶端定位哪一個是Active的NN
　　(3) Mutual exclusion of active state: 保證某一時刻只有一個Active的NN

2.模塊

　　(1) ZKFailoverController(DFSZKFailoverController): 驅動整個ZKFC的運轉，經過向HealthMonitor和ActiveStandbyElector註冊回調函數的方式，subscribe HealthMonitor和ActiveStandbyElector的事件，並作相應的處理
　　(2) HealthMonitor: 按期check NN的健康情況，在NN健康情況發生變化時，經過回調函數把變化通知給ZKFailoverController
　　(3) ActiveStandbyElector: 管理NN在zookeeper上的狀態，zookeeper上對應node的結點發生變化時，經過回調函數把變化通知給ZKFailoverController
　　(4) FailoverController: 提供作graceful failover的相關功能(dfs admin能夠經過命令行工具手工發起failover)

3.架構

4.線程模型

　　ZKFC的線程模型整體上來說比較簡單的，它主要包括三類線程，一是主線程；一是HealthMonitor線程; 一是zookeeper客戶端的線程。它們的主要工做方式是：
　　(1) 主線程在啓動全部的服務後就開始循環等待
　　(2) HealthMonitor是一個單獨的線程，它按期向NN發包，檢查NN的健康情況
　　(3) 當NN的狀態發生變化時，HealthMonitor線程會回調ZKFailoverController註冊進來的回調函數，通知ZKFailoverController NN的狀態發生了變化
　　(4) ZKFailoverController收到通知後，會調用ActiveStandbyElector的API，來管理在zookeeper上的結點的狀態
　　(5) ActiveStandbyElector會調用zookeeper客戶端API監控zookeeper上結點的狀態，發生變化時，回調ZKFailoverController的回調函數，通知ZKFailoverController，作出相應的變化

5.類關係圖

　　ZKFC的主類是org.apache.hadoop.hdfs.tools.DFSZKFailoverController。

6.formatZK

　　建立特定目錄，做爲後續寫節點狀態的父路徑。若是該目錄已經存在，清理原有目錄爲空目錄。

7.繼續說明

HealthMonitor

　　在一個獨立線程中，經過RPC方式，週期性的調用HAServiceProtocol接口的monitorHealth方法，獲取NN的狀態。並把狀態報告給ActiveStandbyElector 。

ActiveStandbyElector

　　ActiveStandbyElector 負責判斷哪一個NN能夠成爲Active。它經過ZK，看哪一個可以成功的建立一個特定的ephemeral lock file (znode)，哪一個就是Active，其它的成爲Standby。在一個節點被通知變成Active後，它必須確保本身可以提供一致性的服務（數據一致性），不然它須要主動退出選舉。

　　若是一個Active因HealthMonitor監控到狀態異常，這裏會做出判斷，先經過Fenceing功能關閉它（確保關閉或者不能提供服務），而後在ZK上刪除它對應ZNode。

　　發送上述事件後，在另一臺機器上的ZKFC中的ActiveStandbyElector 會收到事件，並從新進行選舉（嘗試建立特定ZNode），它將得到成功並更改NN中狀態，從而實現Active節點的變動。

十：formatZK日誌說明

1.日誌

18/01/28 19:48:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/01/28 19:48:30 INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at linux-hadoop01.ibeifeng.com/192.168.140.121:8020
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:host.name=linux-hadoop01.ibeifeng.com
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_67
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.home=/opt/modules/jdk1.7.0_67/jre
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/opt/fullyDistributions/ha/hadoop-2.5.0/etc/hadoop:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/httpcore-4.2.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/activation-1.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/zookeeper-3.4.6.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-net-3.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-io-2.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/xz-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/hadoop-annotations-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/guava-11.0.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/httpclient-4.2.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/hadoop-auth-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jettison-1.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jsch-0.1.42.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-collections-3.2.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/junit-4.11.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jsr305-1.3.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/paranamer-2.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-el-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/asm-3.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/avro-1.7.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/hadoop-common-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/hadoop-common-2.5.0-tests.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/common/hadoop-nfs-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-el-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/hadoop-hdfs-2.5.0-tests.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/hdfs/hadoop-hdfs-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/activation-1.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/xz-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/guava-11.0.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/guice-3.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/asm-3.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jline-0.9.94.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-client-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-common-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-api-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-server-common-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/xz-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/guice-3.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/junit-4.11.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/javax.inject-1.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/asm-3.2.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0-tests.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.5.0.jar:/opt/fullyDistributions/ha/hadoop-2.5.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.5.0.jar:/contrib/capacity-scheduler/*.jar
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/opt/fullyDistributions/ha/hadoop-2.5.0/lib/native
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-358.el6.x86_64
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:user.name=beifeng
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/beifeng
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Client environment:user.dir=/opt/fullyDistributions/ha/hadoop-2.5.0
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=linux-hadoop01.ibeifeng.com:2181,linux-hadoop02.ibeifeng.com:2181,linux-hadoop03.ibeifeng.com:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@32dd803
18/01/28 19:48:31 INFO zookeeper.ClientCnxn: Opening socket connection to server linux-hadoop02.ibeifeng.com/192.168.140.122:2181. Will not attempt to authenticate using SASL (unknown error)
18/01/28 19:48:31 INFO zookeeper.ClientCnxn: Socket connection established to linux-hadoop02.ibeifeng.com/192.168.140.122:2181, initiating session
18/01/28 19:48:31 INFO zookeeper.ClientCnxn: Session establishment complete on server linux-hadoop02.ibeifeng.com/192.168.140.122:2181, sessionid = 0x2613c8b97f50000, negotiated timeout = 5000
18/01/28 19:48:31 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/ns1 in ZK.
18/01/28 19:48:31 INFO zookeeper.ZooKeeper: Session: 0x2613c8b97f50000 closed
18/01/28 19:48:31 WARN ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x2613c8b97f50000
18/01/28 19:48:31 INFO zookeeper.ClientCnxn: EventThread shut down