configure HDFS(hadoop 分佈式文件系統) high available

注：來自尚學堂小陳老師上課筆記

1.安裝啓動zookeeper
a)上傳解壓zookeeper包
b)cp zoo_sample.cfg zoo.cfg修改zoo.cfg文件
c)dataDir=/opt/data/zookeeper
server.1=node1:2888:3888
server.2=node2:2888:3888
server.3=node3:2888:3888
這裏的node1是本身主機名，能夠寫ip
d)分別在node1 node2 node3 的數據目錄/opt/data/zookeeper下面建立myid文件，裏面寫對應server.後面的數字
e)配置環境變量並source生效
export ZK_HOME=/opt/soft/zookeeper-3.4.6
export PATH=$JAVA_HOME/bin:$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZK_HOME/bin
f)啓動 zkServer.sh start啓動，隔一分鐘，經過zkServer.sh status查看狀態

2.配置hadoop配置文件
配置hdfs-site.xml
<property>
<name>dfs.nameservices</name>
<value>sxt</value>
</property>
<property>
<name>dfs.ha.namenodes.sxt</name>
<value>nn1,nn2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.sxt.nn1</name>
<value>node1:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.sxt.nn2</name>
<value>node2:8020</value>
</property>
<property>
<name>dfs.namenode.http-address.sxt.nn1</name>
<value>node1:50070</value>
</property>
<property>
<name>dfs.namenode.http-address.sxt.nn2</name>
<value>node2:50070</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://node1:8485;node2:8485;node3:8485/sxt</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.sxt</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
配置core-site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://sxt</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/opt/data/journal</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>node1:2181,node2:2181,node3:2181</value>
</property>

3.啓動全部journalnode
hadoop-daemon.sh start journalnode
4.其中一個namenode節點執行格式化
hdfs namenode -format
5.另一個namenode節點格式化拷貝
首先要將剛纔格式化以後的namenode啓動起來
hadoop-daemon.sh start namenode
hdfs namenode -bootstrapStandby
6.上傳配置到zookeeper集羣
hdfs zkfc -formatZK
7.啓動
先stop-dfs.sh
而後start-dfs.sh
hadoop-daemon.sh start namenode