hadoop環境搭建：高可用

時間  2021-02-22
標籤 html node git github web shell apache bootstrap 網絡 app 欄目 Hadoop 简体版
原文   原文鏈接

目錄

一、硬件配置

採用3臺虛擬機html

節點名稱 IP地址 內存 硬盤 節點角色
node1 192.168.1.6 2GB 10GB NameNode、ResoucerManager、ZKFC、DataNode、NodeManager、JournalNode、QuorumPeerMain
node2 192.168.1.7 2GB 10GB NameNode、ResoucerManager、ZKFC、DataNode、NodeManager、JournalNode、QuorumPeerMain
node3 192.168.1.8 2GB 10GB DataNode、NodeManager、JournalNode、QuorumPeerMain

二、軟件版本

軟件 版本
JDK jdk-8u271
HADOOP hadoop-3.2.1
ZOOKEEPER zookeeper-3.5.8

三、準備工做

3.一、配置網絡環境

參考 Hadoop徹底分佈式node

3.二、安裝JDK

參考 Hadoop徹底分佈式git

3.三、安裝ZOOKEEPER

參考 ZOOKEEPER徹底分佈式github

[root@node1 opt]# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.5.8/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

[root@node1 opt]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.5.8/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower

[root@node2 opt]# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.5.8/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

[root@node2 opt]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.5.8/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader

[root@node3 opt]# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.5.8/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

[root@node3 opt]# jps
7602 Jps
7562 QuorumPeerMain

[root@node3 opt]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.5.8/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower

四、安裝Hadoop

配置配置文件，具體見文末。web

五、啓動

(1)分別在 node一、node二、node3 節點上啓動 JournalNode 進程：shell

[root@node1 hadoop-3.2.1]# bin/hdfs --daemon start journalnode
[root@node1 hadoop-3.2.1]# jps
7890 JournalNode
7629 QuorumPeerMain
7934 Jps

[root@node2 hadoop-3.2.1]# bin/hdfs --daemon start journalnode
[root@node2 hadoop-3.2.1]# jps
7890 JournalNode
7629 QuorumPeerMain
7934 Jps

[root@node3 hadoop-3.2.1]# bin/hdfs --daemon start journalnode
[root@node3 hadoop-3.2.1]# jps
7890 JournalNode
7629 QuorumPeerMain
7934 Jps

(2)在其中一臺 NameNode 上運行格式化命令：apache

[root@node1 hadoop-3.2.1]# bin/hdfs namenode -format
....
2020-12-16 21:04:43,291 INFO common.Storage: Storage directory /opt/hadoop-3.2.1/dfs/namenode has been successfully formatted.
....

(3)沒有格式化的 NameNode 經過在此機器上運行以下命令，將已格式化的 NameNode 上的元數據目錄複製過來：bootstrap

[root@node2 hadoop-3.2.1]# bin/hdfs namenode -bootstrapStandby
....
=====================================================
About to bootstrap Standby ID node2 from:
           Nameservice ID: hadoopcluster
        Other Namenode ID: node1
  Other NN's HTTP address: http://node1:9870
  Other NN's IPC  address: node1/192.168.1.6:8020
             Namespace ID: 1789437097
            Block pool ID: BP-1119377207-192.168.1.6-1608123883253
               Cluster ID: CID-72958c07-b2b2-46f8-a035-667159c67de9
           Layout version: -65
       isUpgradeFinalized: true
=====================================================
2020-12-16 21:11:41,726 INFO common.Storage: Storage directory /opt/hadoop-3.2.1/dfs/namenode has been successfully formatted.
2020-12-16 21:11:41,753 INFO common.Util: Assuming 'file' scheme for path /opt/hadoop-3.2.1/dfs/namenode in configuration.
2020-12-16 21:11:41,753 INFO common.Util: Assuming 'file' scheme for path /opt/hadoop-3.2.1/dfs/namenode in configuration.
2020-12-16 21:11:41,805 INFO namenode.FSEditLog: Edit logging is async:true
2020-12-16 21:11:41,908 INFO namenode.TransferFsImage: Opening connection to http://node1:9870/imagetransfer?getimage=1&txid=0&storageInfo=-65:1789437097:1608123883253:CID-72958c07-b2b2-46f8-a035-667159c67de9&bootstrapstandby=true
2020-12-16 21:11:42,009 INFO common.Util: Combined time for file download and fsync to all disks took 0.00s. The file download took 0.00s at 0.00 KB/s. Synchronous (fsync) write to disk of /opt/hadoop-3.2.1/dfs/namenode/current/fsimage.ckpt_0000000000000000000 took 0.00s.
2020-12-16 21:11:42,010 INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000000 size 399 bytes.
2020-12-16 21:11:42,056 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at node2/192.168.1.7
************************************************************/

(4)在兩個 NameNode 節點上啓動 NameNode 進程：網絡

[root@node1 hadoop-3.2.1]# bin/hdfs --daemon start namenode
[root@node1 hadoop-3.2.1]# jps
8705 Jps
7666 QuorumPeerMain
8437 JournalNode
8605 NameNode

[root@node2 hadoop-3.2.1]# bin/hdfs --daemon start namenode
[root@node2 hadoop-3.2.1]# jps
8112 Jps
7890 JournalNode
7629 QuorumPeerMain
8045 NameNode

(5)能夠在任意 NameNode 所在的主機上運行以下命令來格式化 ZKFC，並在將要運行 NameNode 的機器上啓動 ZKFC：app

[root@node1 hadoop-3.2.1]# bin/hdfs zkfc -formatZK
····
2020-12-16 21:18:06,960 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/hadoopcluster in ZK.
····
[root@node1 hadoop-3.2.1]# bin/hdfs --daemon start zkfc
[root@node1 hadoop-3.2.1]# jps
8848 DFSZKFailoverController
7666 QuorumPeerMain
8898 Jps
8437 JournalNode
8605 NameNode

[root@node2 hadoop-3.2.1]# bin/hdfs --daemon start zkfc
[root@node2 hadoop-3.2.1]# jps
7890 JournalNode
8230 Jps
8199 DFSZKFailoverController
7629 QuorumPeerMain
8045 NameNode

(6)啓動 datanode

[root@node1 hadoop-3.2.1]# bin/hdfs --daemon start datanode
[root@node1 hadoop-3.2.1]# jps
8848 DFSZKFailoverController
7666 QuorumPeerMain
9010 Jps
8437 JournalNode
8951 DataNode
8605 NameNode

[root@node2 hadoop-3.2.1]# bin/hdfs --daemon start datanode
[root@node2 hadoop-3.2.1]# jps
7890 JournalNode
8199 DFSZKFailoverController
7629 QuorumPeerMain
8045 NameNode
8398 Jps
8335 DataNode

[root@node3 hadoop-3.2.1]# bin/hdfs --daemon start datanode
[root@node3 hadoop-3.2.1]# jps
7968 DataNode
8000 Jps
7841 JournalNode
7562 QuorumPeerMain

(7)啓動 ResourceManager 與 NodeManager

[root@node1 hadoop-3.2.1]# sbin/start-yarn.sh start resoucermanager
Starting resourcemanagers on [ node1 node2]
上一次登陸：三 12月 16 19:08:13 CST 2020從 desktop-0ahq4ftpts/2 上
Starting nodemanagers
上一次登陸：三 12月 16 21:25:35 CST 2020pts/1 上
[root@node1 hadoop-3.2.1]# jps
8848 DFSZKFailoverController
9472 NodeManager
7666 QuorumPeerMain
8437 JournalNode
8951 DataNode
9627 Jps
9340 ResourceManager
8605 NameNode

[root@node2 hadoop-3.2.1]# jps
7890 JournalNode
8199 DFSZKFailoverController
8552 NodeManager
8475 ResourceManager
7629 QuorumPeerMain
8045 NameNode
8861 Jps
8335 DataNode

[root@node3 hadoop-3.2.1]# jps
7968 DataNode
7841 JournalNode
7562 QuorumPeerMain
8204 Jps
8093 NodeManager

(8) web 查看

ha01

ha02

ha03

六、問題

(1)配置完配置文件，執行 bin/hdfs namenode -format，出現 a shared edits dir must not be specified if HA is not enabled. 問題。

緣由：配置文件中的這種屬性 dfs.ha.namenodes.[nameservice ID] 的 [nameservice ID] 沒有改爲本身的名字。

注意：要配置的屬性較多，仔細檢查。

(2)配置完配置文件，執行 bin/hdfs namenode -format，出現 Unable to check if JNs are ready for formatting 問題。

要在各個結點執行 bin/hdfs --daemon start journalnode 先啓動 journalnode。

七、配置文件

core-site.xml

<configuration>
	<property>
            <name>fs.defaultFS</name>  
            <value>hdfs://hadoopcluster</value>  
      	</property>
      	<property>
             <name>io.file.buffer.size</name>
             <value>131072</value>	 
      	</property>
	<property>
            <name>hadoop.tmp.dir</name>
            <value>/opt/hadoop/tmp</value>
        </property> 
        <property>
            <name>ha.zookeeper.quorum</name>
            <value>node1:2181,node2:2181,node3:2181</value>
        </property>
</configuration>

hdfs-site.xml

<configuration>
    <property>
		<name>dfs.nameservices</name>
		<value>hadoopcluster</value>
    </property>
    <property>
        <name>dfs.ha.namenodes.hadoopcluster</name>
        <value>node1,node2</value>
    </property>
    <property>
		<name>dfs.namenode.rpc-address.hadoopcluster.node1</name>
		<value>node1:8020</value>
    </property>
    <property>
		<name>dfs.namenode.rpc-address.hadoopcluster.node2</name>
		<value>node2:8020</value>
    </property>
    <property>
		<name>dfs.namenode.http-address.hadoopcluster.node1</name>
		<value>node1:9870</value>
    </property>
    <property>
		<name>dfs.namenode.http-address.hadoopcluster.node2</name>
		<value>node2:9870</value>
    </property>
    <property>
        <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://node1:8485;node2:8485;node3:8485/hadoopcluster</value>
    </property>
    <property>
        <name>dfs.client.failover.proxy.provider.hadoopcluster</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>
    <property>
        <name>dfs.ha.fencing.methods</name>
        <value>sshfence</value>
        <value>shell(/bin/true)</value>
    </property>
    <property>
        <name>dfs.ha.fencing.ssh.private-key-files</name>
        <value>/root/.ssh/id_dsa</value>
    </property>
    <property>
        <name>dfs.journalnode.edits.dir</name>
        <value>/opt/hadoop-3.2.1/jndata</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>  
        <value>/opt/hadoop-3.2.1/dfs/namenode</value>  
    </property>
    <property>
        <name>dfs.hosts</name>  
        <value>/opt/hadoop-3.2.1/etc/hadoop/workers</value>  
    </property> 
    <property>
        <name>dfs.ha.automatic-failover.enabled</name>
        <value>true</value>
    </property> 
</configuration>

mapred-site.xml

<configuration>
    <property>
        <!-- 指定mapreduce框架爲yarn方式 -->
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <!-- Directory where history files are written by MapReduce jobs -->
        <name>mapreduce.jobhistory.intermediate-done-dir</name>
        <value>/mr-history/tmp</value>
    </property>
    <property>
        <!-- Directory where history files are managed by the MR JobHistory Server -->
        <name>mapreduce.jobhistory.done-dir</name>
        <value>/mr-history/done</value>
    </property> 
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>node1:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>node1:19888</value>
    </property>
</configuration>

yarn-site.xml

<configuration>
    <property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.resourcemanager.cluster-id</name>
        <value>yarncluster</value>
    </property>
    <property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm1,rm2</value>
    </property>
    <property>
        <name>yarn.log-aggregation-enable</name>  
        <value>true</value>  
    </property>
    <property>
        <name>yarn.resourcemanager.hostname.rm1</name>
        <value>node1</value>
        </property>
    <property>
        <name>yarn.resourcemanager.hostname.rm2</name>
        <value>node2</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address.rm1</name>
        <value>node1:8088</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address.rm2</name>
        <value>node2:8088</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address.rm1</name>  
        <value>node1:8033</value>  
    </property> 
    <property>
        <name>yarn.resourcemanager.admin.address.rm2</name>  
        <value>node2:8033</value>  
    </property> 
    <property>
        <name>yarn.resourcemanager.address.rm1</name>  
        <value>node1:8032</value>  
    </property> 
    <property>
        <name>yarn.resourcemanager.address.rm2</name>  
        <value>node2:8032</value>  
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address.rm1</name>  
        <value>node1:8030</value>  
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address.rm2</name>  
        <value>node2:8030</value>  
    </property> 
    <property>
        <name>yarn.resourcemanager.resource-tracker.address.rm1</name>  
        <value>node1:8031</value>  
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address.rm2</name>  
        <value>node2:8031</value>  
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address.rm1</name>  
        <value>node1:8088</value>  
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address.rm2</name>  
        <value>node2:8088</value>  
    </property>
    <property>
        <name>hadoop.zk.address</name>
        <value>node1:2181,node2:2181,node3:2181</value>
    </property>
    <property>
        <name>yarn.resourcemanager.nodes.include-path</name>  
        <value>/opt/hadoop-3.2.1/etc/hadoop/workers</value>  
    </property>
        <property>
        <name>yarn.nodemanager.local-dirs</name>  
        <value>/opt/hadoop-3.2.1/tmp</value>  
    </property> 
    <property>
        <name>yarn.nodemanager.log-dirs</name>  
        <value>/opt/hadoop-3.2.1/logs</value>  
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>  
        <value>mapreduce_shuffle</value> 
    </property>
    <property>
        <name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.application.classpath</name>
        <value>/opt/hadoop-3.2.1/etc/hadoop:/opt/hadoop-3.2.1/share/hadoop/common/lib/*:/opt/hadoop-3.2.1/share/hadoop/common/*:/opt/hadoop-3.2.1/share/hadoop/hdfs:/opt/hadoop-3.2.1/share/hadoop/hdfs/lib/*:/opt/hadoop-3.2.1/share/hadoop/hdfs/*:/opt/hadoop-3.2.1/share/hadoop/mapreduce/lib/*:/opt/hadoop-3.2.1/share/hadoop/mapreduce/*:/opt/hadoop-3.2.1/share/hadoop/yarn:/opt/hadoop-3.2.1/share/hadoop/yarn/lib/*:/opt/hadoop-3.2.1/share/hadoop/yarn/*</value>
    </property>
</configuration>

worker

node1
node2
node3
相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程