hadoop徹底分佈式搭建（非高可用）

1、準備工做

1.新建虛擬機固定ip NAT、固定主機名html

新建虛擬機： VMware安裝Linux詳細教程

## 固定主機名：
vi /etc/sysconfig/network

2.關閉防火牆 or 暴露端口java

service iptables stop  關閉防火牆
chkconfig iptables off 禁止開啓啓動

3.必要軟件 JDK、Hadoop 4.而配置ssh無密碼登陸 [每臺節點都須要生成]node

（1）生成公鑰和私鑰
    ssh-keygen -t rsa
（2）配置hosts文件（/etc/hosts）Ip與hostname的對照關係：
    192.168.121.101 node01
    192.168.121.102 node02
    192.168.121.103 node03
    192.168.121.103 node04
    ...
    對於以上的文件，在node01上修改完畢以後，使用scp 命令 遠程拷貝給node02 node03
（3）導入公鑰到認證文件
    ssh-copy-id -i /root/.ssh/id_rsa.pub node01
    ssh-copy-id -i /root/.ssh/id_rsa.pub node02
    ssh-copy-id -i /root/.ssh/id_rsa.pub node03
    ssh-copy-id -i /root/.ssh/id_rsa.pub node04
    ...

5.配置NTP，使得集羣間的時間同步(非必須)linux

安裝 NTP：

yum install ntp

修改 /etc/ntp.conf文件：

## 註釋掉 server開頭的行，並添加
restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap
    server 127.127.1.0
    fudge 127.127.1.0 stratum 10

在 node0二、0三、04添加以下內容：

## 註釋掉 server開頭的行，並添加
server node01

永久啓動NTP服務：

service ntpd start&chkconfig ntpd on

6.【建議】刪除hadoop的doc文檔，400多M佔據空間~web

2、集羣規劃

node01	node02	node03	node04
NameNode
	DataNode	DataNode	DataNode
	Resoucemanager
	SecondaryNameNode
NodeManager	nodemanager	nodemanager	nodemanager

3、配置的文件

須要配置的文件有7個：app

$HADOOP_HOME/ etc/hadoop/hadopp-env.sh
$HADOOP_HOME/ etc/hadoop/yarn-env.sh
$HADOOP_HOME/ etc/hadoop/slaves [hadoop2.x version ] or etc/hadoop/workers [hadoop3.x version]
$HADOOP_HOME/ etc/hadoop/core-site.xml
$HADOOP_HOME/ etc/hadoop/hdfs-site.xml
$HADOOP_HOME/ etc/hadoop/yarn-site.xml
$HADOOP_HOME/ etc/hadoop/mapred-site.xml

1. 配置 etc/hadoop/hadopp-env.sh

hadoop只會讀這個文件配置的jdk。在hadoop2.x只須要配置jdk，而在hadoop3.x須要配置角色；hadoop3.x對角色有了嚴格的管理，必須在配置。框架

export JAVA_HOME=/opt/app/jdk1.8.0_201
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root

2. 配置 etc/hadoop/yarn-env.sh

export JAVA_HOME=/opt/app/jdk1.8.0_201

3. 配置 etc/hadoop/slaves|etc/hadoop/workers

node02
node03
node04

4. 配置 etc/hadoop/core-site.xml

<configuration>
        <!--說明：hadoop2.x端口默認9000；hadoop3.x端口默認9820-->
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://node01:9820</value>
        </property>
        <!--注意：臨時目錄本身建立下-->
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/opt/tmp/hadoop/full</value>
        </property>
</configuration>

5. 配置 etc/hadoop/hdfs-site.xml

<configuration>
        <!--說明：不配置副本的狀況下默認是3 -->
        <property>
                <name>dfs.replication</name>
                <value>2</value>
        </property>
        <property>
            <!--設置 secondaryNameNode 爲 node02節點的虛擬機； hadoop2.x 端口爲50090-->
            <name>dfs.namenode.secondary.http-address</name>
            <value>node02:9868</value>
        </property>
        <!--關閉 hdfs 讀取權限，即不檢查權限-->
        <property>
            <name>dfs.permissions.enabled</name>
            <value>false</value>
        </property>
</configuration>

6. 配置 etc/hadoop/yarn-site.xml

<configuration>
   <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
   </property>
   <!--指定 resourcemanager 在 node02這臺節點上啓動-->
   <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>node02</value>
   </property>
   
</configuration>

7. 配置 etc/hadoop/mapred-site.xml

<configuration>
    <!--配置 mapreduce的運行的框架名稱爲 yarn （MR 配置爲在 yarn上運行）-->
	<property>
    	<name>mapreduce.framework.name</name>
    	<value>yarn</value>
	</property>

</configuration>

8. 將hadoop分發到其餘節點

scp -r /opt/app/hadoop-3.2.0 node02:/opt/app/hadoop-3.2.0

4、啓動集羣

1 格式化namenode

bin/hdfs namenode -format

2 啓動NameNode、SecondaryNameNode與DataNode

## 在 node01啓動 namenode
sbin/hadoop-daemon.sh start namenode
## 在 node02啓動 secondarynamenode
sbin/haddop-daemon.sh star sencdarynamenode
## 在其餘 node02 、0三、04 啓動 datanode
sbin/hadoop-daemon.sh start datanode

3啓動YARN ,ResouceManager 以及NodeManager

## 在 node02節點啓動 resourcemanager、nodemanager
sbin/yarn-daemon.sh start resourcemanager 
sbin/yarn-daemon.sh start nodemanager
## 在 其餘 node0一、0三、04 節點啓動 nodemanager
sbin/yarn-daemon.sh start nodemanager

說明：能夠配置環境變量就不用到hadoop去執行命令了

[root@node01 hadoop-3.2.0]# vi /etc/profile
## JDK環境變量
export JAVA_HOME=/opt/app/jdk1.8.0_201
## hadoop環境變量
export HADOOP_HOME=/opt/app/hadoop-3.2.0
## hadoop日誌輸出級別設置爲debug
#export HADOOP_ROOT_LOGGER=DEBUG,console
## 依賴的包這兩個
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_COMMON_LIB_NATIVE_DIR"
## path
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

一鍵啓動/關閉 start-all.sh / stop-all.sh

[root@node01 logs]# start-all.sh
Starting namenodes on [node01]
Starting datanodes
Starting secondary namenodes [node02]
2019-06-15 01:15:29,452 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your 
platform... using builtin-java classes where applicableStarting resourcemanager
Starting nodemanagers
[root@node01 logs]# stop-all.sh
Stopping namenodes on [node01]
Stopping datanodes
Stopping secondary namenodes [node02]
2019-06-15 01:21:10,936 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your 
platform... using builtin-java classes where applicableStopping nodemanagers
node03: WARNING: nodemanager did not stop gracefully after 5 seconds: Trying to kill with kill -9
node02: WARNING: nodemanager did not stop gracefully after 5 seconds: Trying to kill with kill -9
node04: WARNING: nodemanager did not stop gracefully after 5 seconds: Trying to kill with kill -9
Stopping resourcemanager
[root@node01 logs]#

問題&解決方案：ssh

[root@node01 ~]# cd /opt/app/hadoop-3.2.0/lib/native
[root@node01 native]# ls
examples     libhadooppipes.a  libhadoop.so.1.0.0  libnativetask.a   libnativetask.so.1.0.0
libhadoop.a  libhadoop.so      libhadooputils.a    libnativetask.so
[root@node01 native]# ldd libhadoop.so.1.0.0
./libhadoop.so.1.0.0: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ./libhadoop.so
.1.0.0)	linux-vdso.so.1 =>  (0x00007fff9bd8a000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f7f51dd7000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f7f51bb9000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f7f51825000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f7f52208000)
[root@node01 native]# ldd --version
ldd (GNU libc) 2.12
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
[root@node01 native]#

5、驗證

能夠經過jps命令查看啓動進程；以及經過 ss -nal命令監控端口進行查看oop

[root@node01 hadoop-3.2.0]# jps
1426 NodeManager
1304 NameNode
1550 Jps
[root@node01 hadoop-3.2.0]# ss -nal
State      Recv-Q Send-Q                 Local Address:Port                   Peer Address:Port 
LISTEN     0      128                                *:9870                              *:*     
LISTEN     0      128                                *:59635                             *:*     
LISTEN     0      128                               :::22                               :::*     
LISTEN     0      128                                *:22                                *:*     
LISTEN     0      100                              ::1:25                               :::*     
LISTEN     0      100                        127.0.0.1:25                                *:*     
LISTEN     0      128                                *:13562                             *:*     
LISTEN     0      128                  192.168.121.101:9820                              *:*     
LISTEN     0      128                                *:8040                              *:*     
LISTEN     0      128                                *:8042                              *:*     
[root@node01 hadoop-3.2.0]#

web儀表盤查看：ui

http://192.168.121.101:9870/dfshealth.html#tab-overview