Hadoop部署方式-徹底分佈式(Fully-Distributed Mode)
Hadoop部署方式-徹底分佈式(Fully-Distributed Mode)
html
做者：尹正傑java
版權聲明：原創做品，謝絕轉載！不然將追究法律責任。node
本博客搭建的虛擬機是僞分佈式環境（http://www.javashuo.com/article/p-ugidapgo-cy.html）連接克隆出來的，咱們只須要修改一下配置文件就能夠輕鬆實現徹底分佈式部署了，部署架構是一個NameNode和三個DataNode，若是身爲一個專業的運維人員你可能會一眼看出來這個集羣存在單點故障，彆着急，關於高可用集羣部署請參考：http://www.javashuo.com/article/p-nhjqpuhg-ds.html。linux
若是你是mac用戶推薦使用"Parallets "，若是你是windows系統推薦使用「VMware Workstation」，若是是Linux用戶的小夥伴推薦使用「VirtualBox」。個人實驗環境在windows上操做的，安裝的是VMware Workstation。git
一.實驗環境準備
須要準備四臺Linux操做系統的服務器，配置參數最好同樣，因爲個人虛擬機是以前僞分佈式部署而來的，所以個人環境都一致，而且天天虛擬機默認都是Hadoop僞分佈式喲！
1>.NameNode服務器（172.16.30.101）web
2>.DataNode服務器（172.16.30.102）shell
3>.DataNode服務器（172.16.30.103）apache
4>.DataNode服務器（172.16.30.104）json
二.修改Hadoop的配置文件windows
修改的配置文件路徑是我以前拷貝的full目錄，絕對路徑是：「/soft/hadoop/etc/full」,修改這個目錄下的文件以後，咱們將hadoop目錄鏈接過來便可，當你須要僞分佈式或者本地模式的時候只須要改變軟鏈接指向的目錄便可，這樣就輕鬆實現了三種模式配置文件和平相處的局面。
1>.core-site.xml 配置文件
[yinzhengjie@s101 ~]$ more /soft/hadoop/etc/hadoop/core-site.xml <?xml version="1.0" encoding="UTF-8"?> <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://s101:8020</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/home/yinzhengjie/hadoop</value> </property> </configuration> <!-- core-site.xml配置文件的做用： 用於定義系統級別的參數，如HDFS URL、Hadoop的臨時 目錄以及用於rack-aware集羣中的配置文件的配置等，此中的參 數定義會覆蓋core-default.xml文件中的默認配置。 fs.defaultFS 參數的做用： #聲明namenode的地址，至關於聲明hdfs文件系統。 hadoop.tmp.dir 參數的做用： #聲明hadoop工做目錄的地址。 --> [yinzhengjie@s101 ~]$
2>.hdfs-site.xml 配置文件
[yinzhengjie@s101 ~]$ more /soft/hadoop/etc/hadoop/hdfs-site.xml <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>dfs.replication</name> <value>3</value> </property> </configuration> <!-- hdfs-site.xml 配置文件的做用： #HDFS的相關設定，如文件副本的個數、塊大小及是否使用強制權限 等，此中的參數定義會覆蓋hdfs-default.xml文件中的默認配置. dfs.replication 參數的做用： #爲了數據可用性及冗餘的目的，HDFS會在多個節點上保存同一個數據 塊的多個副本，其默認爲3個。而只有一個節點的僞分佈式環境中其僅用 保存一個副本便可，這能夠經過dfs.replication屬性進行定義。它是一個 軟件級備份。 --> [yinzhengjie@s101 ~]$
3>.mapred-site.xml 配置文件
[yinzhengjie@s101 ~]$ more /soft/hadoop/etc/hadoop/mapred-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration> <!-- mapred-site.xml 配置文件的做用： #HDFS的相關設定，如reduce任務的默認個數、任務所可以使用內存 的默認上下限等，此中的參數定義會覆蓋mapred-default.xml文件中的 默認配置. mapreduce.framework.name 參數的做用： #指定MapReduce的計算框架，有三種可選，第一種：local(本地),第 二種是classic(hadoop一代執行框架),第三種是yarn(二代執行框架)，我 們這裏配置用目前版本最新的計算框架yarn便可。 --> [yinzhengjie@s101 ~]$
4>.yarn-site.xml配置文件
[yinzhengjie@s101 ~]$ more /soft/hadoop/etc/hadoop/yarn-site.xml <?xml version="1.0"?> <configuration> <property> <name>yarn.resourcemanager.hostname</name> <value>s101</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration> <!-- yarn-site.xml配置文件的做用： #主要用於配置調度器級別的參數. yarn.resourcemanager.hostname 參數的做用： #指定資源管理器(resourcemanager)的主機名 yarn.nodemanager.aux-services 參數的做用： #指定nodemanager使用shuffle --> [yinzhengjie@s101 ~]$
5>.slaves配置文件
[yinzhengjie@s101 ~]$ more /soft/hadoop/etc/hadoop/slaves #該配置文件的做用：是NameNode用與記錄須要鏈接哪些DataNode服務器節點，用與啓動或中止服務時發送遠程命令指令的目標主機。 s102 s103 s104 [yinzhengjie@s101 ~]$
三.在NameNode節點上配置免密碼登陸各DataNode節點
1>.在本地上生成公私祕鑰對(生成以前，把上次部署僞分佈式的祕鑰刪除掉)
[yinzhengjie@s101 ~]$ rm -rf ~/.ssh/* [yinzhengjie@s101 ~]$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa Generating public/private rsa key pair. Your identification has been saved in /home/yinzhengjie/.ssh/id_rsa. Your public key has been saved in /home/yinzhengjie/.ssh/id_rsa.pub. The key fingerprint is: a3:a4:ae:d8:f7:7f:a2:b6:d6:15:74:29:de:fb:14:08 yinzhengjie@s101 The key's randomart image is: +--[ RSA 2048]----+ | . | | E o | | o = . | | o o . | | . S . . . | | o . .. . . | | . .. . o | | o .. o o . . | |. oo.+++.o | +-----------------+ [yinzhengjie@s101 ~]$
2>.使用ssh-copy-id命令分配公鑰到DataNode服務器（172.16.30.101）
[yinzhengjie@s101 ~]$ ssh-copy-id yinzhengjie@s101 The authenticity of host 's101 (172.16.30.101)' can't be established. ECDSA key fingerprint is fa:25:bc:03:7e:99:eb:12:1e:bc:a8:c9:ce:39:ba:7b. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys yinzhengjie@s101's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'yinzhengjie@s101'" and check to make sure that only the key(s) you wanted were added. [yinzhengjie@s101 ~]$ ssh s101 Last login: Fri May 25 18:35:40 2018 from 172.16.30.1 [yinzhengjie@s101 ~]$ who yinzhengjie pts/0 2018-05-25 18:35 (172.16.30.1) yinzhengjie pts/1 2018-05-25 19:17 (s101) [yinzhengjie@s101 ~]$ exit logout Connection to s101 closed. [yinzhengjie@s101 ~]$ who yinzhengjie pts/0 2018-05-25 18:35 (172.16.30.1) [yinzhengjie@s101 ~]$
3>.使用ssh-copy-id命令分配公鑰到DataNode服務器（172.16.30.102）
[yinzhengjie@s101 ~]$ ssh-copy-id yinzhengjie@s102 /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys yinzhengjie@s102's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'yinzhengjie@s102'" and check to make sure that only the key(s) you wanted were added. [yinzhengjie@s101 ~]$ ssh s102 Last login: Fri May 25 18:35:42 2018 from 172.16.30.1 [yinzhengjie@s102 ~]$ who yinzhengjie pts/0 2018-05-25 18:35 (172.16.30.1) yinzhengjie pts/1 2018-05-25 19:19 (s101) [yinzhengjie@s102 ~]$ exit logout Connection to s102 closed. [yinzhengjie@s101 ~]$ who yinzhengjie pts/0 2018-05-25 18:35 (172.16.30.1) [yinzhengjie@s101 ~]$
4>.使用ssh-copy-id命令分配公鑰到DataNode服務器（172.16.30.103）
[yinzhengjie@s101 ~]$ ssh-copy-id yinzhengjie@s103 The authenticity of host 's103 (172.16.30.103)' can't be established. ECDSA key fingerprint is fa:25:bc:03:7e:99:eb:12:1e:bc:a8:c9:ce:39:ba:7b. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys yinzhengjie@s103's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'yinzhengjie@s103'" and check to make sure that only the key(s) you wanted were added. [yinzhengjie@s101 ~]$ ssh s103 Last login: Fri May 25 18:35:45 2018 from 172.16.30.1 [yinzhengjie@s103 ~]$ who yinzhengjie pts/0 2018-05-25 18:35 (172.16.30.1) yinzhengjie pts/1 2018-05-25 19:19 (s101) [yinzhengjie@s103 ~]$ exit logout Connection to s103 closed. [yinzhengjie@s101 ~]$ who yinzhengjie pts/0 2018-05-25 18:35 (172.16.30.1) [yinzhengjie@s101 ~]$
5>.使用ssh-copy-id命令分配公鑰到DataNode服務器（172.16.30.104）
[yinzhengjie@s101 ~]$ ssh-copy-id yinzhengjie@s104 The authenticity of host 's104 (172.16.30.104)' can't be established. ECDSA key fingerprint is fa:25:bc:03:7e:99:eb:12:1e:bc:a8:c9:ce:39:ba:7b. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys yinzhengjie@s104's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'yinzhengjie@s104'" and check to make sure that only the key(s) you wanted were added. [yinzhengjie@s101 ~]$ ssh s104 Last login: Fri May 25 18:35:47 2018 from 172.16.30.1 [yinzhengjie@s104 ~]$ who yinzhengjie pts/0 2018-05-25 18:35 (172.16.30.1) yinzhengjie pts/1 2018-05-25 19:20 (s101) [yinzhengjie@s104 ~]$ exit logout Connection to s104 closed. [yinzhengjie@s101 ~]$ who yinzhengjie pts/0 2018-05-25 18:35 (172.16.30.1) [yinzhengjie@s101 ~]$
注意：以上是普通使配置免密登陸，root用戶配置方法一致，最好也配置上root用戶的免密登陸，由於下文我會執行相應的shell腳本。
[yinzhengjie@s101 ~]$ su Password: [root@s101 yinzhengjie]# cd [root@s101 ~]# [root@s101 ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa Generating public/private rsa key pair. Created directory '/root/.ssh'. Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: 9b:47:9a:ca:d2:f9:a5:57:79:35:40:be:07:3a:ed:40 root@s101 The key's randomart image is: +--[ RSA 2048]----+ | .. | | .. | | E o. | | . o o..| | S .+ + o.| | * * o | | . .= o. o | | ..o. +. | | .o.o. | +-----------------+ [root@s101 ~]# [root@s101 ~]# ssh-copy-id root@s101 The authenticity of host 's101 (172.16.30.101)' can't be established. ECDSA key fingerprint is fa:25:bc:03:7e:99:eb:12:1e:bc:a8:c9:ce:39:ba:7b. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@s101's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@s101'" and check to make sure that only the key(s) you wanted were added. [root@s101 ~]# ssh s101 Last login: Fri May 25 19:44:37 2018 [root@s101 ~]# who yinzhengjie pts/0 2018-05-25 18:35 (172.16.30.1) root pts/1 2018-05-25 19:49 (s101) [root@s101 ~]# exit logout Connection to s101 closed. [root@s101 ~]# who yinzhengjie pts/0 2018-05-25 18:35 (172.16.30.1) [root@s101 ~]#
[root@s101 ~]# ssh-copy-id root@s102 The authenticity of host 's102 (172.16.30.102)' can't be established. ECDSA key fingerprint is fa:25:bc:03:7e:99:eb:12:1e:bc:a8:c9:ce:39:ba:7b. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@s102's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@s102'" and check to make sure that only the key(s) you wanted were added. [root@s101 ~]# ssh s102 Last login: Fri May 25 09:28:59 2018 [root@s102 ~]# who yinzhengjie pts/0 2018-05-25 18:35 (172.16.30.1) root pts/1 2018-05-25 19:50 (s101) [root@s102 ~]# exit logout Connection to s102 closed. [root@s101 ~]# who yinzhengjie pts/0 2018-05-25 18:35 (172.16.30.1) [root@s101 ~]#
[root@s101 ~]# ssh-copy-id root@s103 /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@s103's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@s103'" and check to make sure that only the key(s) you wanted were added. [root@s101 ~]# ssh s103 Last login: Fri May 25 19:51:44 2018 from s101 [root@s103 ~]# who yinzhengjie pts/0 2018-05-25 18:35 (172.16.30.1) root pts/1 2018-05-25 19:52 (s101) [root@s103 ~]# exit logout Connection to s103 closed. [root@s101 ~]# who yinzhengjie pts/0 2018-05-25 18:35 (172.16.30.1) [root@s101 ~]#
[root@s101 ~]# ssh-copy-id root@s104 The authenticity of host 's104 (172.16.30.104)' can't be established. ECDSA key fingerprint is fa:25:bc:03:7e:99:eb:12:1e:bc:a8:c9:ce:39:ba:7b. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@s104's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@s104'" and check to make sure that only the key(s) you wanted were added. [root@s101 ~]# ssh s104 Last login: Fri May 25 09:31:15 2018 [root@s104 ~]# who yinzhengjie pts/0 2018-05-25 18:35 (172.16.30.1) root pts/1 2018-05-25 19:52 (s101) [root@s104 ~]# exit logout Connection to s104 closed. [root@s101 ~]# who yinzhengjie pts/0 2018-05-25 18:35 (172.16.30.1) [root@s101 ~]#
四.自定義shell腳本
1>.編寫在批量執行Linux命令的腳本（起名:xcall.sh）
[yinzhengjie@s101 ~]$ su Password: [root@s101 yinzhengjie]# [root@s101 yinzhengjie]# ll total 390272 drwxrwxr-x. 4 yinzhengjie yinzhengjie 35 May 25 19:08 hadoop -rw-r--r--. 1 root root 214092195 Aug 26 2016 hadoop-2.7.3.tar.gz -rw-r--r--. 1 root root 185540433 May 17 2017 jdk-8u131-linux-x64.tar.gz -rw-rw-r--. 1 yinzhengjie yinzhengjie 517 May 25 20:05 xcall.sh [root@s101 yinzhengjie]# more xcall.sh #!/bin/bash #@author :yinzhengjie #blog:http://www.cnblogs.com/yinzhengjie #EMAIL:y1053419035@qq.com #判斷用戶是否傳參 if [ $# -lt 1 ];then echo "請輸入參數" exit fi #獲取用戶輸入的命令 cmd=$@ for (( i=101;i<=104;i++ )) do #使終端變綠色 tput setaf 2 echo ============= s$i $cmd ============ #使終端變回原來的顏色，即白灰色 tput setaf 7 #遠程執行命令 ssh s$i $cmd #判斷命令是否執行成功 if [ $? == 0 ];then echo "命令執行成功" fi done [root@s101 yinzhengjie]# [root@s101 yinzhengjie]# sh xcall.sh "ln -s /soft/jdk/bin/jps /usr/local/bin/jps" ============= s101 ln -s /soft/jdk/bin/jps /usr/local/bin/jps ============ 命令執行成功 ============= s102 ln -s /soft/jdk/bin/jps /usr/local/bin/jps ============ 命令執行成功 ============= s103 ln -s /soft/jdk/bin/jps /usr/local/bin/jps ============ 命令執行成功 ============= s104 ln -s /soft/jdk/bin/jps /usr/local/bin/jps ============ 命令執行成功 [root@s101 yinzhengjie]# sh xcall.sh jps ============= s101 jps ============ 4977 Jps 命令執行成功 ============= s102 jps ============ 2854 Jps 命令執行成功 ============= s103 jps ============ 2822 Jps 命令執行成功 ============= s104 jps ============ 2788 Jps 命令執行成功 [root@s101 yinzhengjie]#
2>.編寫遠程同步腳本(起名：xrsync.sh)
在寫同步腳本以前，每一個客戶端都應該安裝的有rsync這個軟件，檢查各個虛擬機是否能夠正常連接互聯網，若是能夠訪問互聯網直接yum安裝一下，在各個虛擬機能夠訪問網絡且yum源配置正確的狀況下也能夠用我們上面的腳本（xcall.sh）執行批量安裝喲！
[root@s101 yinzhengjie]# sh xcall.sh "yum -y install rsync" ============= s101 yum -y install rsync ============ Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.huaweicloud.com * updates: mirrors.huaweicloud.com Resolving Dependencies --> Running transaction check ---> Package rsync.x86_64 0:3.1.2-4.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: rsync x86_64 3.1.2-4.el7 base 403 k Transaction Summary ================================================================================ Install 1 Package Total download size: 403 k Installed size: 815 k Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : rsync-3.1.2-4.el7.x86_64 1/1 Verifying : rsync-3.1.2-4.el7.x86_64 1/1 Installed: rsync.x86_64 0:3.1.2-4.el7 Complete! 命令執行成功 ============= s102 yum -y install rsync ============ Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.huaweicloud.com * updates: mirrors.huaweicloud.com Resolving Dependencies --> Running transaction check ---> Package rsync.x86_64 0:3.1.2-4.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: rsync x86_64 3.1.2-4.el7 base 403 k Transaction Summary ================================================================================ Install 1 Package Total download size: 403 k Installed size: 815 k Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : rsync-3.1.2-4.el7.x86_64 1/1 Verifying : rsync-3.1.2-4.el7.x86_64 1/1 Installed: rsync.x86_64 0:3.1.2-4.el7 Complete! 命令執行成功 ============= s103 yum -y install rsync ============ Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.huaweicloud.com * updates: mirrors.huaweicloud.com Resolving Dependencies --> Running transaction check ---> Package rsync.x86_64 0:3.1.2-4.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: rsync x86_64 3.1.2-4.el7 base 403 k Transaction Summary ================================================================================ Install 1 Package Total download size: 403 k Installed size: 815 k Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : rsync-3.1.2-4.el7.x86_64 1/1 Verifying : rsync-3.1.2-4.el7.x86_64 1/1 Installed: rsync.x86_64 0:3.1.2-4.el7 Complete! 命令執行成功 ============= s104 yum -y install rsync ============ Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.huaweicloud.com * updates: mirrors.huaweicloud.com Resolving Dependencies --> Running transaction check ---> Package rsync.x86_64 0:3.1.2-4.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: rsync x86_64 3.1.2-4.el7 base 403 k Transaction Summary ================================================================================ Install 1 Package Total download size: 403 k Installed size: 815 k Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : rsync-3.1.2-4.el7.x86_64 1/1 Verifying : rsync-3.1.2-4.el7.x86_64 1/1 Installed: rsync.x86_64 0:3.1.2-4.el7 Complete! 命令執行成功 [root@s101 yinzhengjie]#
#!/bin/bash #@author :yinzhengjie #blog:http://www.cnblogs.com/yinzhengjie #EMAIL:y1053419035@qq.com #判斷用戶是否傳參 if [ $# -lt 1 ];then echo "請輸入參數"; exit fi #獲取文件路徑 file=$@ #獲取子路徑 filename=`basename $file` #獲取父路徑 dirpath=`dirname $file` #獲取完整路徑 cd $dirpath fullpath=`pwd -P` #同步文件到DataNode for (( i=102;i<=104;i++ )) do #使終端變綠色 tput setaf 2 echo =========== s$i %file =========== #使終端變回原來的顏色，即白灰色 tput setaf 7 #遠程執行命令 rsync -lr $filename `whoami`@s$i:$fullpath #判斷命令是否執行成功 if [ $? == 0 ];then echo "命令執行成功" fi done
3>.將自定義腳本添加執行權限並連接到PATH環境變量中
[yinzhengjie@s101 ~]$ su Password: [root@s101 yinzhengjie]# ll total 390276 drwxrwxr-x. 4 yinzhengjie yinzhengjie 35 May 25 19:08 hadoop -rw-r--r--. 1 root root 214092195 Aug 26 2016 hadoop-2.7.3.tar.gz -rw-r--r--. 1 root root 185540433 May 17 2017 jdk-8u131-linux-x64.tar.gz -rw-rw-r--. 1 yinzhengjie yinzhengjie 517 May 25 20:05 xcall.sh -rw-rw-r--. 1 yinzhengjie yinzhengjie 700 May 25 20:32 xrsync.sh [root@s101 yinzhengjie]# chmod 755 xcall.sh xrsync.sh [root@s101 yinzhengjie]# ll total 390276 drwxrwxr-x. 4 yinzhengjie yinzhengjie 35 May 25 19:08 hadoop -rw-r--r--. 1 root root 214092195 Aug 26 2016 hadoop-2.7.3.tar.gz -rw-r--r--. 1 root root 185540433 May 17 2017 jdk-8u131-linux-x64.tar.gz -rwxr-xr-x. 1 yinzhengjie yinzhengjie 517 May 25 20:05 xcall.sh -rwxr-xr-x. 1 yinzhengjie yinzhengjie 700 May 25 20:32 xrsync.sh [root@s101 yinzhengjie]# [root@s101 yinzhengjie]# cp xcall.sh xrsync.sh /usr/local/bin/ [root@s101 yinzhengjie]# xcall.sh "ip addr | grep global " ============= s101 ip addr | grep global ============ inet 172.16.30.101/24 brd 172.16.30.255 scope global eno16777736 命令執行成功 ============= s102 ip addr | grep global ============ inet 172.16.30.102/24 brd 172.16.30.255 scope global eno16777736 命令執行成功 ============= s103 ip addr | grep global ============ inet 172.16.30.103/24 brd 172.16.30.255 scope global eno16777736 命令執行成功 ============= s104 ip addr | grep global ============ inet 172.16.30.104/24 brd 172.16.30.255 scope global eno16777736 命令執行成功 [root@s101 yinzhengjie]#
五.啓動服務並驗證是否成功
1>.使用自定義腳本同步配置文件
[yinzhengjie@s101 ~]$ xrsync.sh /soft/hadoop/etc/full/ =========== s102 %file =========== 命令執行成功 =========== s103 %file =========== 命令執行成功 =========== s104 %file =========== 命令執行成功 [yinzhengjie@s101 ~]$
2>.修改s102-s104的符號連接
[yinzhengjie@s101 ~]$ xcall.sh "ln -sfT /soft/hadoop/etc/full/ /soft/hadoop/etc/hadoop" ============= s101 ln -sfT /soft/hadoop/etc/full/ /soft/hadoop/etc/hadoop ============ 命令執行成功 ============= s102 ln -sfT /soft/hadoop/etc/full/ /soft/hadoop/etc/hadoop ============ 命令執行成功 ============= s103 ln -sfT /soft/hadoop/etc/full/ /soft/hadoop/etc/hadoop ============ 命令執行成功 ============= s104 ln -sfT /soft/hadoop/etc/full/ /soft/hadoop/etc/hadoop ============ 命令執行成功 [yinzhengjie@s101 ~]$
3>.格式化文件系統
[yinzhengjie@s101 ~]$ hdfs namenode -format 18/05/25 21:04:21 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = s101/172.16.30.101 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 2.7.3 STARTUP_MSG: classpath = /soft/hadoop-2.7.3/etc/hadoop:/soft/hadoop-2.7.3/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/stax-api-1.0-2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/activation-1.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jersey-server-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/asm-3.2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/log4j-1.2.17.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jets3t-0.9.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/httpclient-4.2.5.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/httpcore-4.2.5.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-lang-2.6.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-configuration-1.6.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-digester-1.8.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/avro-1.7.4.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/paranamer-2.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-compress-1.4.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/xz-1.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/gson-2.2.4.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/hadoop-auth-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/zookeeper-3.4.6.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/netty-3.6.2.Final.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/curator-framework-2.7.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/curator-client-2.7.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jsch-0.1.42.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/junit-4.11.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/hamcrest-core-1.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/mockito-all-1.8.5.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/hadoop-annotations-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/guava-11.0.2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jsr305-3.0.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-cli-1.2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-math3-3.1.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/xmlenc-0.52.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-httpclient-3.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-logging-1.1.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-codec-1.4.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-io-2.4.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-net-3.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-collections-3.2.2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/servlet-api-2.5.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jetty-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jetty-util-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jsp-api-2.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jersey-core-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jersey-json-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jettison-1.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/hadoop-common-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/hadoop-common-2.7.3-tests.jar:/soft/hadoop-2.7.3/share/hadoop/common/hadoop-nfs-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/guava-11.0.2.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-io-2.4.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/asm-3.2.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/hadoop-hdfs-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/hadoop-hdfs-2.7.3-tests.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-lang-2.6.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/guava-11.0.2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-cli-1.2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/log4j-1.2.17.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/activation-1.1.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/xz-1.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/servlet-api-2.5.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-codec-1.4.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-core-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-client-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/guice-3.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/javax.inject-1.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/aopalliance-1.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-io-2.4.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-server-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/asm-3.2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-json-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jettison-1.1.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jetty-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-api-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-common-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-common-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-client-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-registry-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/xz-1.0.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/asm-3.2.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/guice-3.0.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/javax.inject-1.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/junit-4.11.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar:/contrib/capacity-scheduler/*.jar STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r baa91f7c6bc9cb92be5982de4719c1c8af91ccff; compiled by 'root' on 2016-08-18T01:41Z STARTUP_MSG: java = 1.8.0_131 ************************************************************/ 18/05/25 21:04:21 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 18/05/25 21:04:21 INFO namenode.NameNode: createNameNode [-format] Formatting using clusterid: CID-4be4a490-3e1b-4675-b613-d37bb9f15790 18/05/25 21:04:22 INFO namenode.FSNamesystem: No KeyProvider found. 18/05/25 21:04:22 INFO namenode.FSNamesystem: fsLock is fair:true 18/05/25 21:04:22 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000 18/05/25 21:04:22 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true 18/05/25 21:04:22 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000 18/05/25 21:04:22 INFO blockmanagement.BlockManager: The block deletion will start around 2018 May 25 21:04:22 18/05/25 21:04:22 INFO util.GSet: Computing capacity for map BlocksMap 18/05/25 21:04:22 INFO util.GSet: VM type = 64-bit 18/05/25 21:04:22 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB 18/05/25 21:04:22 INFO util.GSet: capacity = 2^21 = 2097152 entries 18/05/25 21:04:22 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false 18/05/25 21:04:22 INFO blockmanagement.BlockManager: defaultReplication = 3 18/05/25 21:04:22 INFO blockmanagement.BlockManager: maxReplication = 512 18/05/25 21:04:22 INFO blockmanagement.BlockManager: minReplication = 1 18/05/25 21:04:22 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 18/05/25 21:04:22 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000 18/05/25 21:04:22 INFO blockmanagement.BlockManager: encryptDataTransfer = false 18/05/25 21:04:22 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000 18/05/25 21:04:22 INFO namenode.FSNamesystem: fsOwner = yinzhengjie (auth:SIMPLE) 18/05/25 21:04:22 INFO namenode.FSNamesystem: supergroup = supergroup 18/05/25 21:04:22 INFO namenode.FSNamesystem: isPermissionEnabled = true 18/05/25 21:04:22 INFO namenode.FSNamesystem: HA Enabled: false 18/05/25 21:04:22 INFO namenode.FSNamesystem: Append Enabled: true 18/05/25 21:04:22 INFO util.GSet: Computing capacity for map INodeMap 18/05/25 21:04:22 INFO util.GSet: VM type = 64-bit 18/05/25 21:04:22 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB 18/05/25 21:04:22 INFO util.GSet: capacity = 2^20 = 1048576 entries 18/05/25 21:04:22 INFO namenode.FSDirectory: ACLs enabled? false 18/05/25 21:04:22 INFO namenode.FSDirectory: XAttrs enabled? true 18/05/25 21:04:22 INFO namenode.FSDirectory: Maximum size of an xattr: 16384 18/05/25 21:04:22 INFO namenode.NameNode: Caching file names occuring more than 10 times 18/05/25 21:04:22 INFO util.GSet: Computing capacity for map cachedBlocks 18/05/25 21:04:22 INFO util.GSet: VM type = 64-bit 18/05/25 21:04:22 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB 18/05/25 21:04:22 INFO util.GSet: capacity = 2^18 = 262144 entries 18/05/25 21:04:22 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 18/05/25 21:04:22 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0 18/05/25 21:04:22 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000 18/05/25 21:04:22 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10 18/05/25 21:04:22 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10 18/05/25 21:04:22 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25 18/05/25 21:04:22 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 18/05/25 21:04:22 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 18/05/25 21:04:22 INFO util.GSet: Computing capacity for map NameNodeRetryCache 18/05/25 21:04:22 INFO util.GSet: VM type = 64-bit 18/05/25 21:04:22 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB 18/05/25 21:04:22 INFO util.GSet: capacity = 2^15 = 32768 entries 18/05/25 21:04:22 INFO namenode.FSImage: Allocated new BlockPoolId: BP-692948037-172.16.30.101-1527307462569 18/05/25 21:04:22 INFO common.Storage: Storage directory /home/yinzhengjie/hadoop/dfs/name has been successfully formatted. 18/05/25 21:04:22 INFO namenode.FSImageFormatProtobuf: Saving image file /home/yinzhengjie/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression 18/05/25 21:04:22 INFO namenode.FSImageFormatProtobuf: Image file /home/yinzhengjie/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 358 bytes saved in 0 seconds. 18/05/25 21:04:22 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 18/05/25 21:04:22 INFO util.ExitUtil: Exiting with status 0 18/05/25 21:04:22 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at s101/172.16.30.101 ************************************************************/ [yinzhengjie@s101 ~]$ echo $? 0 [yinzhengjie@s101 ~]$
4>.啓動hadoop
[yinzhengjie@s101 ~]$ start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh Starting namenodes on [s101] s101: starting namenode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-namenode-s101.out s102: starting datanode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-datanode-s102.out s104: starting datanode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-datanode-s104.out s103: starting datanode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-datanode-s103.out Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-secondarynamenode-s101.out starting yarn daemons starting resourcemanager, logging to /soft/hadoop-2.7.3/logs/yarn-yinzhengjie-resourcemanager-s101.out s103: starting nodemanager, logging to /soft/hadoop-2.7.3/logs/yarn-yinzhengjie-nodemanager-s103.out s104: starting nodemanager, logging to /soft/hadoop-2.7.3/logs/yarn-yinzhengjie-nodemanager-s104.out s102: starting nodemanager, logging to /soft/hadoop-2.7.3/logs/yarn-yinzhengjie-nodemanager-s102.out [yinzhengjie@s101 ~]$
5>.用自定義腳本驗證NameNode和DataNode是否已經正常啓動
[yinzhengjie@s101 ~]$ xcall.sh jps ============= s101 jps ============ 7593 NameNode 7946 ResourceManager 8220 Jps 7789 SecondaryNameNode 命令執行成功 ============= s102 jps ============ 3344 NodeManager 3241 DataNode 3453 Jps 命令執行成功 ============= s103 jps ============ 3296 NodeManager 3193 DataNode 3406 Jps 命令執行成功 ============= s104 jps ============ 3395 Jps 3286 NodeManager 3183 DataNode 命令執行成功 [yinzhengjie@s101 ~]$
6>.上傳文件到服務器
[yinzhengjie@s101 ~]$ netstat -tnlp | grep "java" (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp 0 0 0.0.0.0:50090 0.0.0.0:* LISTEN 7789/java tcp 0 0 172.16.30.101:8020 0.0.0.0:* LISTEN 7593/java tcp 0 0 0.0.0.0:50070 0.0.0.0:* LISTEN 7593/java tcp6 0 0 172.16.30.101:8088 :::* LISTEN 7946/java tcp6 0 0 172.16.30.101:8030 :::* LISTEN 7946/java tcp6 0 0 172.16.30.101:8031 :::* LISTEN 7946/java tcp6 0 0 172.16.30.101:8032 :::* LISTEN 7946/java tcp6 0 0 172.16.30.101:8033 :::* LISTEN 7946/java [yinzhengjie@s101 ~]$ [yinzhengjie@s101 ~]$ hdfs dfs -put xcall.sh xrsync.sh / [yinzhengjie@s101 ~]$ hdfs dfs -ls / Found 2 items -rw-r--r-- 3 yinzhengjie supergroup 517 2018-05-25 21:17 /xcall.sh -rw-r--r-- 3 yinzhengjie supergroup 700 2018-05-25 21:17 /xrsync.sh [yinzhengjie@s101 ~]$
7>.客戶端檢查是否可用正常訪問Hadoop的UI界面
五.Hadoop經常使用參數介紹
Hadoop有不少參數，其默認配置大多數僅適用於standalone模式，雖然大多狀況下在徹底分佈式(Fully distributed)模式中也沒有問題，但距最優化的運行模式去相去甚遠。在生產環境中一般須要調整的參數以下：
1>.dfs.name.dir 做用：NameNode節點用於存儲HDFS元數據的本地目錄，官方建議爲/home/hadoop/dfs/name； 2>.dfs.data.dir 做用：DataNode節點用於存儲HDFS文件數據塊的本地目錄，官方建議爲/home/hadoop/dfs/data； 3>.mapred.system.dir 做用：HDFS中用於存儲共享的MapReduce系統文件的目錄，官方建議爲/hadoop/mapred/system； 4>.mapred.local.dir 做用：TaskNode節點用於存儲臨時數據的本地文件目錄； 5>.mapred.tasktracker.{map|reduce}.tarks.maximum 做用：在TaskTracker上可同時運行的的map或reduce任務的最大數目； 6>.hadoop.tmp.dir 做用： Hadoop臨時目錄； 7>.mapred.child.java.opts 做用：每一個子任務可申請使用的heap大小；官方建議爲-Xmx512m； 8>.mapred.reduce.tasks 做用：每任務的reduce數量；
上述參數中的大多數均可以接受多個以逗號分隔的目錄，尤爲是對於dfs.name.dir來講，多個目錄還能夠達到冗餘的目的；而對於擁有多塊磁盤的DataNode，爲其dfs.data.dir指定多個值能夠存儲數據於多個磁盤，並能經過並行加速I/O操做。爲mapred.local.dir指定多個眼光也能起到必定的加速做用。
此外，hadoop.tmp.dir對於不一樣的用戶來講其路徑是不相同的，事實上，應該儘可能避免讓此路徑依賴用戶屬性，好比能夠放在一個公共位置讓全部用戶均可以方便地訪問到。在Linux系統下，hadoop.tmp.dir的默認路徑指向了/tmp，這的確是一個公共位置，但/tmp目錄所在的文件系統大多數都有使用配額，並且空間也一般比較有限，所以，故此此路徑殊非理想之所在。建議將其指向一個有着足夠空間的文件系統上的目錄。
默認配置中，Hadoop能夠在每一個TaskTracker上運行四個任務(兩個map任務，兩個reduce任務)，這能夠經過mapred.tasktracker.{map|reduce}.tarks.maximum進行配置，一般建議其個數爲與CPU核心數目相同或者爲CPU核心數目的2倍，但其最佳值一般依賴於諸多因素，而在CPU密集型的應用場景中也不該該將其最大數目設置得太高。除了CPU以外，還應該考慮每一個任務所可以使用的的heap空間大小所帶來的影響；默認狀況下，Hadoop爲每一個任務指定了200MB的heap空間上限，因爲每一個job可能會申請使用很大的heap，所以，太高的設定可能會帶來意外的結果。
每一個MapReduce任務能夠經過mapred.reduce.tasks配置其運行的reduce任務數，一般也應該爲其指定一個在多數場景下都能工做良好的默認值，好比Hadoop默認將此數目指定爲1個，這對大多數任務來說都有着不錯的性能表現。而實際使用中，通常建議將此值設定爲當前Hadoop集羣能夠運行的reduce任務總數的0.95倍或1.75倍。0.95這個因子意味着Hadoop集羣能夠當即啓動全部的reduce任務並在map任務完成時接收數據並進行處理，而1.75則意味着先啓動部分reduce任務，執行速度快的節點上的reduce完成後可再啓動一個新的reduce任務，而速度慢的節點則無須執行此類操做。這會帶來較爲理想的負載均衡效果。
相關文章
- 1. Apache Hadoop 2.9.2 徹底分佈式部署
- 2. hadoop徹底分佈式部署
- 3. Hadoop徹底分佈式部署
- 4. hadoop 徹底分佈式部署
- 5. 徹底分佈式部署hadoop
- 6. hadoop單機版、僞分佈式、徹底分佈式部署、HBase 徹底分佈式部署
- 7. Hadoop部署安裝 Hadoop3.1.2的徹底分佈式部署
- 8. Hadoop 徹底分佈式
- 9. hadoop筆記之部署hadoop集羣(徹底分佈式)
- 10. Hadoop部署方式-僞分佈式(Pseudo-Distributed Mode)
- 更多相關文章...
- • ionic 頭部與底部 - ionic 教程
- • ionic 頭部和底部 - ionic 教程
- • 常用的分佈式事務解決方案
- • 再有人問你分佈式事務，把這篇扔給他
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
歡迎關注本站公眾號,獲取更多信息
