hdfs節點管理 yarn階段管理 nfs配置

Top

NSD ARCHITECTURE DAY06

1. 案例1：安裝與部署
2. 案例2：Hadoop詞頻統計
3. 案例3：節點管理
4. 案例4：NFS配置

1 案例1：安裝與部署

1.1 問題

本案例要求：

• 對mapred和yarn文件進行配置
• 驗證訪問Hadoop

1.2 方案

在day05準備好的環境下給master （nn01）主機添加ResourceManager的角色，在node1，node2，node3上面添加NodeManager的角色，如表-1所示：

表-1

1.3 步驟

實現此案例須要按照以下步驟進行。

步驟一：安裝與部署hadoop

1）配置mapred-site（nn01上面操做）

1. [root@nn01 ~]# cd /usr/local/hadoop/etc/hadoop/
2. [root@nn01 hadoop]# mv mapred-site.xml.template mapred-site.xml
3. [root@nn01 hadoop]# vim mapred-site.xml
4. <configuration>
5. <property>
6. <name>mapreduce.framework.name</name>
7. <value>yarn</value>
8. </property>
9. </configuration>

2）配置yarn-site（nn01上面操做）

1. [root@nn01 hadoop]# vim yarn-site.xml
2. <configuration>
4. <!-- Site specific YARN configuration properties -->
5. <property>
6. <name>yarn.resourcemanager.hostname</name>
7. <value>nn01</value>
8. </property>
9. <property>
10. <name>yarn.nodemanager.aux-services</name>
11. <value>mapreduce_shuffle</value>
12. </property>
13. </configuration>

3）同步配置（nn01上面操做）

1. [root@nn01 hadoop]# for i in {22..24}; do rsync -aSH --delete /usr/local/hadoop/ 192.168.1.$i:/usr/local/hadoop/ -e 'ssh' & done
2. [1] 712
3. [2] 713
4. [3] 714

4）驗證配置（nn01上面操做）

1. [root@nn01 hadoop]# cd /usr/local/hadoop
2. [root@nn01 hadoop]# ./sbin/start-dfs.sh
3. Starting namenodes on [nn01]
4. nn01: namenode running as process 23408. Stop it first.
5. node1: datanode running as process 22409. Stop it first.
6. node2: datanode running as process 22367. Stop it first.
7. node3: datanode running as process 22356. Stop it first.
8. Starting secondary namenodes [nn01]
9. nn01: secondarynamenode running as process 23591. Stop it first.
10. [root@nn01 hadoop]# ./sbin/start-yarn.sh
11. starting yarn daemons
12. starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-root-resourcemanager-nn01.out
13. node2: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-node2.out
14. node3: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-node3.out
15. node1: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-node1.out
16. [root@nn01 hadoop]# jps //nn01查看有ResourceManager
17. 23408 NameNode
18. 1043 ResourceManager
19. 1302 Jps
20. 23591 SecondaryNameNode
21. [root@nn01 hadoop]# ssh node1 jps        //node1查看有NodeManager
22. 25777 Jps
23. 22409 DataNode
24. 25673 NodeManager
25. [root@nn01 hadoop]# ssh node2 jps        //node1查看有NodeManager
26. 25729 Jps
27. 25625 NodeManager
28. 22367 DataNode
29. [root@nn01 hadoop]# ssh node3 jps        //node1查看有NodeManager
30. 22356 DataNode
31. 25620 NodeManager
32. 25724 Jps

5）web訪問hadoop

1. http://192.168.1.21:50070/            //--namenode web頁面（nn01）
2. http://192.168.1.21:50090/        //--secondory namenode web頁面（nn01）
3. http://192.168.1.22:50075/        //--datanode web頁面（node1,node2,node3）
4. http://192.168.1.21:8088/        //--resourcemanager web頁面（nn01）
5. http://192.168.1.22:8042/        //--nodemanager web頁面（node1,node2,node3）

2 案例2：Hadoop詞頻統計

2.1 問題

本案例要求：

• 在集羣文件系統裏建立文件夾
• 上傳要分析的文件到目錄中
• 分析上傳文件
• 展現結果

2.2 步驟

實現此案例須要按照以下步驟進行。

步驟一：詞頻統計

1. [root@nn01 hadoop]# ./bin/hadoop fs -ls /        //查看集羣文件系統的根，沒有內容
2. [root@nn01 hadoop]# ./bin/hadoop fs -mkdir /aaa        
3. //在集羣文件系統下建立aaa目錄
4. [root@nn01 hadoop]# ./bin/hadoop fs -ls /        //再次查看，有剛建立的aaa目錄
5. Found 1 items
6. drwxr-xr-x - root supergroup 0 2018-09-10 09:56 /aaa
7. [root@nn01 hadoop]# ./bin/hadoop fs -touchz /fa    //在集羣文件系統下建立fa文件
8. [root@nn01 hadoop]# ./bin/hadoop fs -put *.txt /aaa     
9. //上傳*.txt到集羣文件系統下的aaa目錄
10. [root@nn01 hadoop]# ./bin/hadoop fs -ls /aaa //查看
11. Found 3 items
12. -rw-r--r-- 2 root supergroup 86424 2018-09-10 09:58 /aaa/LICENSE.txt
13. -rw-r--r-- 2 root supergroup 14978 2018-09-10 09:58 /aaa/NOTICE.txt
14. -rw-r--r-- 2 root supergroup 1366 2018-09-10 09:58 /aaa/README.txt
15. [root@nn01 hadoop]# ./bin/hadoop fs -get /aaa //下載集羣文件系統的aaa目錄
16. [root@nn01 hadoop]# ./bin/hadoop jar \
17. share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.6.jar wordcount /aaa /bbb    //hadoop集羣分析大數據，hadoop集羣/aaa裏的數據存到hadoop集羣/bbb下
18. [root@nn01 hadoop]# ./bin/hadoop fs -cat /bbb/*        //查看集羣裏的數據

3 案例3：節點管理

3.1 問題

本案例要求：

• 增長一個新的節點
• 查看狀態
• 刪除節點

3.2 方案：

另外準備兩臺主機，node4和nfsgw，做爲新添加的節點和網關，具體要求如表-2所示：

表-2

3.3 步驟

實現此案例須要按照以下步驟進行。

步驟一：增長節點

1）增長一個新的節點node4

1. [root@hadoop5 ~]# echo node4 > /etc/hostname     //更改主機名爲node4
2. [root@hadoop5 ~]# hostname node4
3. [root@node4 ~]# yum -y install rsync
4. [root@node4 ~]# yum -y install java-1.8.0-openjdk-devel
5. [root@node4 ~]# mkdir /var/hadoop
6. [root@nn01 .ssh]# ssh-copy-id 192.168.1.25
7. [root@nn01 .ssh]# vim /etc/hosts
8. 192.168.1.21 nn01
9. 192.168.1.22 node1
10. 192.168.1.23 node2
11. 192.168.1.24 node3
12. 192.168.1.25 node4
13. [root@nn01 .ssh]# scp /etc/hosts 192.168.1.25:/etc/
14. [root@nn01 ~]# cd /usr/local/hadoop/
15. [root@nn01 hadoop]# vim ./etc/hadoop/slaves
16. node1
17. node2
18. node3
19. node4
20. [root@nn01 hadoop]# for i in {22..25}; do rsync -aSH --delete /usr/local/hadoop/
21. \ 192.168.1.$i:/usr/local/hadoop/ -e 'ssh' & done        //同步配置
22. [1] 1841
23. [2] 1842
24. [3] 1843
25. [4] 1844
26. [root@node4 hadoop]# ./sbin/hadoop-daemon.sh start datanode //啓動

2）查看狀態

1. [root@node4 hadoop]# jps
2. 24439 Jps
3. 24351 DataNode

3）設置同步帶寬

1. [root@node4 hadoop]# ./bin/hdfs dfsadmin -setBalancerBandwidth 60000000
2. Balancer bandwidth is set to 60000000
3. [root@node4 hadoop]# ./sbin/start-balancer.sh

4）刪除節點

1. [root@nn01 hadoop]# vim /usr/local/hadoop/etc/hadoop/slaves        
2. //去掉以前添加的node4
3. node1
4. node2
5. node3
6. [root@nn01 hadoop]# vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml        
7. //在此配置文件裏面加入下面四行
8. <property>
9. <name>dfs.hosts.exclude</name>
10. <value>/usr/local/hadoop/etc/hadoop/exclude</value>
11. </property>
13. [root@nn01 hadoop]# vim /usr/local/hadoop/etc/hadoop/exclude
14. node4

5）導出數據

1. [root@nn01 hadoop]# ./bin/hdfs dfsadmin -refreshNodes
2. Refresh nodes successful
3. [root@nn01 hadoop]# ./bin/hdfs dfsadmin -report //查看node4顯示Decommissioned
4. Dead datanodes (1):
6. Name: 192.168.1.25:50010 (node4)
7. Hostname: node4
8. Decommission Status : Decommissioned
9. Configured Capacity: 17168314368 (15.99 GB)
10. DFS Used: 12288 (12 KB)
11. Non DFS Used: 1656664064 (1.54 GB)
12. DFS Remaining: 15511638016 (14.45 GB)
13. DFS Used%: 0.00%
14. DFS Remaining%: 90.35%
15. Configured Cache Capacity: 0 (0 B)
16. Cache Used: 0 (0 B)
17. Cache Remaining: 0 (0 B)
18. Cache Used%: 100.00%
19. Cache Remaining%: 0.00%
20. Xceivers: 1
21. Last contact: Mon Sep 10 10:59:58 CST 2018
23. [root@node4 hadoop]# ./sbin/hadoop-daemon.sh stop datanode    //中止datanode
24. stopping datanode
25. [root@node4 hadoop]# ./sbin/yarn-daemon.sh start nodemanager    
26. //yarn 增長 nodemanager
27. [root@node4 hadoop]# ./sbin/yarn-daemon.sh stop nodemanager //中止nodemanager
28. stopping nodemanager
29. [root@node4 hadoop]# ./bin/yarn node -list        
30. //yarn 查看節點狀態，仍是有node4節點，要過一段時間纔會消失
31. 18/09/10 11:04:50 INFO client.RMProxy: Connecting to ResourceManager at nn01/192.168.1.21:8032
32. Total Nodes:4
33. Node-Id     Node-State    Node-Http-Address    Number-of-Running-Containers
34. node3:34628     RUNNING     node3:8042     0
35. node2:36300     RUNNING     node2:8042     0
36. node4:42459     RUNNING     node4:8042     0
37. node1:39196     RUNNING     node1:8042     0

4 案例4：NFS配置

4.1 問題

本案例要求：

• 建立代理用戶
• 啓動一個新系統，禁用Selinux和firewalld
• 配置NFSWG
• 啓動服務
• 掛載NFS並實現開機自啓

4.2 步驟

實現此案例須要按照以下步驟進行。

步驟一：基礎準備

1）更改主機名，配置/etc/hosts（/etc/hosts在nn01和nfsgw上面配置）

1. [root@localhost ~]# echo nfsgw > /etc/hostname
2. [root@localhost ~]# hostname nfsgw
3. [root@nn01 hadoop]# vim /etc/hosts
4. 192.168.1.21 nn01
5. 192.168.1.22 node1
6. 192.168.1.23 node2
7. 192.168.1.24 node3
8. 192.168.1.25 node4
9. 192.168.1.26 nfsgw

2）建立代理用戶（nn01和nfsgw上面操做），以nn01爲例子

1. [root@nn01 hadoop]# groupadd -g 200 nfs
2. [root@nn01 hadoop]# useradd -u 200 -g nfs nfs

3）配置core-site.xml

1. [root@nn01 hadoop]# ./sbin/stop-all.sh //中止全部服務
2. This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
3. Stopping namenodes on [nn01]
4. nn01: stopping namenode
5. node2: stopping datanode
6. node4: no datanode to stop
7. node3: stopping datanode
8. node1: stopping datanode
9. Stopping secondary namenodes [nn01]
10. nn01: stopping secondarynamenode
11. stopping yarn daemons
12. stopping resourcemanager
13. node2: stopping nodemanager
14. node3: stopping nodemanager
15. node4: no nodemanager to stop
16. node1: stopping nodemanager
17. ...
19. [root@nn01 hadoop]# cd etc/hadoop
20. [root@nn01 hadoop]# >exclude
21. [root@nn01 hadoop]# vim core-site.xml
22. <property>
23. <name>hadoop.proxyuser.nfs.groups</name>
24. <value>*</value>
25. </property>
26. <property>
27. <name>hadoop.proxyuser.nfs.hosts</name>
28. <value>*</value>
29. </property>

4）同步配置到node1，node2，node3

1. [root@nn01 hadoop]# for i in {22..24}; do rsync -aSH --delete /usr/local/hadoop/ 192.168.1.$i:/usr/local/hadoop/ -e 'ssh' & done
2. [4] 2722
3. [5] 2723
4. [6] 2724

5）啓動集羣

1. [root@nn01 hadoop]# /usr/local/hadoop/sbin/start-dfs.sh

6）查看狀態

1. [root@nn01 hadoop]# /usr/local/hadoop/bin/hdfs dfsadmin -report

步驟二：NFSGW配置

1）安裝java-1.8.0-openjdk-devel和rsync

1. [root@nfsgw ~]# yum -y install java-1.8.0-openjdk-devel
2. [root@nfsgw ~]# yum -y install rsync
3. [root@nn01 hadoop]# rsync -avSH --delete \
4. /usr/local/hadoop/ 192.168.1.26:/usr/local/hadoop/ -e 'ssh'

2）建立數據根目錄 /var/hadoop（在NFSGW主機上面操做）

1. [root@nfsgw ~]# mkdir /var/hadoop

3）建立轉儲目錄，並給用戶nfs 賦權

1. [root@nfsgw ~]# mkdir /var/nfstmp
2. [root@nfsgw ~]# chown nfs:nfs /var/nfstmp

4）給/usr/local/hadoop/logs賦權（在NFSGW主機上面操做）

1. [root@nfsgw ~]# setfacl -m u:nfs:rwx /usr/local/hadoop/logs
2. [root@nfsgw ~]# vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml
3. <property>
4. <name>nfs.exports.allowed.hosts</name>
5. <value>* rw</value>
6. </property>
7. <property>
8. <name>nfs.dump.dir</name>
9. <value>/var/nfstmp</value>
10. </property>

5）能夠建立和刪除便可

1. [root@nfsgw ~]# su - nfs
2. [nfs@nfsgw ~]$ cd /var/nfstmp/
3. [nfs@nfsgw nfstmp]$ touch 1
4. [nfs@nfsgw nfstmp]$ ls
5. 1
6. [nfs@nfsgw nfstmp]$ rm -rf 1
7. [nfs@nfsgw nfstmp]$ ls
8. [nfs@nfsgw nfstmp]$ cd /usr/local/hadoop/logs/
9. [nfs@nfsgw logs]$ touch 1
10. [nfs@nfsgw logs]$ ls
11. 1 hadoop-root-secondarynamenode-nn01.log yarn-root-resourcemanager-nn01.log
12. hadoop-root-namenode-nn01.log hadoop-root-secondarynamenode-nn01.out yarn-root-resourcemanager-nn01.out
13. hadoop-root-namenode-nn01.out hadoop-root-secondarynamenode-nn01.out.1
14. hadoop-root-namenode-nn01.out.1 SecurityAuth-root.audit
15. [nfs@nfsgw logs]$ rm -rf 1
16. [nfs@nfsgw logs]$ ls

6）啓動服務

1. [root@nfsgw ~]# /usr/local/hadoop/sbin/hadoop-daemon.sh --script ./bin/hdfs start portmap        //portmap服務只能用root用戶啓動
2. starting portmap, logging to /usr/local/hadoop/logs/hadoop-root-portmap-nfsgw.out
3. [root@nfsgw ~]# jps
4. 23714 Jps
5. 23670 Portmap
7. [root@nfsgw ~]# su - nfs
8. Last login: Mon Sep 10 12:31:58 CST 2018 on pts/0
9. [nfs@nfsgw ~]$ cd /usr/local/hadoop/
10. [nfs@nfsgw hadoop]$ ./sbin/hadoop-daemon.sh --script ./bin/hdfs start nfs3
11. //nfs3只能用代理用戶啓動
12. starting nfs3, logging to /usr/local/hadoop/logs/hadoop-nfs-nfs3-nfsgw.out
13. [nfs@nfsgw hadoop]$ jps                    
14. 1362 Jps
15. 1309 Nfs3
16. [root@nfsgw hadoop]# jps            //root用戶執行能夠看到portmap和nfs3
17. 1216 Portmap
18. 1309 Nfs3
19. 1374 Jps

7）實現客戶端掛載（客戶端能夠用node4這臺主機）

1. [root@node4 ~]# rm -rf /usr/local/hadoop
2. [root@node4 ~]# yum -y install nfs-utils
3. [root@node4 ~]# mount -t nfs -o \
4. vers=3,proto=tcp,nolock,noatime,sync,noacl 192.168.1.26:/ /mnt/ //掛載
5. [root@node4 ~]# cd /mnt/
6. [root@node4 mnt]# ls
7. aaa bbb fa system tmp
8. [root@node4 mnt]# touch a
9. [root@node4 mnt]# ls
10. a aaa bbb fa system tmp
11. [root@node4 mnt]# rm -rf a
12. [root@node4 mnt]# ls
13. aaa bbb fa system tmp

8）實現開機自動掛載

1. [root@node4 ~]# vim /etc/fstab
2. 192.168.1.26:/ /mnt/ nfs vers=3,proto=tcp,nolock,noatime,sync,noacl,_netdev 0 0
3. [root@node4 ~]# mount -a
4. [root@node4 ~]# df -h
5. 192.168.1.26:/ 64G 6.2G 58G 10% /mnt
7. [root@node4 ~]# rpcinfo -p 192.168.1.26
8. program vers proto port service
9. 100005 3 udp 4242 mountd
10. 100005 1 tcp 4242 mountd
11. 100000 2 udp 111 portmapper
12. 100000 2 tcp 111 portmapper
13. 100005 3 tcp 4242 mountd
14. 100005 2 tcp 4242 mountd
15. 100003 3 tcp 2049 nfs
16. 100005 2 udp 4242 mountd
17. 100005 1 udp 4242 mountd