大數據概述 Hadoop配置

時間 2019-12-11

標籤數據概述 hadoop 配置欄目 Hadoop 简体版

原文原文鏈接

Top

NSD ARCHITECTURE DAY05

1 案例1：安裝Hadoop

1.1 問題

本案例要求安裝單機模式Hadoop：html

單機模式安裝Hadoop
安裝JAVA環境
設置環境變量，啓動運行

1.2 步驟

實現此案例須要按照以下步驟進行。java

步驟一：環境準備node

1）配置主機名爲nn01，ip爲192.168.1.21，配置yum源（系統源）vim

備註：因爲在以前的案例中這些都已經作過，這裏再也不重複，不會的學員能夠參考以前的案例app

2）安裝java環境dom

[root@nn01 ~]# yum -y install java-1.8.0-openjdk-devel
[root@nn01 ~]# java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b12)
OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)
[root@nn01 ~]# jps
1235 Jps

3）安裝hadoopssh

[root@nn01 ~]# tar -xf hadoop-2.7.6.tar.gz
[root@nn01 ~]# mv hadoop-2.7.6 /usr/local/hadoop
[root@nn01 ~]# cd /usr/local/hadoop/
[root@nn01 hadoop]# ls
bin include libexec NOTICE.txt sbin
etc lib LICENSE.txt README.txt share
[root@nn01 hadoop]# ./bin/hadoop //報錯，JAVA_HOME沒有找到
Error: JAVA_HOME is not set and could not be found.
[root@nn01 hadoop]#

4）解決報錯問題curl

[root@nn01 hadoop]# rpm -ql java-1.8.0-openjdk
[root@nn01 hadoop]# cd ./etc/hadoop/
[root@nn01 hadoop]# vim hadoop-env.sh
25 export \
JAVA_HOME="/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/jre"
33 export HADOOP_CONF_DIR="/usr/local/hadoop/etc/hadoop"
[root@nn01 ~]# cd /usr/local/hadoop/
[root@nn01 hadoop]# ./bin/hadoop
Usage: hadoop [--config confdir] [COMMAND | CLASSNAME]
CLASSNAME run the class named CLASSNAME
or
where COMMAND is one of:
fs run a generic filesystem user client
version print the version
jar <jar> run a jar file
note: please use "yarn jar" to launch
YARN applications, not this command.
checknative [-a|-h] check native hadoop and compression libraries availability
distcp <srcurl> <desturl> copy file or directories recursively
archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive
classpath prints the class path needed to get the
credential interact with credential providers
Hadoop jar and the required libraries
daemonlog get/set the log level for each daemon
trace view and modify Hadoop tracing settings
Most commands print help when invoked w/o parameters.
[root@nn01 hadoop]# mkdir /usr/local/hadoop/aa
[root@nn01 hadoop]# ls
bin etc include lib libexec LICENSE.txt NOTICE.txt aa README.txt sbin share
[root@nn01 hadoop]# cp *.txt /usr/local/hadoop/aa
[root@nn01 hadoop]# ./bin/hadoop jar \
share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.6.jar wordcount aa bb //wordcount爲參數統計aa這個文件夾，存到bb這個文件裏面（這個文件不能存在，要是存在會報錯，是爲了防止數據覆蓋）
[root@nn01 hadoop]# cat bb/part-r-00000 //查看

2 案例2：安裝配置Hadoop

2.1 問題

本案例要求：jvm

另備三臺虛擬機，安裝Hadoop
使全部節點可以ping通，配置SSH信任關係
節點驗證

2.2 方案

準備四臺虛擬機，因爲以前已經準備過一臺，因此只需再準備三臺新的虛擬機便可，安裝hadoop，使全部節點能夠ping通，配置SSH信任關係，如圖-1所示：tcp

圖-1

2.3 步驟

實現此案例須要按照以下步驟進行。

步驟一：環境準備

1）三臺機器配置主機名爲node一、node二、node3，配置ip地址（ip如圖-1所示），yum源（系統源）

2）編輯/etc/hosts（四臺主機一樣操做，以nn01爲例）

[root@nn01 ~]# vim /etc/hosts
192.168.1.21 nn01
192.168.1.22 node1
192.168.1.23 node2
192.168.1.24 node3

3）安裝java環境，在node1，node2，node3上面操做（以node1爲例）

[root@node1 ~]# yum -y install java-1.8.0-openjdk-devel

4）佈置SSH信任關係

[root@nn01 ~]# vim /etc/ssh/ssh_config //第一次登錄不須要輸入yes
Host *
GSSAPIAuthentication yes
StrictHostKeyChecking no
[root@nn01 .ssh]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:Ucl8OCezw92aArY5+zPtOrJ9ol1ojRE3EAZ1mgndYQM root@nn01
The key's randomart image is:
+---[RSA 2048]----+
| o*E*=. |
| +XB+. |
| ..=Oo. |
| o.+o... |
| .S+.. o |
| + .=o |
| o+oo |
| o+=.o |
| o==O. |
+----[SHA256]-----+
[root@nn01 .ssh]# for i in 21 22 23 24 ; do ssh-copy-id 192.168.1.$i; done
//部署公鑰給nn01，node1，node2，node3

5）測試信任關係

[root@nn01 .ssh]# ssh node1
Last login: Fri Sep 7 16:52:00 2018 from 192.168.1.21
[root@node1 ~]# exit
logout
Connection to node1 closed.
[root@nn01 .ssh]# ssh node2
Last login: Fri Sep 7 16:52:05 2018 from 192.168.1.21
[root@node2 ~]# exit
logout
Connection to node2 closed.
[root@nn01 .ssh]# ssh node3

步驟二：配置hadoop

1）修改slaves文件

[root@nn01 ~]# cd /usr/local/hadoop/etc/hadoop
[root@nn01 hadoop]# vim slaves
node1
node2
node3

2）hadoop的核心配置文件core-site

[root@nn01 hadoop]# vim core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://nn01:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/var/hadoop</value>
</property>
</configuration>
[root@nn01 hadoop]# mkdir /var/hadoop //hadoop的數據根目錄
[root@nn01 hadoop]# ssh node1 mkdir /var/hadoop
[root@nn01 hadoop]# ssh node2 mkdir /var/hadoop
[root@nn01 hadoop]# ssh node3 mkdir /var/hadoop

3）配置hdfs-site文件

[root@nn01 hadoop]# vim hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.http-address</name>
<value>nn01:50070</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>nn01:50090</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
</configuration>

4）同步配置到node1，node2，node3

[root@nn01 hadoop]# yum –y install rsync //同步的主機都要安裝rsync
[root@nn01 hadoop]# for i in 22 23 24 ; do rsync -aSH --delete /usr/local/hadoop/
\ 192.168.1.$i:/usr/local/hadoop/ -e 'ssh' & done
[1] 23260
[2] 23261
[3] 23262