Linux下安裝配置Hadoop

背景

CentOS7
jdk1.8
hadoop-2.9.2

下載hadoop

wget http://mirrors.shu.edu.cn/apache/hadoop/common/hadoop-2.9.2/hadoop-2.9.2.tar.gz

tar -xzvf hadoop-2.9.2.tar.gz

mv hadoop-2.9.2 /usr/local/hadoop

安裝jdk8環境

參考Linux下安裝配置JDK

vim /etc/profile

JAVA_HOME=/usr/local/jdk1.8.0_171
JRE_HOME=/usr/local/jdk1.8.0_171/jre
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
export PATH

source /etc/profile

echo $JAVA_HOME /usr/local/jdk1.8.0_171

hadoop環境變量

vim /etc/profile

HADOOP_HOME=/usr/local/hadoop
PATH=$PATH:$HADOOP_HOME/bin
export PATH

source /etc/profile

hadoop version

hadoop環境配置

hadoop 配製文件

hadoop/etc/hadoop/hadoop-env.sh 
hadoop/etc/hadoop/yarn-env.sh 
hadoop/etc/hadoop/core-site.xml 
hadoop/etc/hadoop/hdfs-site.xml 
hadoop/etc/hadoop/mapred-site.xml 
hadoop/etc/hadoop/yarn-site.xml

hadoop-env.sh

cd /usr/local/hadoop/etc/hadoop/

vim hadoop-env.sh

# The java implementation to use.
#export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=/usr/local/jdk1.8.0_171

yarn-env.sh

cd /usr/local/hadoop/etc/hadoop/

vim yarn-env.sh

export JAVA_HOME=/usr/local/jdk1.8.0_171

core-site.xml 

cd /usr/local/hadoop

mkdir tmp

cd /usr/local/hadoop/etc/hadoop/

vim core-site.xml 

<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://192.168.140.134:9000</value>
        <description>HDFS的URI，文件系統://namenode標識:端口號</description>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/hadoop/tmp</value>
        <description>namenode上本地的hadoop臨時文件夾</description>
    </property>
</configuration>

--添加權限

hdfs dfs -chmod -R 777 /tmp

注意：這裏的ip地址 192.168.140.134 須要根據須要修改；

hdfs-site.xml 

cd /usr/local/hadoop/etc/hadoop/

vim hdfs-site.xml 

<configuration>
    <!—hdfs-site.xml-->
    <property>
        <name>dfs.name.dir</name>
        <value>/data0/hadoop/hdfs/name</value>
        <description>namenode上存儲hdfs名字空間元數據 </description> 
    </property>
    <property>
        <name>dfs.data.dir</name>
        <value>/data0/hadoop/hdfs/data</value>
        <description>datanode上數據塊的物理存儲位置</description>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
        <description>副本個數，配置默認是3,應小於datanode機器數量</description>
    </property>
</configuration>

數據目錄 不存在，記的建立

mkdir -p /data0/hadoop/hdfs/name

mkdir -p /data0/hadoop/hdfs/data

mapred-site.xml 

cd /usr/local/hadoop/etc/hadoop/

vim mapred-site.xml

<configuration>
    <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
    </property>
</configuration>

yarn-site.xml 

cd /usr/local/hadoop/etc/hadoop/

vim yarn-site.xml

<configuration>
    <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
    </property>
    <property>
            <name>yarn.resourcemanager.webapp.address</name>
            <value>localhost:8099</value>
    </property>
</configuration>

Hadoop啓動 

cd /usr/local/hadoop/

格式化namenode

./bin/hdfs namenode –format

start-all.sh 啓動

cd /usr/local/hadoop

./sbin/start-all.sh

分批啓動

• 啓動NameNode 和 DataNode 守護進程

./sbin/start-dfs.sh

啓動日誌

./sbin/start-dfs.sh
Starting namenodes on [localhost]
root@localhost's password: 
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-localhost.localdomain.out
root@localhost's password: 
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-localhost.localdomain.out
Starting secondary namenodes [0.0.0.0]
root@0.0.0.0's password: 
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-root-secondarynamenode-localhost.localdomain.out

• 啓動ResourceManager 和 NodeManager 守護進程

./sbin/start-yarn.sh

啓動日誌

./sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-root-resourcemanager-localhost.localdomain.out
root@localhost's password: 
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-localhost.localdomain.out

### 啓動驗證 

jps

6097 NodeManager
11044 Jps
7497 -- process information unavailable
8256 Worker
5999 ResourceManager
5122 SecondaryNameNode
8106 Master
4836 NameNode
4957 DataNode

centos7關閉防火牆

systemctl stop firewalld.service