Hadoop3.2.0集羣（4節點-無HA）

時間  2019-11-11
標籤 hadoop3.2.0 hadoop 集羣 節點 欄目 Hadoop 简体版
原文   原文鏈接

1.準備環境

1.1配置dns

# cat /etc/hosts
    172.27.133.60  hadoop-01
  172.27.133.61  hadoop-02
  172.27.133.62  hadoop-03
  172.27.133.63  hadoop-04

1.2配置免密登錄

# ssh-keygen
# ssh-copy-id root@hadoop-02/03/04

1.3關閉防火牆

# cat /etc/selinux/config
    SELINUX=disabled
# systemctl stop firewalld
# systemctl disable firewalld

1.4配置Java環境,Hadoop環境

# tar -xf /data/software/jdk-8u171-linux-x64.tar.gz -C /usr/local/java
# tar -xf /data/software/hadoop-3.2.0.tar.gz -C /data/hadoop
# cat /etc/profile
  export HADOOP_HOME=/data/hadoop/hadoop-3.2.0
  export JAVA_HOME=/usr/local/java/jdk1.8.0_171
  export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
  export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
  export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
# source /etc/profile
# java -version
  java version "1.8.0_171"
  Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
  Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

2.配置Hadoop

2.1配置Hadoop環境腳本文件中的JAVA_HOME參數

# cd /data/hadoop/hadoop-3.2.0/etc/hadoop
# vim hadoop-env.sh 和 mapred-env.sh 和 yarn-env.sh，向腳本添加JAVA_HOME
    export JAVA_HOME=/usr/local/java/jdk1.8.0_171
# hadoop version
  Hadoop 3.2.0
  Source code repository https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf
  Compiled by sunilg on 2019-01-08T06:08Z
  Compiled with protoc 2.5.0
  From source with checksum d3f0795ed0d9dc378e2c785d3668f39
  This command was run using /data/hadoop/hadoop-3.2.0/share/hadoop/common/hadoop-common-3.2.0.jar

2.2修改Hadoop配置文件

在hadoop-3.2.0/etc/hadoop目錄下，修改core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml、workers文件，具體參數按照實際狀況修改。java

2.2.1修改core-site.xml

（文件系統）node

<configuration>
  <property>
      <!-- 配置hdfs地址 -->
      <name>fs.defaultFS</name>
      <value>hdfs://hadoop-01:9000</value>
  </property>
  <property>
      <!-- 保存臨時文件目錄，需先在/data/hadoop下建立tmp目錄 -->
      <name>hadoop.tmp.dir</name>
     <value>/data/hadoop/tmp</value>
 </property>
</configuration>
# mkdir /data/hadoop/tmp
2.2.2修改hdfs-site.xml

(副本數)linux

<configuration>
      <property>
         <!-- 主節點地址 -->
          <name>dfs.namenode.http-address</name>
          <value>hadoop-01:50070</value>
      </property>
      <property>
          <name>dfs.secondary.http.address</name>
          <value>hadoop-02:50070</value>
      </property>
      <property>
          <name>dfs.namenode.name.dir</name>
          <value>file:/data/hadoop/dfs/name</value>
     </property>
     <property>
         <name>dfs.datanode.data.dir</name>
         <value>file:/data/hadoop/dfs/data</value>
     </property>
     <property>
        <!-- 備份數爲默認值2 -->
        <name>dfs.replication</name>
        <value>2</value>
     </property>
     <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
     </property>

     <property>
        <name>dfs.permissions</name>
        <value>false</value>
        <description>配置爲false後，能夠容許不要檢查權限就生成dfs上的文件，方便卻是方便了，可是你須要防止誤刪除.</description>
      </property>
</configuration>
2.2.3修改mapred-site.xml

（資源調度框架）git

<configuration>
      <property>
          <name>mapreduce.framework.name</name>
          <value>yarn</value>
      </property>
      <property>
          <name>mapreduce.jobhistory.address</name>
          <value>hadoop-01:10020</value>
      </property>
      <property>
         <name>mapreduce.jobhistory.webapp.address</name>
         <value>hadoop-01:19888</value>
      </property>
　　   <property>
　　 　　<name>mapreduce.application.classpath</name>
      <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
　　   </property>
</configuration>
2.2.4修改yarn-site.xml
<configuration>

<!-- Site specific YARN configuration properties -->
     <property>
         <name>yarn.resourcemanager.hostname</name>
         <value>hadoop-01</value>
     </property>
     <property>
         <name>yarn.nodemanager.aux-services</name>
         <value>mapreduce_shuffle</value>
     </property>
     <property>
         <name>yarn.resourcemanager.webapp.address</name>
         <value>172.27.133.60:8088</value>
         <description>配置外網只須要替換外網ip爲真實ip，不然默認爲 localhost:8088</description>
     </property>
     <property>
         <name>yarn.scheduler.maximum-allocation-mb</name>
         <value>2048</value>
         <description>每一個節點可用內存,單位MB,默認8182MB</description>
     </property>
     <property>
         <name>yarn.nodemanager.vmem-check-enabled</name>
         <value>false</value>
         <description>忽略虛擬內存的檢查，若是你是安裝在虛擬機上，這個配置頗有用，配上去以後後續操做不容易出問題。</description>
     </property>

     <property>
         <name>yarn.nodemanager.env-whitelist</name>
         <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
     </property>
</configuration>
2.2.5修改workers

添加從節點地址，IP或者hostnamegithub

hadoop-02
hadoop-03
hadoop-04

2.3將配置好的文件夾拷貝到其餘從節點

# scp -r /data/hadoop root@hadoop-02:/data/hadoop
# scp -r /data/hadoop root@hadoop-03:/data/hadoop
# scp -r /data/hadoop root@hadoop-04:/data/hadoop

2.4配置啓動腳本Yarn權限

2.4.1添加hdfs權限
在第二行空白位置添加hdfs權限
# cat sbin/start-dfs.sh
# cat sbin/stop-dfs.sh

  #!/usr/bin/env bash
  HDFS_DATANODE_USER=root
  HDFS_DATANODE_SECURE_USER=hdfs
  HDFS_NAMENODE_USER=root
  HDFS_SECONDARYNAMENODE_USER=root
  ···
2.4.2添加Yarn權限
在第二行空白位置添加Yarn權限
# cat sbin/start-yarn.sh
# cat sbin/stop-yarn.sh

  #!/usr/bin/env bash
  YARN_RESOURCEMANAGER_USER=root
  HDFS_DATANODE_SECURE_USER=yarn
  YARN_NODEMANAGER_USER=root
  ···

若是不添加權限，在啓動時候會報錯web

ERROR: Attempting to launch hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting launch.
Starting datanodes
ERROR: Attempting to launch hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting launch.
Starting secondary namenodes [localhost.localdomain]
ERROR: Attempting to launch hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting launch.

3.初始化並啓動

3.1格式化hdfs

# bin/hdfs namenode -format

3.2啓動

3.2.1一步啓動
# sbin/start-all.sh
3.2.2分佈啓動
# sbin/start-dfs.sh
# sbin/start-yarn.sh

4.驗證

4.1列出進程

  1. 主節點hadoop-01shell

    # jps
32069 SecondaryNameNode
2405 NameNode
2965 ResourceManager
3645 Jps

  2. 從節點hadoop-02apache

    # jps
25616 NodeManager
25377 DataNode
25508 SecondaryNameNode
25945 Jps

  3. 從節點hadoop-03vim

    # jps
14946 NodeManager
15207 Jps
14809 DataNode

  4. 從節點hadoop-04bash

    # jps
14433 Jps
14034 DataNode
14171 NodeManager4

4.2訪問頁面

http://hadoop-01:8088打開ResourceManager頁面

http://hadoop-01:50070打開Hadoop Namenode頁面

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程