hadoop-3.0.0集羣搭建

hadoop-3.0.0集羣搭建

• 下載hadoop包

wget -c http://ftp.jaist.ac.jp/pub/apache/hadoop/common/hadoop-3.0.0/hadoop-3.0.0.tar.gz

• 解壓

tar -zxvf hadoop-3.0.0.tar.gz -C /usr/java/

• 配置
  • 配置環境變量，打開vim /etc/profile文件

  export HADOOP_HOME=/usr/java/hadoop-3.0.0
  export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

  使之當即生效

  source /etc/profile

  • 配置/etc/hosts

  192.168.56.101 master
  192.168.56.102 slave1
  192.168.56.103 slave2

  • 關閉防火牆

  systemctl stop firewalld

  • 配置vim core-site.xml文件（etc/hadoop目錄下），新增以下配置

  <configuration>
    <property>
      <name>fs.defaultFS</name>
      <value>hdfs://master:9000</value>
    </property>
    <property>
      <name>hadoop.proxyuser.wujinlei.groups</name>
      <value>*</value>
    </property>
    <property>
      <name>hadoop.proxyuser.wujinlei.hosts</name>
      <value>*</value>
    </property>
  </configuration>

  • 配置vim hdfs-site.xml文件，新增以下配置。
    • master機器上開放端口9870，供外部訪問web頁面（NameNode HTTP UI），查看集羣狀況。
    • slave機器上開放端口9864，供外部訪問web頁面（DataNode HTTP UI）。

  <configuration>
    <property>
      <name>dfs.replication</name>
      <value>2</value>
    </property>
    <property>
      <name>dfs.namenode.name.dir</name>
      <value>file:/home/wujinlei/hadoop/dfs/name</value>
    </property>
    <property>
      <name>dfs.datanode.data.dir</name>
      <value>file:/home/wujinlei/hadoop/dfs/data</value>
    </property>
    <property>
      <name>dfs.namenode.http-address</name>
      <value>master:9870</value>
    </property>
    <property>
      <name>dfs.datanode.http.address</name>
      <value>master:9864</value>
    </property>
  </configuration>

  • 配置vim yarn-site.xml，新增以下配置，master機器上開放端口8088，供外部訪問web頁面，查看集羣任務調度狀況

  <configuration>
    <property>
      <name>yarn.resourcemanager.hostname</name>
      <value>master</value>
    </property>
    <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value>
    </property>
    <property>
      <name>yarn.resourcemanager.webapp.address</name>
      <value>master:8088</value>
    </property>
    <property>
      <name>yarn.application.classpath</name>
      <value>
          /usr/java/hadoop-3.0.0/etc/hadoop,
          /usr/java/hadoop-3.0.0/share/hadoop/common/lib/*,
          /usr/java/hadoop-3.0.0/share/hadoop/common/*,
          /usr/java/hadoop-3.0.0/share/hadoop/hdfs,
          /usr/java/hadoop-3.0.0/share/hadoop/hdfs/lib/*,
          /usr/java/hadoop-3.0.0/share/hadoop/hdfs/*,
          /usr/java/hadoop-3.0.0/share/hadoop/mapreduce/*,
          /usr/java/hadoop-3.0.0/share/hadoop/yarn,
          /usr/java/hadoop-3.0.0/share/hadoop/yarn/lib/*,
          /usr/java/hadoop-3.0.0/share/hadoop/yarn/*,
          /usr/java/jdk1.8.0_45/lib/tools.jar
      </value>
    </property>
    <property>
      <name>yarn.log-aggregation-enable</name>
      <value>true</value>
    </property>
  </configuration>

  • 配置vim mapred-site.xml，新增以下配置

  <configuration>
    <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
    </property>
    <property>
      <name>mapreduce.jobhistory.address</name>
      <value>0.0.0.0:10020</value>
    </property>
    <property>
      <name>mapreduce.jobhistory.webapp.address</name>
      <value>0.0.0.0:19888</value>
    </property>
  </configuration>

  • 在etc/hadoop下編輯works文件，內容以下

  slave1
  slave2

• 啓動集羣
  • 將上述配置好的hadoop文件複製到另外兩個節點slave1和slave2
  • 啓動前先格式化，命令hdfs namenode -format
  • 單獨啓動
    • 啓動dfs，命令start-dfs.sh
    • 啓動yarn，命令start-yarn.sh
  • 所有啓動
    • 命令start-all.sh
• 訪問web頁面
• http://192.168.56.101:8088
• http://192.168.56.101:9870
• 啓動歷史服務

mapred historyserver

此服務用來訪問歷史任務詳情