Hadoop集羣部署與啓動，Yarn模式要考慮Container內存資源分配

Hadoop集羣的部署

開始安裝Hadoop以前，爲了讓 Master 節點能夠無密碼 SSH 登錄到各個 Slave 節點上，因此須要配置 SSH 無密碼登錄

安裝版本： hadoop-2.8.3.tar.gz

mkdir /usr/local/hadoop
tar zxvf hadoop-2.8.3.tar.gz -C /usr/local/hadoop

修改域名與IP的對應關係(hadoop2和hadoop3一樣也須要修改hosts文件）

vi /etc/hosts
10.2.15.176 hadoop1
10.2.15.177 hadoop2
10.2.15.170 hadoop3

配置環境變量(hadoop2和hadoop3一樣也須要修改hosts文件）

vi /etc/profile
export FLINK_HOME=/usr/local/hadoop/hadoop-2.8.3
export PATH=$FLINK_HOME/bin:$PATH
source /etc/profile

先建好稍後須要用到的文件夾

mkdir /usr/local/hadoop
mkdir /usr/local/hadoop/tmp
mkdir /usr/local/hadoop/var
mkdir /usr/local/hadoop/dfs
mkdir /usr/local/hadoop/dfs/name
mkdir /usr/local/hadoop/dfs/data

修改core-site.xml文件

vi /usr/local/hadoop/hadoop-2.8.3/etc/hadoop/core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
<description>Abase for other temporary directories.</description>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop1:9000</value>
</property>
</configuration>

修改mapred-site.xml文件

cp /usr/local/hadoop/hadoop-2.8.3/etc/hadoop/mapred-site.xml.template /usr/local/hadoop/hadoop-2.8.3/etc/hadoop/mapred-site.xml  
vi /usr/local/hadoop/hadoop-2.8.3/etc/hadoop/mapred-site.xml  
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop1:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop1:19888</value>
</property>

修改hdfs-site.xml文件

vi /usr/local/hadoop/hadoop-2.8.3/etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/usr/local/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/usr/local/hadoop/dfs/data</value>
</property>
<property>
    <name>dfs.replication</name>
    <value>2</value>
     <description>HDFS 的數據塊的副本存儲個數, 默認是3</description>
  </property>
  <property>
      <name>dfs.permissions</name>
      <value>false</value>
      <description>need not permissions</description>
</property>
</configuration>

修改hdfs-site.xml文件

vi /usr/local/hadoop/hadoop-2.8.3/etc/hadoop/yarn-site.xml
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop1</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoop1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hadoop1:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>hadoop1:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>hadoop1:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.am.max-attempts</name>
  <value>4</value>
  <description>
    The maximum number of application master execution attempts.
  </description>
</property>

若是是以Flink on Yarn方式啓動的，由於Hadoop Yarn是一個資源調度器，因此咱們應該考慮好每一個Conatiner被分配到的內存資源，因此須要在文件hdfs-site.xml中配置好 yarn.nodemanager.resource.memory-mb, yarn.scheduler.minimum-allocation-mb, yarn.scheduler.maximum-allocation-mb, yarn.app.mapreduce.am.resource.mb和yarn.app.mapreduce.am.command-opts，否則會發生內存不足，致使Application啓動失敗。

Current usage: 303.2 MB of 1 GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container.

vi /usr/local/hadoop/hadoop-2.8.3/etc/hadoop/yarn-site.xml

<property>
  <name>yarn.nodemanager.vmem-check-enbaled</name>
  <value>false</value>
</property>
<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>106496</value>
</property>
<property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>2048</value>
</property>
<property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>106496</value>
</property>
<property>
    <name>yarn.app.mapreduce.am.resource.mb</name>
    <value>4096</value>
</property>
<property>
    <name>yarn.app.mapreduce.am.command-opts</name>
    <value>-Xmx3276m</value>
</property>

修改 hadoop-env.sh , mapred-env.sh 和 yarn-env.sh

vi /usr/local/hadoop/hadoop-2.8.3/etc/hadoop/hadoop-env.sh
export JAVA_HOME="/usr/local/jdk/jdk1.8.0_251"
vi /usr/local/hadoop/hadoop-2.8.3/etc/hadoop/mapred-env.sh
export JAVA_HOME="/usr/local/jdk/jdk1.8.0_251"
vi /usr/local/hadoop/hadoop-2.8.3/etc/hadoop/yarn-env.sh
export JAVA_HOME="/usr/local/jdk/jdk1.8.0_251"

把/hadoop發送給另外兩臺服務器

scp -r /usr/local/hadoop hadoop2:/usr/local
scp -r /usr/local/hadoop hadoop3:/usr/local

啓動Hadoop集羣

初始化HDFS系統

/usr/local/hadoop/hadoop-2.8.3/bin/hdfs namenode -format

開啓 NameNode 和 DataNode 守護進程

/usr/local/hadoop/hadoop-2.8.3/sbin/start-all.sh

在瀏覽器中輸入 http://hadoop1:50070，可查看相關信息

運行wordcount demo

bin/hdfs dfs -mkdir /input
bin/hdfs dfs -ls /
bin/hdfs dfs -put /usr/local/hadoop/tmp/input_hadoop_demo_test.txt /input/
bin/hdfs dfs -ls /input/
bin/hadoop jar /usr/local/hadoop/hadoop-2.8.5/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar wordcount /input/input_hadoop_demo_test.txt /output
bin/hdfs dfs -ls /output
bin/hdfs dfs -cat /output/part-r-00000