Centos6.5 Hadoop徹底分佈集羣搭建詳解

#系統:centos6.5	64位
#JDK:1.7.0_79
#Hadoop:2.4.0 	64位
#注：（配置網絡，首先配置好一臺主機，另外的克隆主機便可）

1、安裝SSH

yum install openssh

2、關閉防火牆

chkconfig iptables off
sudo gedit /etc/selinux/config  # 改成 SELINUX=disabled

3、修改hosts文件

sudo gedit /etc/hosts

把裏面全部的內容使用#註釋掉，而後添加主機的ip和主機名

192.168.10.110		Master
192.168.10.111		Slave1
192.168.10.112		Slave2

4、設置免密碼登錄

ssh-keygen -t rsa		#一路空格

切換到ssh的目錄

cd /root/.ssh
cat id_rsa.pub >> authorized_keys

測試是否能夠免密碼登錄

ssh Master

5、配置JDK 把JDK 解壓到 /usr/java/ 目錄下 配置JDK的環境變量

sudo gedit /etc/profile
#在最後一行增長如下內容：
JAVA_HOME=/usr/java/jdk1.7.0_79
JRE_HOME=/usr/java/jdk1.7.0_79/jre
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
CLASSPATH=:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib

5、配置Hadoop 解壓Hadoop-2.4.0到 /usr/java 的目錄下 須要配置文件：core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml、hadoop-env.sh、yarn-env.sh、slaves。 一、core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://Master:9000</value>
	<description>默認HDFS的路徑</description>
        <final>true</final>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/java/hadoop-2.4.0/tmp</value>
	<description>臨時文件夾路徑</description>
    </property>
    <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
	<description>緩衝區的大小:io.file.buffer.size 默認4KB</description>
    </property>
    <property>
        <name>hadoop.proxyuser.hduser.hosts</name>
        <value>*</value>
    </property>
    <property>
        <name>hadoop.proxyuser.hduser.groups</name>
        <value>*</value>
    </property>
</configuration>

二、hdfs-site.xml

<configuration>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>Master:9001</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/usr/java/hadoop-2.4.0/dfs/namenode</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/usr/java/hadoop-2.4.0/dfs/datanode</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
</configuration>

三、mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
        <description>Execution framework set to Hadoop YARN.</description>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>Master:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>Master:19888</value>
    </property>
</configuration>

四、yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>Master:8032</value>
        <description>The address of the applications manager interface in the RM.</description>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>Master:8030</value>
        <description>The address of the scheduler interface,in order for the RM to obtain the resource from scheduler</description>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>Master:8031</value>
        <description>The address of the resource tracker interface for the nodeManagers</description>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>Master:8033</value>
        <description>The address for admin manager</description>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>Master:8088</value>
        <description>The address of the RM web application.</description>
    </property>

</configuration>

五、hadoop-env.sh （增長JDK的安裝目錄）

export JAVA_HOME=/usr/java/jdk1.7.0_79

六、yarn-env.sh （增長JDK的安裝目錄）

export JAVA_HOME=/usr/java/jdk1.7.0_79

七、slaves

Slave1
Slave2

6、配置Hadoop的環境變量

sudo gedit /etc/profile

 # set hadoop path
export HADOOP_HOME=/usr/hadoop
export PATH=$PATH :$HADOOP_HOME/bin

7、克隆兩臺虛擬機，配置好網絡。192.168.10.111 192.168.10.112 8、互相ping ip地址,看是否能夠ping通。互相 ssh ip地址，看是否能夠免密碼登錄。 9、回到Master的主機，切換到Hadoop的目錄下執行

[hadoop@Master hadoop]$ bin/hadoop namenode -format
[hadoop@Master hadoop]$ bin/start-all.sh

10、打開瀏覽器，輸入 192.168.10.110:50075 看是否有兩個節點。