Centos 6.5 hadoop 2.2.0 全分佈式安裝

hadoop 2.2.0 cluster setup

環境：

操做系統:Centos 6.5
jdk:jdk1.7.0_51
hadoop版本:2.2.0

hostname    ip

master 192.168.1.180

slave1 192.168.1.181

slave2 192.168.1.182

slave3 192.168.1.183

1、前期系統環境配置

設置主機名
臨時生效修改主機名，重啓失效

[lxj@master ~]$ hostnamemaster

永久生效修改主機名,需重啓.

[lxj@master ~]$ vim/etc/sysconfig/network

 

NETWORKING=yes

HOSTNAME=master

2.設置ip

[lxj@master ~]$ vim/etc/sysconfig/network-scripts/ifcfg-eth0

 

DEVICE=eth0

TYPE=Ethernet

UUID=390e4922-0d95-4c34-9dde-897fb8acef0f

ONBOOT=yes

NM_CONTROLLED=yes

BOOTPROTO=none

HWADDR=08:00:27:1C:57:24

IPADDR=192.168.1.180

PREFIX=24

GATEWAY=192.168.1.1

DNS1=192.168.1.1

DOMAIN=114.114.114.114

DEFROUTE=yes

IPV4_FAILURE_FATAL=yes

IPV6INIT=no

NAME="Systemeth0"

LAST_CONNECT=1393996666

3.host 映射

[lxj@master ~]$ vim/etc/hosts

 

127.0.0.1   localhost

192.168.1.180 master

192.168.1.181 slave1

192.168.1.182 slave2

192.168.1.183 slave3

其餘節點hosts改爲這樣。

4.關閉防火牆
開機啓動關閉

[lxj@master ~]$ sudoservice iptables stop

[lxj@master ~]$ sudochkconfig iptables off

5.ssh 無密碼登陸，master 須要啓動slave節點的相關進程

[lxj@master ~]$ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

[lxj@master ~]$ cat~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

測試是否能無密碼登陸了

[lxj@master ~]$ sshlocalhost

若是以上操做完成後 還須要輸入密碼 請作一下操做
/home/用戶名/.ssh 目錄權限 700
/home/用戶名/.ssh/authorized_keys 文件權限 600

[lxj@master ~]$ chmod 700/home/lxj/.ssh

[lxj@master ~]$ chmod 600/home/lxj/.ssh/authorized_keys

6.拷貝公鑰到其餘slave節點
因爲master主機公鑰已寫入authorized_keys受權文件，因此只要拷貝受權文件到其餘slave節點便可

注意節點.ssh目錄和authorized_keys文件權限也分別要是700,600。

[lxj@master ~]$ scp/home/lxj/.ssh/authorized_keys lxj@slave1:/home/lxj/.ssh/

[lxj@master ~]$ scp/home/lxj/.ssh/authorized_keys lxj@slave2:/home/lxj/.ssh/

[lxj@master ~]$ scp/home/lxj/.ssh/authorized_keys lxj@slave3:/home/lxj/.ssh/

測試是否能夠無密碼登陸slave節點了

[lxj@master ~]$ sshslave1

[lxj@master ~]$ exit

[lxj@master ~]$ sshslave2

[lxj@master ~]$ exit

[lxj@master ~]$ sshslave3

能夠的話，這樣master無密碼登陸其餘slave節點就配配置好了！

2、hadoop 配置文件

先設置下環境變量

JAVA_HOME=/home/lxj/jdk1.7.0_51

CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

 

HADOOP_HOME=/home/lxj/hadoop-2.2.0

HADOOP_MAPRED_HOME=$HADOOP_HOME

HADOOP_COMMON_HOME=$HADOOP_HOME

HADOOP_HDFS_HOME=$HADOOP_HOME

YARN_HOME=$HADOOP_HOME

HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

 

PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

 

export JAVA_HOME CLASSPATHHADOOP_HOME HADOOP_MAPRED_HOME HADOOP_COMMON_HOME HADOOP_HDFS_HOME YARN_HOMEHADOOP_CONF_DIR YARN_CONF_DIR  PATH

主要需修改的配置文件
咱們先在master機器上修改這些配置文件，而後傳到各個節點hadoop 目錄在~/hadoop-2.2.0

hadoop-env.sh、core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-env.sh、yarn-site.xml、masters、slaves

以上配置文件，若是沒有則本身新建

節點信息
NameNode: master,slave1
SecondaryNameNode: master,slave1
DataNode:slave1,slave2,slave3

3、master 機器配置

1.hadoop-env.sh 修改JAVA_HOME

[lxj@master ~]$  cd ~/hadoop-2.2.0/etc/hadoop

[lxj@master hadoop]$ vimhadoop-env.sh

# The onlyrequired environment variable is JAVA_HOME. All others are

#optional.  When running a distributedconfiguration it is best to

# setJAVA_HOME in this file, so that it is correctly defined on

# remotenodes.

 

# The javaimplementation to use.

export JAVA_HOME=/home/lxj/jdk-1.7.0_15

2.core-site.xml

<?xmlversion="1.0" encoding="UTF-8"?>

<?xml-stylesheettype="text/xsl" href="configuration.xsl"?>

 

<configuration>

    <property>

      <name>fs.default.name</name>

       <value>hdfs://master:9000</value>

    </property>

    <property>

       <name>hadoop.tmp.dir</name>

       <value>/home/lxj/hadoop/tmp</value>

    </property>

</configuration>

3.hdfs-site.xml

<?xmlversion="1.0" encoding="UTF-8"?>

<?xml-stylesheettype="text/xsl" href="configuration.xsl"?>

<!--Put site-specific property overrides in this file. -->

 

<configuration>

    <property>

        <name>dfs.namenode.secondary.http-address</name>

        <value>master:50090</value>

    </property>

    <property>

        <name>fs.checkpoint.period</name>

        <value>600</value>

    </property>

    <property>

        <name>fs.checkpoint.size</name>

        <value>67108894</value>

    </property>

    <property>

        <name>dfs.namenode.name.dir</name>

        <value>file:/home/lxj/hadoop/hdfs/namenode</value>

    </property>

    <property>

        <name>dfs.datanode.data.dir</name>

        <value>file:/home/lxj/hadoop/hdfs/datanode</value>

    </property>

    <property>

        <name>dfs.permissions</name>

        <value>false</value>

    </property>

    <property>

        <name>dfs.replication</name>

        <value>3</value>

    </property>

    <property>

        <name>dfs.webhdfs.enabled</name>

        <value>true</value>

    </property>

</configuration>

4.mapred-site.xml

<?xmlversion="1.0"?>

<?xml-stylesheettype="text/xsl" href="configuration.xsl"?>

 

<configuration>

    <property>

        <name>mapreduce.framework.name</name>

        <value>yarn</value>

    </property>

    <property>

        <name>mapred.job.tracker</name>

        <value>master:9001</value>

    </property>

    <property>

        <name>mapreduce.jobhistory.address</name>

        <value>master:10020</value>

    </property>

    <property>

        <name>mapreduce.jobhistory.webapp.address</name>

        <value>master:19888</value>

    </property>

</configuration>

5.yarn-env.sh 修改JAVA_HOME

# someJava parameters

export JAVA_HOME=/home/lxj/jdk1.7.0_51

if["$JAVA_HOME" !=""];then

  #echo "run java in$JAVA_HOME"

  JAVA_HOME=$JAVA_HOME

fi

6.yarn-site.xml

<?xmlversion="1.0"?>

 

<configuration>

        <property>

            <name>yarn.nodemanager.aux-services</name>

            <value>mapreduce_shuffle</value>

        </property>

        <property>

            <name>yarn.nodemanager.aux-servicex.mapreduce.shuffle.class</name>

            <value>org.apache.hadoop.mapred.ShuffleHandler</value>

        </property>

        <property>

            <name>yarn.resourcemanager.address</name>

            <value>master:8032</value>

        </property>

        <property>

            <name>yarn.resourcemanager.scheduler.address</name>

            <value>master:8030</value>

        </property>

        <property>

            <name>yarn.resourcemanager.resource-tracker.address</name>

            <value>master:8031</value>

        </property>

        <property>

            <name>yarn.resourcemanager.admin.address</name>

            <value>master:8033</value>

        </property>

        <property>

            <name>yarn.resourcemanager.webapp.address</name>

            <value>master:8088</value>

        </property>

</configuration>

7.masters 添加master主機 多個換行寫

master

8.slaves 添加slave節點 當加入master時，master即看成NameNode又看成DataNode節點

slave1 

slave2

slave3

4、slave 節點配置

其餘節點各拷貝一份。

[lxj@master ~]$ scp -r~/hadoop-2.2.0 lxj@slave1:/home/lxj/

[lxj@master ~]$ scp -r~/hadoop-2.2.0 lxj@slave2:/home/lxj/

[lxj@master ~]$ scp -r~/hadoop-2.2.0 lxj@slave3:/home/lxj/

slave節點的hadoop路徑需和master的路徑同樣，固然系統環境須要保持一致。
只要將master主機的配置好的文件拷貝到各個節點便可。

5、 hadoop 啓動

1.格式化一個新的分佈式文件系統
須要在master和slave1都要執行

[lxj@master ~]$ hdfsnamenode -format

2.啓動hdfs和yarn 只需在master執行

[lxj@master ~]$start-dfs.sh

[lxj@master ~]$start-yarn.sh

[lxj@master ~]$mr-jobhistory-daemon.sh start historyserver

或者

[lxj@mater ~]$start-all.sh

start-all.sh已廢棄，不推薦使用！
3.測試
master 進程

[lxj@master ~]$ jps

11597 JobHistoryServer

4529 Jps

4460 ResourceManager

4293 NameNode

4377 SecondaryNameNode

slaves進程

[lxj@master ~]$ jps

1941 DataNode

2176 Jps

2046 NodeManager

建立輸入目錄和文件

[lxj@master ~]$ hdfs dfs-mkdir /input

[lxj@master ~]$ vim./test

hello hadoop test

拷貝文件到hdfs文系統，運行wordcount程序計算輸入文件的單詞出現次數

[lxj@master ~]$ hdfs dfs-copyFromLocal ./test /input

[lxj@master ~]$ hadoopjar/home/lxj/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jarwordcount /input/ /out

[lxj@master ~]$ hdfs dfs-cat /out/*

hadoop  1

hello   1

test    1

搞了下ha federationqjm 配置，沒起來，仍是從基礎配置開始吧，之後在擴展。

其餘主題:eclipse安裝hadoop-2.2.0插件