hadoop2.2.0部署

時間  2019-11-08
標籤 hadoop2.2.0 hadoop 部署 欄目 Hadoop 简体版
原文   原文鏈接
hadoop2.2.0部署

下載安裝包node

hadoop-2.2.0-cdh5.0.0-beta-1.tar.gzweb

解壓apache

tar zxvf hadoop-2.2.0-cdh5.0.0-beta-1.tar.gzapp

ln -s /opt/hadoop-2.2.0-cdh5.0.0-beta-1 ~/hadoopssh

全部節點都解壓webapp

ssh雙向認證oop

配置全部機器hosts測試

vi /etc/hostsspa

10.10.1.1 hadoop1orm

10.10.1.2 hadoop2

10.10.1.3 hadoop3

 

使用hostname 設置主機名

修改/etc/sysconfig/network 中的內容改成相應的主機名

 

在.ssh目錄生成id_rsa.pub ，添加到本機~/.ssh/authorized_keys文件裏面

ssh-keygen -q -t rsa

cat ~/.ssh/id_rsa.pub >>~/.ssh/authorized_keys

本機登入驗證：ssh localhost

複製authorized_keys到其餘機器上

scp ~/.ssh/id_rsa.pub hadoop3:~/

cat ~/id_rsa.pub >> ~/.ssh/authorized_keys 

chmod 600 ~/.ssh/authorized_keys

chmod 700 ~/.ssh

其餘機器重複以上步驟

修改環境參數

vi /etc/profile

增長如下內容：

export JAVA_HOME=/opt/jdk1.7.0_51

export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:/lib/dt.jar

export PATH=$PATH:$JAVA_HOME/bin

export HADOOP_DEV_HOME=/home/hadoop/hadoop

export PATH=$PATH:$HADOOP_DEV_HOME/bin

 

修改ulimit

vi /etc/security/limits.d/90-nproc.conf 

* soft nproc 502400

 

執行環境參數 

source /etc/profile

修改hadoop配置

1.編輯同步腳本:vi hadoop/cp2slaves.sh

BASE_PATH=`dirname $0`

cd $BASE_PATH

echo `/bin/pwd`

scp etc/hadoop/* hadoop2:~/hadoop/etc/hadoop/

scp etc/hadoop/* hadoop3:~/hadoop/etc/hadoop/

 

2.配置hadoop/etc/hadoop/hadoop-env.sh

JAVA_HOME修改成：

export JAVA_HOME=/opt/jdk

HADOOP_PID_DIR修改成：

export HADOOP_PID_DIR=${HADOOP_LOG_DIR}

 

3.建立hadoop工做目錄

mkdir -p /home/hadoop/tmp

mkdir -p /home/hadoop/hdfs/name

mkdir -p /home/hadoop/hdfs/data

mkdir -p /home/hadoop/hadoop-yarn

4.配置hadoop/etc/hadoop/core-site.xml

<property>

<name>hadoop.tmp.dir</name>

<value>/home/hadoop/tmp</value>

<description>A base for other temporary directories.</description>

</property>

<property>

<name>fs.default.name</name>

<value>hdfs://hadoop1:9000</value>

</property>

<property>

<name>io.file.buffer.size</name>

<value>131072</value>

</property>

<property>

<name>hadoop.proxyuser.root.hosts</name>

<value>hadoop1</value>

</property>

<property>

<name>hadoop.proxyuser.root.groups</name>

<value>*</value>

</property>

5.配置hadoop/etc/hadoop/mapred-site.xml

mv hadoop/etc/hadoop/mapred-site.xml.template hadoop/etc/hadoop/mapred-site.xml

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

<property>

<name>mapreduce.jobhistory.address</name>

<value>hadoop1:10020</value>

</property>

<property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>hadoop1:19888</value>

</property>

<property>

    <name>mapreduce.jobhistory.intermediate-done-dir</name>

    <value>/mr-history/tmp</value>

</property>

<property>

    <name>mapreduce.jobhistory.done-dir</name>

    <value>/mr-history/done</value>

</property>

6.配置hadoop/etc/hadoop/hdfs-site.xml

<property>

<name>dfs.name.dir</name>

<value>file:/home/hadoop/hdfs/name</value>

<description> </description>

</property>

<property>

<name>dfs.namenode.secondary.http-address</name>

<value>hadoop1:9001</value>

</property>

<property>

<name>dfs.data.dir</name>

<value>file:/home/hadoop/hdfs/data</value>

</property>

<property>

<name>dfs.http.address</name>

<value>hadoop1:9002</value>

</property>

<property>

<name>dfs.replication</name>

<value>3</value>

</property>

<property>

<name>dfs.permissions</name>

<value>false</value>

</property>

7.編輯hadoop/etc/hadoop/masters 

haoop1

8.編輯hadoop/etc/hadoop/slaves

hadoop1

hadoop2

hadoop3

9.編輯hadoop/etc/hadoop/yarn-site.xml 

        <property>

                <name>yarn.resourcemanager.address</name>

                <value>hadoop1:8032</value>

        </property>

        <property>

                <description>The address of the scheduler interface.</description>

                <name>yarn.resourcemanager.scheduler.address</name>

                <value>hadoop1:8030</value>

        </property>

        <property>

                <description>The address of the RM web application.</description>

                <name>yarn.resourcemanager.webapp.address</name>

                <value>hadoop1:8088</value>

        </property>

        <property>

                <name>yarn.resourcemanager.resource-tracker.address</name>

                <value>hadoop1:8031</value>

        </property>

        <property>

                <description>The address of the RM admin interface.</description>

                <name>yarn.resourcemanager.admin.address</name>

                <value>hadoop1:8033</value>

        </property>

        <property>

                <description>The hostname of the NM.</description>

                <name>yarn.nodemanager.hostname</name>

                <value>0.0.0.0</value>

        </property>

        <property>

                <description>The address of the container manager in the NM.

                </description>

                <name>yarn.nodemanager.address</name>

                <value>${yarn.nodemanager.hostname}:0</value>

        </property>

        <property>

                <name>yarn.nodemanager.aux-services</name>

                <value>mapreduce_shuffle</value>

        </property>

        <property>

                <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

                <value>org.apache.hadoop.mapred.ShuffleHandler</value>

        </property>

        <property>

                <name>yarn.log-aggregation-enable</name>

                <value>true</value>

        </property>

        <property>

                <description>List of directories to store localized files in.

                </description>

                <name>yarn.nodemanager.local-dirs</name>

                <value>/home/hadoop/hadoop-yarn/cache/${user.name}/nm-local-dir</value>

        </property>

        <property>

                <description>Where to store container logs.</description>

                <name>yarn.nodemanager.log-dirs</name>

                <value>/home/hadoop/hadoop-yarn/containers</value>

        </property>

        <property>

                <name>yarn.nodemanager.remote-app-log-dir</name>

                <value>/hadoop/hadoop-yarn/apps</value>

        </property>

        <property>

                <name>yarn.app.mapreduce.am.staging-dir</name>

                <value>/hadoop/staging</value>

        </property>

 

10.同步配置文件

sh hadoop/cp2slaves.sh

格式化namenode

hadoop/bin/hdfs namenode -format

啓動

 
hadoop/sbin/start-all.sh

測試驗證

 
http://hadoop1:8088
hadoop/bin/hadoop fs -df -h
hadoop/bin/hadoop jar hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0-cdh5.0.0-beta-1.jar pi 5 10

中止

 
hadoop/sbin/stop-all.sh
相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程