hadoop 分佈式集羣安裝

時間 2019-12-07

原文原文鏈接

centos7 安裝hadoop 2.7 - 準備工做

三臺機器（內存大於2G）分別寫hosts、設定hostname

172.7.15.113 master
 
172.7.15.114 slave1
 
172.7.15.115 slave2
 
關閉selinux
 
關閉firewalld
 
systemctl disable firewalld
 
systemctl stop firewalld
 
yum install -y iptables-services
 
systemctl enable iptables
 
systemctl start iptables
 
service iptables save

centos7 安裝hadoop 2.7 - 密鑰登錄

master能夠經過密鑰登錄本機和兩臺slave

master上生成密鑰對：

ssh-keygen 一直回車

複製~/.ssh/id_rsa.pub 內容到本機和兩臺slave的 ~/.ssh/authorized_keys

設置本機和兩臺slave機器上的~/.ssh/authorized_keys文件權限爲600

chmod 600 ~/.ssh/authorized_keys

在master上

ssh master

ssh slave1

ssh slave2

能夠直接登錄

2.1 安裝hadoop-安裝jdk

hadoop2.7 須要安裝jdk1.7版本

下載地址

http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html

解壓壓縮包 tar zxf jdk1.7.0_79.tar.gz

mv jdk1.7.0_79 /usr/local/

編寫環境變量配置 vim /etc/profile.d/java.sh 寫入

export JAVA_HOME=/usr/local/jdk1.7.0_79

export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export PATH=$PATH:$JAVA_HOME/bin

source /etc/profile.d/java.sh

java -version 查看是否生效

slave1 和 slave2 重複上面的操做

如下操做在master上執行

下載地址 http://hadoop.apache.org/releases.html 鏡像站 http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.1/

下載2.7.1 binary版本 wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.1/hadoop-2.7.1.tar.gz

其餘下載源參考 http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.15.1.tar.gz

解壓 tar zxf hadoop-2.7.1.tar.gz
 
mv hadoop-2.7.1 /usr/local/hadoop
 
cd /usr/local/hadoop
 
mkdir tmp dfs dfs/data dfs/name
 
把/usr/local/hadoop 目錄分別拷貝至兩個slave上  (slave1,slave2 上都須要安裝 rsync)
 
rsync -av /usr/local/hadoop  slave1:/usr/local/
 
rsync -av /usr/local/hadoop  slave2:/usr/local/

2.3 安裝hadoop-配置hadoop

master上

vim /usr/local/hadoop/etc/hadoop/core-site.xml
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://172.7.15.113:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/usr/local/hadoop/tmp</value>
    </property>
    <property>
        <name>io.file.buffer.size</name>
        <value>131702</value>
    </property>
</configuration>

master上

  vim  /usr/local/hadoop/etc/hadoop/hdfs-site.xml
<configuration>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/usr/local/hadoop/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/usr/local/hadoop/dfs/data</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>172.7.15.113:9001</value>
    </property>
    <property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
    </property>
</configuration>

master上

vim  /usr/local/hadoop/etc/hadoop/mapred-site.xml
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>172.7.15.113:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>172.7.15.113:19888</value>
    </property>
</configuration>

master上

vim  /usr/local/hadoop/etc/hadoop/yarn-site.xml
<configuration>
<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>172.7.15.113:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>172.7.15.113:8030</value>
    </property>
 
<property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>172.7.15.113:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>172.7.15.113:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>172.7.15.113:8088</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>768</value>
    </property>
</configuration>

如下在master上操做

cd  /usr/local/hadoop/etc/hadoop
vi  hadoop-env.sh  //更改JAVA_HOME
export JAVA_HOME=/usr/local/jdk1.7.0_79
vi yarn-env.sh  //更改JAVA_HOME
export JAVA_HOME=/usr/local/jdk1.7.0_79
vi  slaves //改成以下
172.7.15.114
172.7.15.115

將master上的etc目錄同步至兩個slave

rsync -av /usr/local/hadoop/etc/ slave1:/usr/local/hadoop/etc/
rsync -av /usr/local/hadoop/etc/ slave2:/usr/local/hadoop/etc/

centos7 安裝hadoop 2.7 - 啓動hadoop

在master上操做便可，兩個slave會自動啓動

初始化
/usr/local/hadoop/bin/hdfs namenode -format
啓動服務
/usr/local/hadoop/sbin/start-all.sh
中止服務
/usr/local/hadoop/sbin/stop-all.sh
訪問
瀏覽器打開http://172.7.15.113:8088/
瀏覽器打開http://172.7.15.113:50070/

centos7 安裝hadoop 2.7 - 測試hadoop

如下操做在master上實現

cd /usr/local/hadoop

創建測試目錄 bin/hdfs dfs -mkdir /123

若是提示 copyFromLocal: Cannot create directory /123/. Name node is in safe mode.

這是由於開啓了安全模式，解決辦法

bin/hdfs dfsadmin -safemode leave

將當前目錄下的LICENSE.txt複製到hadopp中

bin/hdfs dfs -copyFromLocal ./LICENSE.txt /123

查看/123/下有哪些文件 bin/hdfs dfs -ls /123

用wordcount分析LICENSE.txt bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount /123/LICENSE.txt /output/123

bin/hdfs dfs -ls /output/123 查看分析後的文件