hadoop 僞分佈式安裝步驟

操做系統  centos7

java verson ： jdk1.8.0_151

hadoop version ： hadoop-2.8.1

 

一、把安裝包放到 opt文件夾下面。並進行解壓。

    cd /opt

    tar -zxvf hadoop-2.8.1

二、配置jdk、hadoop文件路徑

vi /etc/profile

export JAVA_HOME=/opt/jdk1.8.0_151
export JRE_HOME=${JAVA_HOME}/jre
export HADOOP_HOME=/opt/hadoop-2.8.1

export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH:${HADOOP_HOME}/bin

三、進入 etc/hadoop 目錄下進行配置文件設置

    修改指定運行的java路徑地址

    hadoop-env.sh 

# The java implementation to use.
#  export JAVA_HOME=${JAVA_HOME}
    export JAVA_HOME=/opt/jdk1.8.0_151

core-site.xml 默認的配置文件

    
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://localhost:9000/</value>
        </property>
       
        <property>
        　      <name>hadoop.tmp.dir</name>
        　      <value>/home/hadoop/hadoop-2.8.1/tmp</value>
        </property>

hdfs-site.xml

<property>
<name>dfs.replication</name>
<value>1</value>
</property>

mapred-site.xml

 

<property>
		<name>mapreduce.framework.name</name>
		<value>yarn</value>
	</property>

yarn-site.xml

<!-- <property>
		<name>yarn.resourcemanager.address</name>
		<value>localhost:9000</value>
	</property> -->

        <property>
		<name>yarn.resourcemanager.hostname</name>
		<value>localhost</value>
	</property>

	<property>
		<name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
               
	</property>
<property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

hadoop namenode -format

四、啓動

cd sbin

./start-dfs.sh

./start-yarn.sh

五、校驗查看

jps

六、瀏覽器地址訪問

http://localhost:8088/cluster

http://localhost:50070

七、向dfs中上傳文件

hadoop fs -put jdk-8u131-linux-x64.tar.gz  hdfs://localhost:9000/devtools

八、下載文件

hadoop fs -get hdfs://localhost:9000/jdk-8u131-linux-x64.tar.gz

九、運行mapreduce程序

hadoop jar hadoop-map

hadoop jar hadoop-mapreduce-examples-2.8.3.jar  wordcount /article /artout     //計算文本中詞彙的個數

hadoop jar hadoop-mapreduce-examples-2.8.3.jar  pi 5 5    //計算圓周率

10\免密登陸

 

$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
  $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
  $ chmod 0600 ~/.ssh/authorized_keys

ssh localhost

11\開放端口

firewall-cmd --zone=public --add-port=6379/tcp --permanent