六、Hadoop 2.6.0 運行

運行方式

1. Local (Standalone) Mode
2. Pseudo-Distributed Mode
3. Fully-Distributed Mode

---

Standalone Operation

$ sudo mkdir input
  $ sudo cp etc/hadoop/*.xml input
  $ sudo bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output 'dfs[a-z.]+'
  $ cat output/*

---

Pseudo-Distributed Operation

Configuration
Use the following:

etc/hadoop/core-site.xml:

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hw-TM1612:8020</value>
    </property>
</configuration>
    
<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/hadoop-2.6.0-cdh5.5.0/data/tmp</value>
    </property>
</configuration>

etc/hadoop/hdfs-site.xml:

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

sudo bin/hdfs namenode -format  ##格式化namenode
sudo sbin/hadoop-daemon.sh start namenode   ##啓動namenode
sudo sbin/hadoop-daemon.sh start datanode   ##啓動datanode
jps
sudo bin/hdfs dfs -mkdir -p /user/hw/   ##建立目錄
sudo bin/hdfs dfs -ls -R /  ##循環列出目錄 -R

sudo bin/hdfs dfs -mkdir -p /user/hw/mapreduce/wordcount/input  ##建立hdfs文件
sudo bin/hdfs dfs -put wcinput/wc.input /user/hw/mapreduce/wordcount/input    ## 上傳文件
sudo bin/hdfs dfs -ls /user/hw/mapreduce/wordcount/input
sudo bin/hdfs dfs -cat /user/hw/mapreduce/wordcount/input/wc.input  ##查看
sudo bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.5.0.jar wordcount /user/hw/mapreduce/wordcount/input/ /user/hw/mapreduce/wordcount/output ##hdfs下運行
sudo bin/hdfs dfs -cat /user/hw/mapreduce/wordcount/output/part-r-00000

• 默認web端口號：50070

---

YARN on Single Node

etc/hadoop/mapred-site.xml:

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

etc/hadoop/yarn-site.xml:

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

sudo gedit etc/hadoop/yarn-env.sh 
sudo gedit etc/hadoop/yarn-site.xml 
sudo gedit etc/hadoop/slaves    ##更改主機名
sudo sbin/yarn-daemon.sh start resourcemanager  ##啓動resourcemanager
jps
sudo sbin/yarn-daemon.sh start nodemanager  ##啓動nodemanager
jps
sudo gedit etc/hadoop/mapred-env.sh 
sudo mv etc/hadoop/mapred-site.xml.template etc/hadoop/mapred-site.xml
sudo gedit etc/hadoop/mapred-site.xml 
sudo bin/hdfs dfs -rm -r /user/hw/mapreduce/wordcount/output
sudo bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.5.0.jar wordcount /user/hw/mapreduce/wordcount/input/ /user/hw/mapreduce/wordcount/output

• 默認web端口號：8088

---

啓動歷史服務器

sudo sbin/mr-jobhistory-daemon.sh start historyserver