hadoop-2.6.0.tar.gz + spark-1.5.2-bin-hadoop2.6.tgz的集羣搭建（單節點）（Ubuntu系統）

時間 2019-11-17

標籤 hadoop 2.6.0.tar.gz tar spark 1.5.2 bin hadoop2.6.tgz tgz 集羣搭建節點 ubuntu 系統欄目 Hadoop 简体版

原文原文鏈接

關於幾個疑問和幾處心得！html

a.用NAT，仍是橋接，仍是only-host模式？java

答： hostonly、橋接和NAT

b.用static的ip，仍是dhcp的？node

答：staticpython

c.別認爲快照和克隆不重要，小技巧，比別人靈活用，會很節省時間和大大減小錯誤。linux

d.重用起來腳本語言的編程，如paython或shell編程。git

　對於用scp -r命令或deploy.conf（配置文件），deploy.sh（實現文件複製的shell腳本文件），runRemoteCdm.sh(在遠程節點上執行命令的shell腳本文件)。web

e.重要Vmare Tools加強工具，或者，rz上傳、sz下載。sql

f.大多數人經常使用docker

Xmanager Enterprise *安裝步驟

用到的所需：shell

　　一、VMware-workstation-full-11.1.2.61471.1437365244.exe

　　二、ubuntukylin-14.04-desktop-amd64.iso

　　三、jdk-8u60-linux-x64.tar.gz

　　四、hadoop-2.6.0.tar.gz

　　五、scala-2.10.4.tgz

　　六、spark-1.5.2-bin-hadoop2.6.tgz

機器規劃：

　　192.168.80.128 ---------------- SparkSignleNode

目錄規劃：

　　一、下載目錄

　　　/home/spark/Downloads/Spark_Cluster_Software ---------------- 存放全部安裝軟件

二、新建目錄

三、安裝目錄

　　jdk-8u60-linux-x64.tar.gz -------------------------------------------------- /usr/local/jdk/jdk1.8.0_60

　　hadoop-2.6.0.tar.gz ---------------------------------------------------------- /usr/local/hadoop/hadoop-2.6.0

　　scala-2.10.4.tgz --------------------------------------------------------------- /usr/local/scala/scala-2.10.4

spark-1.5.2-bin-hadoop2.6.tgz ---------------------------------------------- /usr/local/spark/spark-1.5.2-bin-hadoop2.6

四、快照步驟

　　快照一：

　　　　剛安裝完畢，且能連上網

快照二：

　　　 root用戶的開啓、vim編輯器的安裝、ssh的安裝、靜態IP的設置、/etc/hostname和/etc/hosts和永久關閉防火牆

　　　 SSH安裝完以後的免密碼配置，放在後面
　　靜態IP是192.168.80.128
/etc/hostname是SparkSingleNode
/etc/hosts是
192.168.80.128 SparkSingleNode

快照三：

　　　　安裝jdk、安裝scala、配置SSH免密碼登陸、安裝python及ipython （這裏，選擇跳過也能夠，ubuntu系統自帶安裝了python）

　　　　新建spark用戶，（即用spark用戶，去安裝jdk、scala、配置SSH免密碼、安裝hadoop、安裝spark...）

　　快照四：

　　　　安裝hadoop（沒格式化）、安裝lrzsz、將本身寫好的替換掉默認的配置文件、創建好目錄

　　快照五：

　　　　安裝hadoop（格式化）成功、進程啓動正常

　　快照六：

　　　　spark的安裝和配置工做完成

　　快照七：

　　　　啓動hadoop、spark集羣成功、查看50070、808八、8080、4040頁面

第一步：

　　　　安裝VMware-workstation虛擬機，我這裏是VMware-workstation11版本。

　　詳細見 ->

VMware workstation 11 的下載　　　　　　

VMWare Workstation 11的安裝

VMware Workstation 11安裝以後的一些配置

　第二步：

　　　　安裝ubuntukylin-14.04-desktop系統（最好安裝英文系統）

　　　　詳細見 ->

Ubuntu各版本的歷史發行界面

Ubuntukylin-14.04-desktop（帶分區）安裝步驟詳解　

Ubuntukylin-14.04-desktop（不帶分區）安裝步驟詳解

第三步：VMware Tools加強工具安裝

　　　　詳細見 ->

VMware裏Ubuntukylin-14.04-desktop的VMware Tools安裝圖文詳解

　　第四步：準備小修改（學會用快照和克隆，根據自身要求狀況，合理位置快照）　

　　　　詳細見 ->　　　

CentOS經常使用命令、快照、克隆大揭祕

E：Package 'Vim' has no installation candidate問題解決

解決Ubuntu系統的每次開機重啓後，resolv.conf清空的問題

新建用戶組、用戶、用戶密碼、刪除用戶組、用戶（適合CentOS、Ubuntu）

　　　　一、root用戶的開啓（Ubuntu系統，安裝以後默認是沒有root用戶）

　　　　二、vim編輯器的安裝

　　　　三、ssh的安裝（SSH安裝完以後的免密碼配置，放在後面）

　　　　四、靜態IP的設置

Ubuntu14.04安裝以後的一些配置

五、/etc/hostname和/etc/hosts

root@SparkSingleNode:~# sudo cat /etc/hostname

SparkSingleNode

root@SparkSingleNode:~# sudo cat /etc/hosts

127.0.0.1 localhost

127.0.1.1 zhouls-virtual-machine

192.168.80.128 SparkSingleNode

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback

fe00::0 ip6-localnet

ff00::0 ip6-mcastprefix

ff02::1 ip6-allnodes

ff02::2 ip6-allrouters

　　六、永久關閉防火牆

　　　　　　　　通常，在搭建hadoop/spark集羣時，最好是永久關閉防火牆，由於，防火牆會保護其進程間通訊。

root@SparkSingleNode:~# sudo ufw status

Status: inactive

root@SparkSingleNode:~#

由此，代表Ubuntu14.04是默認沒開啓防火牆的。

三臺機器都照作！

新建用戶組、用戶、用戶密碼、刪除用戶組、用戶（適合CentOS、Ubuntu）

　這個知識點，模糊了很久。！！！

　　生產中，習慣以下：

　　useradd，默認會將自身新建用戶，添加到同名的用戶組中。如，useradd zhouls，執行此命令後，默認就添加到同名的zhouls用戶組中。

　可是，在生產中，通常都不這麼幹。一般是，useradd -m -g 。不然，出現到時，用戶創建出來了，但出現家目錄沒有哦。慎重！！！（重要的話，說三次）

###################Ubuntu系統裏###########################

Ubuntu系統裏，root用戶執行，先怎麼開啓，見 Ubuntu14.04安裝以後的一些配置

第一步：sudo groupadd 新建用戶組

sudo groupadd spark 這是建立spark用戶組

第二步：sudo useradd -m -g 已建立用戶組新建用戶

sudo useradd -m -g spark spark 這是新建spark用戶和家目錄也建立，並增長到spark組中

第三步：sudo passwd 已建立用戶

passwd spark spark用戶密碼

Changing password for user spark

New password ：

Retype new password:

###################################

root@SparkSingleNode:~# sudo groupadd spark
root@SparkSingleNode:~# sudo useradd -m -g spark spark
root@SparkSingleNode:~# sudo passwd spark
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
root@SparkSingleNode:~# su spark
spark@SparkSingleNode:/root$ cd
spark@SparkSingleNode:~$ pwd
/home/spark
spark@SparkSingleNode:~$

安裝前的思路梳理：

　　***********************************************************************************

　　*　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　*

　　* 編程語言 -> hadoop 集羣 -> spark 集羣　　　　　　　　　　　　　　　　　　　　　　　　　　 *

　　* 一、安裝jdk 　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　 *

　　* 二、安裝scala　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　 *

　　* 三、配置SSH免密碼登陸（SparkSingleNode自身）

　　* 四、安裝python及ipython （這裏，選擇跳過也能夠，ubuntu系統自帶安裝了python）

　　* 五、安裝hadoop　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　 *

　　* 六、安裝spark　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　*

　　* 七、啓動集羣　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　*

　　* 八、查看頁面　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　*

　　* 九、成功（記得快照）　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　*

　　*******************************************************

用wget命令在線下載，養成習慣，放到/home/spark/Downloads/Spark_Cluster_Software/目錄下，或者，安裝了Vmare加強工具Tools，直接拖進去。也能夠。

1、安裝jdk

　　jdk-8u60-linux-x64.tar.gz -------------------------------------------------- /usr/local/jdk/jdk1.8.0_60

　　　　一、jdk-8u60-linux-x64.tar.gz的下載

　　　　下載，http://download.csdn.net/download/aqtata/9022063

　　　　二、jdk-8u60-linux-x64.tar.gz的上傳

　　三臺機器都照作！

　　三、首先，檢查Ubuntu系統的自帶openjdk

spark@SparkSingleNode:~$ java -version
The program 'java' can be found in the following packages:
* default-jre
* gcj-4.8-jre-headless
* openjdk-7-jre-headless
* gcj-4.6-jre-headless
* openjdk-6-jre-headless
Ask your administrator to install one of them
spark@SparkSingleNode:~$ sudo apt-get purge openjdk*
[sudo] password for spark:
spark is not in the sudoers file. This incident will be reported.
spark@SparkSingleNode:~$

　　由此，可見，此Ubuntu系統，沒有自帶的openjdk。

出現了， XXX 用戶 is not in the sudoers file. This incident will be reported 的問題？

解決辦法：

　　　　 http://www.cnblogs.com/zox2011/archive/2013/05/28/3103824.html

spark@SparkSingleNode:~$ sudo apt-get purge openjdk*
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'openjdk-jre' for regex 'openjdk*'
Note, selecting 'openjdk-6-jre-lib' for regex 'openjdk*'
Note, selecting 'openjdk-7' for regex 'openjdk*'
Note, selecting 'openjdk-6-jdk' for regex 'openjdk*'
Note, selecting 'openjdk-7-jre-zero' for regex 'openjdk*'
Note, selecting 'openjdk-6-source' for regex 'openjdk*'
Note, selecting 'openjdk-6-jre-headless' for regex 'openjdk*'
Note, selecting 'openjdk-6-dbg' for regex 'openjdk*'
Note, selecting 'openjdk-7-jdk' for regex 'openjdk*'
Note, selecting 'openjdk-7-jre-headless' for regex 'openjdk*'
Note, selecting 'openjdk-6-jre' for regex 'openjdk*'
Note, selecting 'openjdk-7-dbg' for regex 'openjdk*'
Note, selecting 'openjdk-7-jre-lib' for regex 'openjdk*'
Note, selecting 'uwsgi-plugin-jvm-openjdk-6' for regex 'openjdk*'
Note, selecting 'uwsgi-plugin-jvm-openjdk-7' for regex 'openjdk*'
Note, selecting 'openjdk-6-doc' for regex 'openjdk*'
Note, selecting 'openjdk-7-jre' for regex 'openjdk*'
Note, selecting 'openjdk-7-source' for regex 'openjdk*'
Note, selecting 'openjdk-6-jre-zero' for regex 'openjdk*'
Note, selecting 'openjdk-7-demo' for regex 'openjdk*'
Note, selecting 'openjdk-7-doc' for regex 'openjdk*'
Note, selecting 'openjdk-6-demo' for regex 'openjdk*'
Note, selecting 'uwsgi-plugin-jwsgi-openjdk-6' for regex 'openjdk*'
Note, selecting 'uwsgi-plugin-jwsgi-openjdk-7' for regex 'openjdk*'
Package 'openjdk-7' is not installed, so not removed
Package 'openjdk-jre' is not installed, so not removed
Package 'uwsgi-plugin-jvm-openjdk-6' is not installed, so not removed
Package 'uwsgi-plugin-jvm-openjdk-7' is not installed, so not removed
Package 'uwsgi-plugin-jwsgi-openjdk-6' is not installed, so not removed

四、如今，新建/usr/loca/下的jdk目錄

spark@SparkSingleNode:~$ su root
Password:
root@SparkSingleNode:/home/spark# cd
root@SparkSingleNode:~# mkdir -p /usr/local/jdk
root@SparkSingleNode:~# cd /usr/local/jdk/
root@SparkSingleNode:/usr/local/jdk# ls
root@SparkSingleNode:/usr/local/jdk#

五、將下載的jdk文件移到剛剛建立的/usr/local/jdk下

root@SparkSingleNode:/usr/local/jdk# su spark
spark@SparkSingleNode:/usr/local/jdk$ sudo cp /home/spark/Downloads/Spark_Cluster_Software/jdk-8u60-linux-x64.tar.gz /usr/local/jdk/
spark@SparkSingleNode:/usr/local/jdk$ cd /usr/local/jdk/
spark@SparkSingleNode:/usr/local/jdk$ ls
jdk-8u60-linux-x64.tar.gz
spark@SparkSingleNode:/usr/local/jdk$

　　最好用cp，不要輕易要mv

　　六、解壓jdk文件

spark@SparkSingleNode:/usr/local/jdk$ ll
total 177000
drwxr-xr-x 2 root root 4096 9月 9 09:34 ./
drwxr-xr-x 11 root root 4096 9月 9 09:07 ../
-rwxr--r-- 1 root root 181238643 9月 9 09:34 jdk-8u60-linux-x64.tar.gz*
spark@SparkSingleNode:/usr/local/jdk$ su root
Password:
root@SparkSingleNode:/usr/local/jdk# ll
total 177000
drwxr-xr-x 2 root root 4096 9月 9 09:34 ./
drwxr-xr-x 11 root root 4096 9月 9 09:07 ../
-rwxr--r-- 1 root root 181238643 9月 9 09:34 jdk-8u60-linux-x64.tar.gz*
root@SparkSingleNode:/usr/local/jdk# ls
jdk-8u60-linux-x64.tar.gz
root@SparkSingleNode:/usr/local/jdk# tar -zxvf jdk-8u60-linux-x64.tar.gz

　　七、刪除解壓包，留下解壓完成的文件目錄，並修改權限（這是最重要的！）

root@SparkSingleNode:/usr/local/jdk# ll
total 177004
drwxr-xr-x 3 root root 4096 9月 9 09:54 ./
drwxr-xr-x 11 root root 4096 9月 9 09:07 ../
drwxr-xr-x 8 uucp 143 4096 8月 5 2015 jdk1.8.0_60/
-rwxr--r-- 1 root root 181238643 9月 9 09:34 jdk-8u60-linux-x64.tar.gz*
root@SparkSingleNode:/usr/local/jdk# ls
jdk1.8.0_60 jdk-8u60-linux-x64.tar.gz
root@SparkSingleNode:/usr/local/jdk# rm -rf jdk-8u60-linux-x64.tar.gz
root@SparkSingleNode:/usr/local/jdk# ls
jdk1.8.0_60
root@SparkSingleNode:/usr/local/jdk# ll
total 12
drwxr-xr-x 3 root root 4096 9月 9 09:55 ./
drwxr-xr-x 11 root root 4096 9月 9 09:07 ../
drwxr-xr-x 8 uucp 143 4096 8月 5 2015 jdk1.8.0_60/
root@SparkSingleNode:/usr/local/jdk#

root@SparkSingleNode:/usr/local/jdk# ll
total 12
drwxr-xr-x 3 root root 4096 9月 9 09:55 ./
drwxr-xr-x 11 root root 4096 9月 9 09:07 ../
drwxr-xr-x 8 uucp 143 4096 8月 5 2015 jdk1.8.0_60/
root@SparkSingleNode:/usr/local/jdk# chown -R spark:spark jdk1.8.0_60/
root@SparkSingleNode:/usr/local/jdk# ll
total 12
drwxr-xr-x 3 root root 4096 9月 9 09:55 ./
drwxr-xr-x 11 root root 4096 9月 9 09:07 ../
drwxr-xr-x 8 spark spark 4096 8月 5 2015 jdk1.8.0_60/
root@SparkSingleNode:/usr/local/jdk# su spark
spark@SparkSingleNode:/usr/local/jdk$ ll
total 12
drwxr-xr-x 3 root root 4096 9月 9 09:55 ./
drwxr-xr-x 11 root root 4096 9月 9 09:07 ../
drwxr-xr-x 8 spark spark 4096 8月 5 2015 jdk1.8.0_60/
spark@SparkSingleNode:/usr/local/jdk$

***********************************************

　　chown -R 用戶組：用戶文件

　　通常，咱們也能夠在以前，新建用戶組時，爲sparkuser，而後，它裏面的用戶，有spark1，spark2...

　　那麼，對應就是， chown -R sparkuser:spark1 jdk1.8.0_60

　　如，對hadoop-2.6.0.tar.gz的被解壓文件，作權限修改。

　　chown -R hduser:hadoop hadoop-2.6.0

**********************************************

八、修改環境變量　

　　vim ~./bash_profile 或 vim /etc/profile

　　配置在這個文件~/.bash_profile，或者也能夠，配置在那個全局的文件裏，也能夠喲。/etc/profile。

　　這裏，我vim /etc/profile

#java
export JAVA_HOME=/usr/local/jdk/jdk1.8.0_60
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin

root@SparkSingleNode:/usr/local/jdk/jdk1.8.0_60# vim /etc/profile
root@SparkSingleNode:/usr/local/jdk/jdk1.8.0_60# source /etc/profile
root@SparkSingleNode:/usr/local/jdk/jdk1.8.0_60# java -version
java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
root@SparkSingleNode:/usr/local/jdk/jdk1.8.0_60#

spark@SparkSingleNode:/usr/local/jdk/jdk1.8.0_60$ java -version
java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
spark@SparkSingleNode:/usr/local/jdk/jdk1.8.0_60$

　　至此，代表java安裝結束。

　　其餘兩臺都照作！

2、安裝scala

　　 scala-2.10.4.tgz --------------------------------------------------------------- /usr/local/scala/scala-2.10.4

　　一、scala的下載

　　http://www.scala-lang.org/files/archive/

　　二、scala-2.10.4.tgz 的上傳

　　其餘兩臺都照作！

　　三、如今，新建/usr/loca/下的sacla目錄

root@SparkSingleNode:/usr/local# pwd
/usr/local
root@SparkSingleNode:/usr/local# mkdir -p /usr/local/scala
root@SparkSingleNode:/usr/local#

　　四、將下載的scala文件移到剛剛建立的/usr/local/scala下

root@SparkSingleNode:/usr/local/scala# pwd
/usr/local/scala
root@SparkSingleNode:/usr/local/scala# ls
root@SparkSingleNode:/usr/local/scala# sudo cp /home/spark/Downloads/Spark_Cluster_Software/scala-2.10.4.tgz /usr/local/scala/
root@SparkSingleNode:/usr/local/scala# ls
scala-2.10.4.tgz
root@SparkSingleNode:/usr/local/scala#

最好用cp，不要輕易要mv

　　五、解壓scala文件

root@SparkSingleNode:/usr/local/scala# pwd
/usr/local/scala
root@SparkSingleNode:/usr/local/scala# ls
scala-2.10.4.tgz
root@SparkSingleNode:/usr/local/scala# ll
total 29244
drwxr-xr-x 2 root root 4096 9月 9 10:15 ./
drwxr-xr-x 12 root root 4096 9月 9 10:14 ../
-rwxr--r-- 1 root root 29937534 9月 9 10:15 scala-2.10.4.tgz*
root@SparkSingleNode:/usr/local/scala# tar -zxvf scala-2.10.4.tgz

　　六、刪除解壓包，留下解壓完成的文件目錄，並修改權限（這是最重要的！！！）

root@SparkSingleNode:/usr/local/scala# ls
scala-2.10.4 scala-2.10.4.tgz
root@SparkSingleNode:/usr/local/scala# ll
total 29248
drwxr-xr-x 3 root root 4096 9月 9 10:17 ./
drwxr-xr-x 12 root root 4096 9月 9 10:14 ../
drwxrwxr-x 9 2000 2000 4096 3月 18 2014 scala-2.10.4/
-rwxr--r-- 1 root root 29937534 9月 9 10:15 scala-2.10.4.tgz*
root@SparkSingleNode:/usr/local/scala# rm -rf scala-2.10.4.tgz
root@SparkSingleNode:/usr/local/scala# ll
total 12
drwxr-xr-x 3 root root 4096 9月 9 10:18 ./
drwxr-xr-x 12 root root 4096 9月 9 10:14 ../
drwxrwxr-x 9 2000 2000 4096 3月 18 2014 scala-2.10.4/
root@SparkSingleNode:/usr/local/scala# chown -R spark:spark scala-2.10.4/
root@SparkSingleNode:/usr/local/scala# ll
total 12
drwxr-xr-x 3 root root 4096 9月 9 10:18 ./
drwxr-xr-x 12 root root 4096 9月 9 10:14 ../
drwxrwxr-x 9 spark spark 4096 3月 18 2014 scala-2.10.4/
root@SparkSingleNode:/usr/local/scala#

　七、修改環境變量　

　　vim ~./bash_profile 或 vim /etc/profile

　　配置在這個文件~/.bash_profile，或者也能夠，配置在那個全局的文件裏，也能夠喲。/etc/profile。

　　這裏，我vim /etc/profile

　　#scala
　　export SCALA_HOME=/usr/local/scala/scala-2.10.4
　　export PATH=$PATH:$SCALA_HOME/bin

root@SparkSingleNode:/usr/local/scala/scala-2.10.4# vim /etc/profile
root@SparkSingleNode:/usr/local/scala/scala-2.10.4# source /etc/profile
root@SparkSingleNode:/usr/local/scala/scala-2.10.4# scala -version
Scala code runner version 2.10.4 -- Copyright 2002-2013, LAMP/EPFL
root@SparkSingleNode:/usr/local/scala/scala-2.10.4#

至此，代表scala安裝結束。

其餘兩臺都照作！

八、輸入scala命令，可直接進入scala的命令行交互界面。

root@SparkSingleNode:/usr/local/scala/scala-2.10.4# scala
Welcome to Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_60).
Type in expressions to have them evaluated.
Type :help for more information.

scala> 9*9
res0: Int = 81

scala> exit;
warning: there were 1 deprecation warning(s); re-run with -deprecation for details
root@SparkSingleNode:/usr/local/scala/scala-2.10.4#

3、配置免密碼登陸

　　一、配置SSH實現無密碼驗證配置，首先切換到剛建立的spark用戶下。

　　　　由於，我後續，是先搭建hadoop集羣，在其基礎上，再搭建spark集羣，目的，是在spark用戶下操做進行的。

　　因此，在這裏，要梳理下的是，root和zhouls，都是管理員權限。在生產環境裏，通常是不會動用這兩個管理員用戶的。

　　因爲spark須要無密碼登陸做爲worker的節點，而因爲部署單節點的時候，當前節點既是master又是worker，因此此時須要生成無密碼登陸的ssh。方法以下：

root@SparkSingleNode:/usr/local/scala/scala-2.10.4# cd
root@SparkSingleNode:~# su spark
spark@SparkSingleNode:/root$ cd
spark@SparkSingleNode:~$ pwd
/home/spark
spark@SparkSingleNode:~$

　　2 、建立.ssh目錄，生成密鑰

　　mkdir .ssh

　　ssh-keygen -t rsa 注意，ssh與keygen之間是沒有空格的

spark@SparkSingleNode:~$ mkdir .ssh
spark@SparkSingleNode:~$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/spark/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/spark/.ssh/id_rsa.
Your public key has been saved in /home/spark/.ssh/id_rsa.pub.
The key fingerprint is:
85:28:3f:f3:5b:47:3a:1d:bb:ed:6c:59:af:3e:9f:6b spark@SparkSingleNode
The key's randomart image is:
+--[ RSA 2048]----+
| |
| . . |
| . . . . |
| o . |
| + S o |
| + + o .|
| . + + o.|
| o o ++Eo|
| . .+B*o|
+-----------------+
spark@SparkSingleNode:~$

　　3 、切換到.ssh目錄下,進行查看公鑰和私鑰

　　cd .ssh

spark@SparkSingleNode:~$ cd .ssh
spark@SparkSingleNode:~/.ssh$ ls
id_rsa id_rsa.pub
spark@SparkSingleNode:~/.ssh$

　　四、將公鑰複製到日誌文件裏。查看是否複製成功

　　cp id_rsa.pub authorized_keys

spark@SparkSingleNode:~/.ssh$ cp id_rsa.pub authorized_keys
spark@SparkSingleNode:~/.ssh$ ls
authorized_keys id_rsa id_rsa.pub
spark@SparkSingleNode:~/.ssh$

　　五、查看日記文件具體內容

spark@SparkSingleNode:~/.ssh$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCqLwZVCWJOQT57Y9MAYw8YJtzqvJTnBob656jvKgLSaM5X8/cikS0HHGlfNqzldbP03+Z6ZrpaF2hyEV1v43kOhlqA9SFwTVhzbPzou2K0e7mgCjJlM4PQMOSZY+DUlHn08hDxdbgAhczj6pix4VNSORg2nBRLvk1CDFYSiviv+FRTxy4IhYfG0M74fOE/9jHnbXKNRmryexzSwEylVqISQFmt5X5ksqurTsIxc2M70mGnkoTAVNOMC/qNVw98FsTBwFLT9J8X3vtic7nn5PjLNi/Khyc/vOhiDpzRsJJ7r7BuaKvd/ENIu9WAjvSGvJKLfqx6SSGcociom7ol1S/Z spark@SparkSingleNode
spark@SparkSingleNode:~/.ssh$

　　六、退回到/home/spark/，來賦予權限

　　cd ..

　　chmod 700 .ssh 將.ssh文件夾的權限賦予700

　　chmod 600 .ssh/* 將.ssh文件夾裏面的文件（id_rsa、id_rsa.pub、authorized_keys）的權限賦予600

spark@SparkSingleNode:~/.ssh$ cd ..
spark@SparkSingleNode:~$ pwd
/home/spark
spark@SparkSingleNode:~$ chmod 700 .ssh
spark@SparkSingleNode:~$ chmod 600 .ssh/*
spark@SparkSingleNode:~$

　　七、測試ssh無密碼訪問

spark@SparkSingleNode:~$ ssh SparkSingleNode
The authenticity of host 'sparksinglenode (192.168.80.128)' can't be established.
ECDSA key fingerprint is c7:ae:2f:38:e6:88:6f:ed:ee:f0:14:d8:98:f4:9e:3b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'sparksinglenode,192.168.80.128' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-24-generic x86_64)

* Documentation: https://help.ubuntu.com/

Last login: Fri Sep 9 08:51:53 2016 from 192.168.80.1
$ pwd
/home/spark

spark@SparkSingleNode:~$ ssh SparkSingleNode
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-24-generic x86_64)

* Documentation: https://help.ubuntu.com/

Last login: Fri Sep 9 10:35:48 2016 from sparksinglenode
$ exit;
Connection to sparksinglenode closed.
spark@SparkSingleNode:~$

　　4、安裝python及ipython （這裏，選擇跳過也能夠，ubuntu系統自帶安裝了python）

　　　　默認的安裝目錄，是在/usr/lib/下

spark@SparkSingleNode:~$ sudo apt-get install python ipython -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
python is already the newest version.
The following extra packages will be installed:
python-decorator python-simplegeneric
Suggested packages:
ipython-doc ipython-notebook ipython-qtconsole python-matplotlib python-numpy python-zmq
The following NEW packages will be installed:
ipython python-decorator python-simplegeneric
0 upgraded, 3 newly installed, 0 to remove and 740 not upgraded.
Need to get 619 kB of archives.
After this operation, 3,436 kB of additional disk space will be used.
Get:1 http://cn.archive.ubuntu.com/ubuntu/ trusty/main python-decorator all 3.4.0-2build1 [19.2 kB]
Get:2 http://cn.archive.ubuntu.com/ubuntu/ trusty/main python-simplegeneric all 0.8.1-1 [11.5 kB]
Get:3 http://cn.archive.ubuntu.com/ubuntu/ trusty/universe ipython all 1.2.1-2 [588 kB]
Fetched 619 kB in 31s (19.8 kB/s)
Selecting previously unselected package python-decorator.
(Reading database ... 147956 files and directories currently installed.)
Preparing to unpack .../python-decorator_3.4.0-2build1_all.deb ...
Unpacking python-decorator (3.4.0-2build1) ...
Selecting previously unselected package python-simplegeneric.
Preparing to unpack .../python-simplegeneric_0.8.1-1_all.deb ...
Unpacking python-simplegeneric (0.8.1-1) ...
Selecting previously unselected package ipython.
Preparing to unpack .../ipython_1.2.1-2_all.deb ...
Unpacking ipython (1.2.1-2) ...
Processing triggers for man-db (2.6.7.1-1) ...
Processing triggers for hicolor-icon-theme (0.13-1) ...
Processing triggers for gnome-menus (3.10.1-0ubuntu2) ...
Processing triggers for desktop-file-utils (0.22-1ubuntu1) ...
Processing triggers for bamfdaemon (0.5.1+14.04.20140409-0ubuntu1) ...

Rebuilding /usr/share/applications/bamf-2.index...
Processing triggers for mime-support (3.54ubuntu1) ...
Setting up python-decorator (3.4.0-2build1) ...
Setting up python-simplegeneric (0.8.1-1) ...
Setting up ipython (1.2.1-2) ...
spark@SparkSingleNode:~$

　　測試是否安裝成功

spark@SparkSingleNode:~$ python --version
Python 2.7.6
spark@SparkSingleNode:~$ ipython --version
1.2.1
spark@SparkSingleNode:~$

　　同時，對ipython，想說的是。

　　IPYTHON and IPYTHON_OPTS are removed in Spark 2.0+ . Remove these from the environment and set PYSPARK_DRIVER_PYTHON and PYSPARK_DRIVER_PYTHON_OPTS　instead .

　在任何路徑下，均可以執行python。

spark@SparkSingleNode:~$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
spark@SparkSingleNode:~$ cd /usr/local/
spark@SparkSingleNode:/usr/local$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
spark@SparkSingleNode:/usr/local$ cd /usr/lib/
spark@SparkSingleNode:/usr/lib$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
spark@SparkSingleNode:/usr/lib$

　　5、安裝hadoop

　　　　hadoop-2.6.0.tar.gz ---------------------------------------------------------- /usr/local/hadoop/hadoop-2.6.0

　　一、hadoop的下載

http://archive.apache.org/dist/hadoop/common/hadoop-2.6.0/

　　二、hadoop-2.6.0.tar.gz的上傳

　　三、如今，新建/usr/loca/下的hadoop目錄

root@SparkSingleNode:/usr/local# pwd
/usr/local
root@SparkSingleNode:/usr/local# mkdir -p /usr/local/hadoop
root@SparkSingleNode:/usr/local# ls
bin etc games hadoop include jdk lib man sbin scala share src
root@SparkSingleNode:/usr/local# cd hadoop/
root@SparkSingleNode:/usr/local/hadoop# pwd
/usr/local/hadoop
root@SparkSingleNode:/usr/local/hadoop# ls
root@SparkSingleNode:/usr/local/hadoop#

　　四、將下載的hadoop文件移到剛剛建立的/usr/local/hadoop下

　　最好用cp，不要輕易要mv

root@SparkSingleNode:/usr/local/hadoop# sudo cp /home/spark/Downloads/Spark_Cluster_Software/hadoop-2.6.0.tar.gz /usr/local/hadoop/
root@SparkSingleNode:/usr/local/hadoop# ls
hadoop-2.6.0.tar.gz
root@SparkSingleNode:/usr/local/hadoop#

　　五、解壓hadoop文件

root@SparkSingleNode:/usr/local/hadoop# ls
hadoop-2.6.0.tar.gz
root@SparkSingleNode:/usr/local/hadoop# tar -zxvf hadoop-2.6.0.tar.gz

六、刪除解壓包，留下解壓完成的文件目錄

　並修改所屬的用戶組和用戶（這是最重要的！）

root@SparkSingleNode:/usr/local/hadoop# ls
hadoop-2.6.0 hadoop-2.6.0.tar.gz
root@SparkSingleNode:/usr/local/hadoop# rm -rf hadoop-2.6.0.tar.gz
root@SparkSingleNode:/usr/local/hadoop# ls
hadoop-2.6.0
root@SparkSingleNode:/usr/local/hadoop# ll
total 12
drwxr-xr-x 3 root root 4096 9月 9 11:33 ./
drwxr-xr-x 13 root root 4096 9月 9 11:28 ../
drwxr-xr-x 9 20000 20000 4096 11月 14 2014 hadoop-2.6.0/
root@SparkSingleNode:/usr/local/hadoop# chown -R spark:spark hadoop-2.6.0/
root@SparkSingleNode:/usr/local/hadoop# ll
total 12
drwxr-xr-x 3 root root 4096 9月 9 11:33 ./
drwxr-xr-x 13 root root 4096 9月 9 11:28 ../
drwxr-xr-x 9 spark spark 4096 11月 14 2014 hadoop-2.6.0/
root@SparkSingleNode:/usr/local/hadoop#

七、修改環境變量　

　　　　vim ~./bash_profile 或 vim /etc/profile

　　　　配置在這個文件~/.bash_profile，或者也能夠，配置在那個全局的文件裏，也能夠喲。/etc/profile。

　　　　這裏，我vim /etc/profile

#hadoop
export HADOOP_HOME=/usr/local/hadoop/hadoop-2.6.0
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

root@SparkSingleNode:/usr/local/hadoop# vim /etc/profile
root@SparkSingleNode:/usr/local/hadoop# source /etc/profile
root@SparkSingleNode:/usr/local/hadoop# hadoop version
Hadoop 2.6.0
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1
Compiled by jenkins on 2014-11-13T21:10Z
Compiled with protoc 2.5.0
From source with checksum 18e43357c8f927c0695f1e9522859d6a
This command was run using /usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0.jar
root@SparkSingleNode:/usr/local/hadoop#

至此，代表hadoop安裝結束。

　　配置hadoop的配置文件

　　　　經驗起見，通常都是在NotePad++裏，弄好，丟上去。

　　在windows裏解壓，打開它的配置，寫好。

核心

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->

<configuration>
　　<property>
　　　　<name>fs.default.name</name>
　　　　<value>hdfs://SparkSingleNode:9000</value>
　　</property>
　　<property>
　　　　<name>hadoop.tmp.dir</name>
　　　　<value>/usr/local/hadoop/hadoop-2.6.0/tmp</value>
　　</property>
　　<property>
　　　　<name>hadoop.proxyuser.hadoop.hosts</name>
　　　　<value>*</value>
　　</property>
　　<property>
　　　　<name>hadoop.proxyuser.hadoop.groups</name>
　　　　<value>*</value>
　　</property>
</configuration>

上面配置的是，由於在hadoop1.0中引入了安全機制，因此從客戶端發出的做業提交者全變成了hadoop，無論原始提交者是哪一個用戶，爲了解決該問題，引入了安全違章功能，容許一個超級用戶來代替其餘用戶來提交做業或者執行命令，而對外來看，執行者仍然是普通用戶。因此，配置設爲任意客戶端和配置設爲任意用戶組。

存儲

http://www.apache.org/licenses/LICENSE-2.0

<configuration>
　　<property>
　　　　<name>dfs.permissions</name>
　　　　<value>false</value>
　　</property>
　　<property>
　　　　<name>dfs.replication</name>
　　　　<value>1</value>
　　</property>
　　<property>
　　　　<name>dfs.namenode.name.dir</name>
　　　　<value>/usr/local/hadoop/hadoop-2.6.0/dfs/name</value>
　　</property>
　　<property>
　　　　<name>dfs.datanode.data.dir</name>
　　　　<value>/usr/local/hadoop/hadoop-2.6.0/dfs/data</value>
　　</property>
　</configuration>

計算

變成

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

<configuration>
　　<property>
　　　　<name>mapreduce.framework.name</name>
　　　　<value>yarn</value>
　　</property>
</configuration>

管理

<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

<property>
　　　　<name>yarn.nodemanager.aux-services</name>
　　　　<value>mapreduce_shuffle</value>
　　</property>
</configuration>

環境

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Set Hadoop-specific environment variables here.

# The only required environment variable is JAVA_HOME. All others are
# optional. When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.

# The java implementation to use.
export JAVA_HOME=/usr/local/jdk/jdk1.8.0_60

# The jsvc implementation to use. Jsvc is required to run secure datanodes
# that bind to privileged ports to provide authentication of data transfer
# protocol. Jsvc is not required if SASL is configured for authentication of
# data transfer protocol using non-privileged ports.
#export JSVC_HOME=${JSVC_HOME}

export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}

# Extra Java CLASSPATH elements. Automatically insert capacity-scheduler.
for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
if [ "$HADOOP_CLASSPATH" ]; then
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
else
export HADOOP_CLASSPATH=$f
fi
done

# The maximum amount of heap to use, in MB. Default is 1000.
#export HADOOP_HEAPSIZE=
#export HADOOP_NAMENODE_INIT_HEAPSIZE=""

# Extra Java runtime options. Empty by default.
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"

# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS"

export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"

export HADOOP_NFS3_OPTS="$HADOOP_NFS3_OPTS"
export HADOOP_PORTMAP_OPTS="-Xmx512m $HADOOP_PORTMAP_OPTS"

# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS"
#HADOOP_JAVA_PLATFORM_OPTS="-XX:-UsePerfData $HADOOP_JAVA_PLATFORM_OPTS"

# On secure datanodes, user to run the datanode as after dropping privileges.
# This **MUST** be uncommented to enable secure HDFS if using privileged ports
# to provide authentication of data transfer protocol. This **MUST NOT** be
# defined if SASL is configured for authentication of data transfer protocol
# using non-privileged ports.
export HADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER}

# Where log files are stored. $HADOOP_HOME/logs by default.
#export HADOOP_LOG_DIR=${HADOOP_LOG_DIR}/$USER

# Where log files are stored in the secure data environment.
export HADOOP_SECURE_DN_LOG_DIR=${HADOOP_LOG_DIR}/${HADOOP_HDFS_USER}

###
# HDFS Mover specific parameters
###
# Specify the JVM options to be used when starting the HDFS Mover.
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
# export HADOOP_MOVER_OPTS=""

###
# Advanced Users Only!
###

# The directory where pid files are stored. /tmp by default.
# NOTE: this should be set to a directory that can only be written to by
# the user that will run the hadoop daemons. Otherwise there is the
# potential for a symlink attack.
export HADOOP_PID_DIR=${HADOOP_PID_DIR}
export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}

# A string representing this instance of hadoop. $USER by default.
export HADOOP_IDENT_STRING=$USER

主、從節點

SparkSingleNode

　　　　將SparkSingleNode的各自原有配置這幾個文件，刪去。

spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop$ pwd
/usr/local/hadoop/hadoop-2.6.0/etc/hadoop
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop$ ls
capacity-scheduler.xml hadoop-env.cmd hadoop-policy.xml httpfs-signature.secret kms-log4j.properties mapred-env.sh ssl-client.xml.example yarn-site.xml
configuration.xsl hadoop-env.sh hdfs-site.xml httpfs-site.xml kms-site.xml mapred-queues.xml.template ssl-server.xml.example
container-executor.cfg hadoop-metrics2.properties httpfs-env.sh kms-acls.xml log4j.properties mapred-site.xml.template yarn-env.cmd
core-site.xml hadoop-metrics.properties httpfs-log4j.properties kms-env.sh mapred-env.cmd slaves yarn-env.sh
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop$ rm -rf core-site.xml
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop$ rm -rf hdfs-site.xml
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop$ rm -rf mapred-site.xml.template
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop$ rm -rf yarn-site.xml
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop$ rm -rf hadoop-env.sh
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop$ rm -rf slaves
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop$ ls
capacity-scheduler.xml hadoop-env.cmd hadoop-policy.xml httpfs-signature.secret kms-env.sh log4j.properties mapred-queues.xml.template yarn-env.cmd
configuration.xsl hadoop-metrics2.properties httpfs-env.sh httpfs-site.xml kms-log4j.properties mapred-env.cmd ssl-client.xml.example yarn-env.sh
container-executor.cfg hadoop-metrics.properties httpfs-log4j.properties kms-acls.xml kms-site.xml mapred-env.sh ssl-server.xml.example
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop$

將寫好的，丟上去。

root@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop# pwd
/usr/local/hadoop/hadoop-2.6.0/etc/hadoop
root@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop# rz
The program 'rz' is currently not installed. You can install it by typing:
apt-get install lrzsz
root@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop# sudo apt-get install lrzsz

spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop$ pwd
/usr/local/hadoop/hadoop-2.6.0/etc/hadoop
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop$ rz
rz waiting to receive.
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop$ rz
rz waiting to receive.
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop$ rz
rz waiting to receive.
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop$ rz
rz waiting to receive.
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop$ rz
rz waiting to receive.
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop$ rz
rz waiting to receive.
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop$ ls
capacity-scheduler.xml hadoop-env.cmd hadoop-policy.xml httpfs-signature.secret kms-log4j.properties mapred-env.sh ssl-client.xml.example yarn-site.xml
configuration.xsl hadoop-env.sh hdfs-site.xml httpfs-site.xml kms-site.xml mapred-queues.xml.template ssl-server.xml.example
container-executor.cfg hadoop-metrics2.properties httpfs-env.sh kms-acls.xml log4j.properties mapred-site.xml yarn-env.cmd
core-site.xml hadoop-metrics.properties httpfs-log4j.properties kms-env.sh mapred-env.cmd slaves yarn-env.sh
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop$

　　新建目錄

　　　　root@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0# mkdir -p /usr/local/hadoop/hadoop-2.6.0/dfs/name

　　　　root@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0# mkdir -p /usr/local/hadoop/hadoop-2.6.0/dfs/data

　　　　root@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0# mkdir -p /usr/local/hadoop/hadoop-2.6.0/tmp

　　　　推薦這種新建！可是得要到hadoop-2.6.0.tar.gz被解壓完成，獲得才作！

　　與

　　　　root@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0# mkdir dfs

　　　　root@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/dfs# mkdir name

　　　　root@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0/dfs# mkdir data

　　是同樣的。

　　至此，hadoop的配置工做完成！

　　hadoop的格式化

　　　　　在主節點(SparkSingleNode)的hadoop的安裝目錄下，進行以下命令操做

　　　　./bin/hadoop namenode -format

spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0$ pwd
/usr/local/hadoop/hadoop-2.6.0
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0$ ./bin/hadoop namenode -format
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

16/09/09 12:12:41 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = SparkSingleNode/192.168.80.128
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.6.0
STARTUP_MSG: classpath = /usr/local/hadoop/hadoop-2.6.0/etc/hadoop:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/asm-3.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/hadoop-auth-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/htrace-core-3.0.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/gson-2.2.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/curator-client-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/activation-1.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/xz-1.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/junit-4.11.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/curator-framework-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-collections-3.2.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jsr305-1.3.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-el-1.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/l

ocal/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/hadoop-annotations-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0-tests.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/hadoop-nfs-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/htrace-core-3.0.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-el-1.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/hadoop-hdfs-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/hadoop-hdfs-2.6.0-tests.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/asm-3.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jline-0.9.94.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/activation-1.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/xz-1.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/local/

hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/guice-3.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-registry-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-common-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-api-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-client-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/xz-1.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-tests.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.0.jar:/usr/local/hadoop/hadoop-2.6.0/contrib/capacity-scheduler/*.jar:/usr/local/hadoop/hadoop-2.6.0/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1; compiled by 'jenkins' on 2014-11-13T21:10Z
STARTUP_MSG: java = 1.8.0_60
************************************************************/
16/09/09 12:12:41 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
16/09/09 12:12:41 INFO namenode.NameNode: createNameNode [-format]
16/09/09 12:12:48 WARN common.Util: Path /usr/local/hadoop/hadoop-2.6.0/dfs/name should be specified as a URI in configuration files. Please update hdfs configuration.

16/09/09 12:12:48 WARN common.Util: Path /usr/local/hadoop/hadoop-2.6.0/dfs/name should be specified as a URI in configuration files. Please update hdfs configuration.
Formatting using clusterid: CID-fcd6d126-a431-4df6-a9b3-f5caf6f14330
16/09/09 12:12:48 INFO namenode.FSNamesystem: No KeyProvider found.
16/09/09 12:12:49 INFO namenode.FSNamesystem: fsLock is fair:true
16/09/09 12:12:49 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
16/09/09 12:12:49 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
16/09/09 12:12:49 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
16/09/09 12:12:49 INFO blockmanagement.BlockManager: The block deletion will start around 2016 Sep 09 12:12:49
16/09/09 12:12:49 INFO util.GSet: Computing capacity for map BlocksMap
16/09/09 12:12:49 INFO util.GSet: VM type = 64-bit
16/09/09 12:12:49 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
16/09/09 12:12:49 INFO util.GSet: capacity = 2^21 = 2097152 entries
16/09/09 12:12:49 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
16/09/09 12:12:49 INFO blockmanagement.BlockManager: defaultReplication = 1
16/09/09 12:12:49 INFO blockmanagement.BlockManager: maxReplication = 512
16/09/09 12:12:49 INFO blockmanagement.BlockManager: minReplication = 1
16/09/09 12:12:49 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
16/09/09 12:12:49 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
16/09/09 12:12:49 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
16/09/09 12:12:49 INFO blockmanagement.BlockManager: encryptDataTransfer = false
16/09/09 12:12:49 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
16/09/09 12:12:50 INFO namenode.FSNamesystem: fsOwner = spark (auth:SIMPLE)
16/09/09 12:12:50 INFO namenode.FSNamesystem: supergroup = supergroup
16/09/09 12:12:50 INFO namenode.FSNamesystem: isPermissionEnabled = false
16/09/09 12:12:50 INFO namenode.FSNamesystem: HA Enabled: false
16/09/09 12:12:50 INFO namenode.FSNamesystem: Append Enabled: true
16/09/09 12:12:52 INFO util.GSet: Computing capacity for map INodeMap
16/09/09 12:12:52 INFO util.GSet: VM type = 64-bit
16/09/09 12:12:52 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
16/09/09 12:12:52 INFO util.GSet: capacity = 2^20 = 1048576 entries

16/09/09 12:12:52 INFO namenode.NameNode: Caching file names occuring more than 10 times
16/09/09 12:12:52 INFO util.GSet: Computing capacity for map cachedBlocks
16/09/09 12:12:52 INFO util.GSet: VM type = 64-bit
16/09/09 12:12:52 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
16/09/09 12:12:52 INFO util.GSet: capacity = 2^18 = 262144 entries
16/09/09 12:12:52 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
16/09/09 12:12:52 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
16/09/09 12:12:52 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
16/09/09 12:12:52 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
16/09/09 12:12:52 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
16/09/09 12:12:53 INFO util.GSet: Computing capacity for map NameNodeRetryCache
16/09/09 12:12:53 INFO util.GSet: VM type = 64-bit
16/09/09 12:12:53 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
16/09/09 12:12:53 INFO util.GSet: capacity = 2^15 = 32768 entries
16/09/09 12:12:53 INFO namenode.NNConf: ACLs enabled? false
16/09/09 12:12:53 INFO namenode.NNConf: XAttrs enabled? true
16/09/09 12:12:53 INFO namenode.NNConf: Maximum size of an xattr: 16384
16/09/09 12:12:53 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1409354162-192.168.80.128-1473394373253
16/09/09 12:12:54 INFO common.Storage: Storage directory /usr/local/hadoop/hadoop-2.6.0/dfs/name has been successfully formatted.
16/09/09 12:12:54 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
16/09/09 12:12:54 INFO util.ExitUtil: Exiting with status 0
16/09/09 12:12:55 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at SparkSingleNode/192.168.80.128
************************************************************/
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0$

　　啓動hadoop

　　　　./sbin/start-all.sh

spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0$ pwd
/usr/local/hadoop/hadoop-2.6.0
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0$ ./sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [SparkSingleNode]
SparkSingleNode: starting namenode, logging to /usr/local/hadoop/hadoop-2.6.0/logs/hadoop-spark-namenode-SparkSingleNode.out
SparkSingleNode: starting datanode, logging to /usr/local/hadoop/hadoop-2.6.0/logs/hadoop-spark-datanode-SparkSingleNode.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/hadoop-2.6.0/logs/hadoop-spark-secondarynamenode-SparkSingleNode.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/hadoop-2.6.0/logs/yarn-spark-resourcemanager-SparkSingleNode.out
SparkSingleNode: starting nodemanager, logging to /usr/local/hadoop/hadoop-2.6.0/logs/yarn-spark-nodemanager-SparkSingleNode.out
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0$ jps

spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0$ jps
4787 DataNode
4679 NameNode
5610 Jps
5132 ResourceManager
5245 NodeManager
4959 SecondaryNameNode
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0$

5、安裝spark

　　　　spark-1.5.2-bin-hadoop2.6.tgz ---------------------------------------------- /usr/loca/spark/spark-1.5.2-bin-hadoop2.6

　　　　一、spark的下載　　　

http://mirror.bit.edu.cn/apache/spark/spark-1.5.2/

　　二、spark-1.5.2-bin-hadoop2.6.tgz的上傳

　　三、如今，新建/usr/local下的spark目錄

root@SparkSingleNode:/usr/local# pwd
/usr/local
root@SparkSingleNode:/usr/local# ls
bin etc games hadoop include jdk lib man sbin scala share src
root@SparkSingleNode:/usr/local# mkdir -p /usr/local/spark
root@SparkSingleNode:/usr/local# ls
bin etc games hadoop include jdk lib man sbin scala share spark src
root@SparkSingleNode:/usr/local# cd spark/
root@SparkSingleNode:/usr/local/spark# ls
root@SparkSingleNode:/usr/local/spark#

　　四、將下載的spark文件移到剛剛建立的/usr/local/spark下

　　最好用cp，不要輕易要mv

root@SparkSingleNode:/usr/local/spark# pwd
/usr/local/spark
root@SparkSingleNode:/usr/local/spark# sudo cp /home/spark/Downloads/Spark_Cluster_Software/spark-1.5.2-bin-hadoop2.6.tgz /usr/local/spark/
root@SparkSingleNode:/usr/local/spark# ls
spark-1.5.2-bin-hadoop2.6.tgz
root@SparkSingleNode:/usr/local/spark#

　　五、解壓spark文件

root@SparkSingleNode:/usr/local/spark# ls
spark-1.5.2-bin-hadoop2.6.tgz
root@SparkSingleNode:/usr/local/spark# tar -zxvf spark-1.5.2-bin-hadoop2.6.tgz

　　六、刪除解壓包，留下解壓完成的文件目錄

　並修改所屬的用戶組和用戶（這是最重要的！）

root@SparkSingleNode:/usr/local/spark# ls
spark-1.5.2-bin-hadoop2.6 spark-1.5.2-bin-hadoop2.6.tgz
root@SparkSingleNode:/usr/local/spark# rm -rf spark-1.5.2-bin-hadoop2.6.tgz
root@SparkSingleNode:/usr/local/spark# ls
spark-1.5.2-bin-hadoop2.6
root@SparkSingleNode:/usr/local/spark# ll
total 12
drwxr-xr-x 3 root root 4096 9月 9 15:04 ./
drwxr-xr-x 14 root root 4096 9月 9 14:58 ../
drwxr-xr-x 12 500 500 4096 11月 4 2015 spark-1.5.2-bin-hadoop2.6/
root@SparkSingleNode:/usr/local/spark# chown -R spark:spark spark-1.5.2-bin-hadoop2.6/
root@SparkSingleNode:/usr/local/spark# ll
total 12
drwxr-xr-x 3 root root 4096 9月 9 15:04 ./
drwxr-xr-x 14 root root 4096 9月 9 14:58 ../
drwxr-xr-x 12 spark spark 4096 11月 4 2015 spark-1.5.2-bin-hadoop2.6/
root@SparkSingleNode:/usr/local/spark#

　　七、修改環境變量　

　　　　　　vim ~./bash_profile 或 vim /etc/profile

　　　　　　配置在這個文件~/.bash_profile，或者也能夠，配置在那個全局的文件裏，也能夠喲。/etc/profile。

　　　　　　這裏，我vim /etc/profile

#spark
export SPARK_HOME=/usr/local/spark/spark-1.5.2-bin-hadoop2.6
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

root@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6# vim /etc/profile
root@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6# source /etc/profile
root@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6#

　　至此，代表spark安裝結束。

　　其餘兩臺機器都照作!

　　配置spark的配置文件

　　　　　　經驗起見，通常都是在NotePad++裏，弄好，丟上去

　　變成

#!/usr/bin/env bash

# This file is sourced when running various Spark programs.
# Copy it as spark-env.sh and edit that to configure Spark for your site.

# Options read when launching programs locally with
# ./bin/run-example or ./bin/spark-submit
# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
# - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node
# - SPARK_PUBLIC_DNS, to set the public dns name of the driver program
# - SPARK_CLASSPATH, default classpath entries to append

# Options read by executors and drivers running inside the cluster
# - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node
# - SPARK_PUBLIC_DNS, to set the public DNS name of the driver program
# - SPARK_CLASSPATH, default classpath entries to append
# - SPARK_LOCAL_DIRS, storage directories to use on this node for shuffle and RDD data
# - MESOS_NATIVE_JAVA_LIBRARY, to point to your libmesos.so if you use Mesos

# Options read in YARN client mode
# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
# - SPARK_EXECUTOR_INSTANCES, Number of workers to start (Default: 2)
# - SPARK_EXECUTOR_CORES, Number of cores for the workers (Default: 1).
# - SPARK_EXECUTOR_MEMORY, Memory per Worker (e.g. 1000M, 2G) (Default: 1G)
# - SPARK_DRIVER_MEMORY, Memory for Master (e.g. 1000M, 2G) (Default: 1G)
# - SPARK_YARN_APP_NAME, The name of your application (Default: Spark)
# - SPARK_YARN_QUEUE, The hadoop queue to use for allocation requests (Default: ‘default’)
# - SPARK_YARN_DIST_FILES, Comma separated list of files to be distributed with the job.
# - SPARK_YARN_DIST_ARCHIVES, Comma separated list of archives to be distributed with the job.

# Options for the daemons used in the standalone deploy mode
# - SPARK_MASTER_IP, to bind the master to a different IP address or hostname
# - SPARK_MASTER_PORT / SPARK_MASTER_WEBUI_PORT, to use non-default ports for the master
# - SPARK_MASTER_OPTS, to set config properties only for the master (e.g. "-Dx=y")
# - SPARK_WORKER_CORES, to set the number of cores to use on this machine
# - SPARK_WORKER_MEMORY, to set how much total memory workers have to give executors (e.g. 1000m, 2g)
# - SPARK_WORKER_PORT / SPARK_WORKER_WEBUI_PORT, to use non-default ports for the worker
# - SPARK_WORKER_INSTANCES, to set the number of worker processes per node
# - SPARK_WORKER_DIR, to set the working directory of worker processes
# - SPARK_WORKER_OPTS, to set config properties only for the worker (e.g. "-Dx=y")
# - SPARK_DAEMON_MEMORY, to allocate to the master, worker and history server themselves (default: 1g).
# - SPARK_HISTORY_OPTS, to set config properties only for the history server (e.g. "-Dx=y")
# - SPARK_SHUFFLE_OPTS, to set config properties only for the external shuffle service (e.g. "-Dx=y")
# - SPARK_DAEMON_JAVA_OPTS, to set config properties for all daemons (e.g. "-Dx=y")
# - SPARK_PUBLIC_DNS, to set the public dns name of the master or workers

# Generic options for the daemons used in the standalone deploy mode
# - SPARK_CONF_DIR Alternate conf dir. (Default: ${SPARK_HOME}/conf)
# - SPARK_LOG_DIR Where log files are stored. (Default: ${SPARK_HOME}/logs)
# - SPARK_PID_DIR Where the pid file is stored. (Default: /tmp)
# - SPARK_IDENT_STRING A string representing this instance of spark. (Default: $USER)
# - SPARK_NICENESS The scheduling priority for daemons. (Default: 0)

export JAVA_HOME=/usr/local/jdk/jdk1.8.0_60
export SCALA_HOME=/usr/local/scala/scala-2.10.4
export HADOOP_HOME=/usr/local/hadoop/hadoop-2.6.0
export HADOOP_CONF_DIR=/usr/local/hadoop/hadoop-2.6.0/etc/hadoop
export SPARK_MASTER_IP=SparkSingleNode
export SPARK_WORKER_MERMORY=2G （官網上說，至少是1g起步）

　　這裏啊，我考慮在單節點裏，玩玩spark，做爲學習的入門。設爲2G。固然，這個值，往後也能夠更改，好比，變大到4G均可以的。

　　從節點

SparkSingleNode

　　將SparkSingleNode的各自原有配置這幾個文件，刪去

　在這裏，最好是複製，由於權限。

spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6$ pwd
/usr/local/spark/spark-1.5.2-bin-hadoop2.6
spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6$ ls
bin CHANGES.txt conf data ec2 examples lib LICENSE licenses NOTICE python R README.md RELEASE sbin
spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6$ cd conf/
spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6/conf$ ls
docker.properties.template fairscheduler.xml.template log4j.properties.template metrics.properties.template slaves.template spark-defaults.conf.template spark-env.sh.template
spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6/conf$ ll
total 40
drwxr-xr-x 2 spark spark 4096 11月 4 2015 ./
drwxr-xr-x 12 spark spark 4096 11月 4 2015 ../
-rw-r--r-- 1 spark spark 202 11月 4 2015 docker.properties.template
-rw-r--r-- 1 spark spark 303 11月 4 2015 fairscheduler.xml.template
-rw-r--r-- 1 spark spark 949 11月 4 2015 log4j.properties.template
-rw-r--r-- 1 spark spark 5886 11月 4 2015 metrics.properties.template
-rw-r--r-- 1 spark spark 80 11月 4 2015 slaves.template
-rw-r--r-- 1 spark spark 507 11月 4 2015 spark-defaults.conf.template
-rwxr-xr-x 1 spark spark 3418 11月 4 2015 spark-env.sh.template*
spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6/conf$

spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6/conf$ ls
docker.properties.template fairscheduler.xml.template log4j.properties.template metrics.properties.template slaves.template spark-defaults.conf.template spark-env.sh.template
spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6/conf$ cp spark-env.sh.template spark-env.sh
spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6/conf$ ls
docker.properties.template log4j.properties.template slaves.template spark-env.sh
fairscheduler.xml.template metrics.properties.template spark-defaults.conf.template spark-env.sh.template
spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6/conf$ ll
total 44
drwxr-xr-x 2 spark spark 4096 9月 9 15:22 ./
drwxr-xr-x 12 spark spark 4096 11月 4 2015 ../
-rw-r--r-- 1 spark spark 202 11月 4 2015 docker.properties.template
-rw-r--r-- 1 spark spark 303 11月 4 2015 fairscheduler.xml.template
-rw-r--r-- 1 spark spark 949 11月 4 2015 log4j.properties.template
-rw-r--r-- 1 spark spark 5886 11月 4 2015 metrics.properties.template
-rw-r--r-- 1 spark spark 80 11月 4 2015 slaves.template
-rw-r--r-- 1 spark spark 507 11月 4 2015 spark-defaults.conf.template
-rwxr-xr-x 1 spark spark 3418 9月 9 15:22 spark-env.sh*
-rwxr-xr-x 1 spark spark 3418 11月 4 2015 spark-env.sh.template*
spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6/conf$ rm -rf spark-env.sh.template
spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6/conf$ ls
docker.properties.template fairscheduler.xml.template log4j.properties.template metrics.properties.template slaves.template spark-defaults.conf.template spark-env.sh
spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6/conf$ vim spark-env.sh

spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6/conf$ ls
docker.properties.template fairscheduler.xml.template log4j.properties.template metrics.properties.template slaves.template spark-defaults.conf.template spark-env.sh
spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6/conf$ cp slaves.template slaves
spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6/conf$ ll
total 44
drwxr-xr-x 2 spark spark 4096 9月 9 15:25 ./
drwxr-xr-x 12 spark spark 4096 11月 4 2015 ../
-rw-r--r-- 1 spark spark 202 11月 4 2015 docker.properties.template
-rw-r--r-- 1 spark spark 303 11月 4 2015 fairscheduler.xml.template
-rw-r--r-- 1 spark spark 949 11月 4 2015 log4j.properties.template
-rw-r--r-- 1 spark spark 5886 11月 4 2015 metrics.properties.template
-rw-r--r-- 1 spark spark 80 9月 9 15:25 slaves
-rw-r--r-- 1 spark spark 80 11月 4 2015 slaves.template
-rw-r--r-- 1 spark spark 507 11月 4 2015 spark-defaults.conf.template
-rwxr-xr-x 1 spark spark 3697 9月 9 15:25 spark-env.sh*
spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6/conf$ rm -rf slaves.template
spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6/conf$ ls
docker.properties.template fairscheduler.xml.template log4j.properties.template metrics.properties.template slaves spark-defaults.conf.template spark-env.sh
spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6/conf$ vim slaves

# A Spark Worker will be started on each of the machines listed below.
SparkSingleNode

　　　　至此，spark的配置工做完成！

　　7、啓動集羣

　　　　　　一、在haoop的安裝目錄下，啓動hadoop集羣。

　　　　　　　　/usr/local/hadoop/hadoop-2.6.0下，執行./sbin/start-all.sh

　　　　　　或，在任何路徑下，$HADOOP_HOME/sbin/start-all.sh

spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0$ pwd
/usr/local/hadoop/hadoop-2.6.0
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0$ jps
8970 Jps
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0$ ./sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [SparkSingleNode]
SparkSingleNode: starting namenode, logging to /usr/local/hadoop/hadoop-2.6.0/logs/hadoop-spark-namenode-SparkSingleNode.out
SparkSingleNode: starting datanode, logging to /usr/local/hadoop/hadoop-2.6.0/logs/hadoop-spark-datanode-SparkSingleNode.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/hadoop-2.6.0/logs/hadoop-spark-secondarynamenode-SparkSingleNode.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/hadoop-2.6.0/logs/yarn-spark-resourcemanager-SparkSingleNode.out
SparkSingleNode: starting nodemanager, logging to /usr/local/hadoop/hadoop-2.6.0/logs/yarn-spark-nodemanager-SparkSingleNode.out
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0$ jps
9408 SecondaryNameNode
9234 DataNode
9704 NodeManager
10014 Jps
9583 ResourceManager
9119 NameNode
spark@SparkSingleNode:/usr/local/hadoop/hadoop-2.6.0$

　二、在spark的安裝目錄下，啓動spark集羣。

　　　　/usr/local/spark/spark-1.5.2-bin-hadoop2.6下，執行./sbin/start-all.sh

　　　或，在任何路徑下，執行 $SPARK_HOME/sbin/start-all.sh

spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6$ pwd
/usr/local/spark/spark-1.5.2-bin-hadoop2.6
spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6$ jps
9408 SecondaryNameNode
9234 DataNode
9704 NodeManager
10602 Jps
9583 ResourceManager
9119 NameNode
spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6$ ./sbin/start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark/spark-1.5.2-bin-hadoop2.6/sbin/../logs/spark-spark-org.apache.spark.deploy.master.Master-1-SparkSingleNode.out
SparkSingleNode: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark/spark-1.5.2-bin-hadoop2.6/sbin/../logs/spark-spark-org.apache.spark.deploy.worker.Worker-1-SparkSingleNode.out
spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6$ jps
9408 SecondaryNameNode
10848 Worker
9234 DataNode
10678 Master
10887 Jps
9704 NodeManager
9583 ResourceManager
9119 NameNode
spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6$

　　由此，可見，hadoop的啓動、spark的啓動都正常！

　8、查看頁面

　　　　進入hadoop的hdfs的web頁面。訪問http://SparkSingleNode:50070 （安裝以後，當即能夠看到）

　　　　進入hadoop的yarn的web頁面。訪問http://SparkSingleNode:8088 （安裝以後，當即能夠看到）

　　　　進入spark的web頁面。訪問 http://SparkSingleNode:8080 （安裝以後，當即能夠看到）

　　　　進入spark的shell的web頁面。訪問http//:SparkSingleNode:4040 （需開啓spark shell）

　　咱們也能夠進入scala狀態下的spark

　　　　在spark的安裝目錄下，執行./bin/spark-shell 注意，沒空格

　　　　或者，在任何路徑下，執行 $SPARK_HOME/bin/spark-shell --SparkSingleNode spark://SparkSingleNode:7077 注意，$SPARK_HOME，沒空格

spark@SparkSingleNode:/usr/local/spark/spark-1.5.2-bin-hadoop2.6$ ./bin/spark-shell
16/09/09 16:26:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/09/09 16:26:09 INFO spark.SecurityManager: Changing view acls to: spark
16/09/09 16:26:09 INFO spark.SecurityManager: Changing modify acls to: spark
16/09/09 16:26:09 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)
16/09/09 16:26:10 INFO spark.HttpServer: Starting HTTP Server
16/09/09 16:26:11 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/09/09 16:26:11 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:41641
16/09/09 16:26:11 INFO util.Utils: Successfully started service 'HTTP class server' on port 41641.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.5.2
/_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_60)
Type in expressions to have them evaluated.
Type :help for more information.
16/09/09 16:26:45 INFO spark.SparkContext: Running Spark version 1.5.2
16/09/09 16:26:46 INFO spark.SecurityManager: Changing view acls to: spark
16/09/09 16:26:46 INFO spark.SecurityManager: Changing modify acls to: spark
16/09/09 16:26:46 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)
16/09/09 16:26:49 INFO slf4j.Slf4jLogger: Slf4jLogger started
16/09/09 16:26:50 INFO Remoting: Starting remoting

16/09/09 16:26:52 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.80.128:44949]
16/09/09 16:26:52 INFO util.Utils: Successfully started service 'sparkDriver' on port 44949.
16/09/09 16:26:52 INFO spark.SparkEnv: Registering MapOutputTracker
16/09/09 16:26:52 INFO spark.SparkEnv: Registering BlockManagerMaster
16/09/09 16:26:53 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-8615ba01-4240-4c41-b85d-ca901305577b
16/09/09 16:26:53 INFO storage.MemoryStore: MemoryStore started with capacity 534.5 MB
16/09/09 16:26:54 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-ee23c689-2009-455d-ba60-995a42ef529a/httpd-13fe8fa0-3850-48d9-88ba-c78bb78edd91
16/09/09 16:26:54 INFO spark.HttpServer: Starting HTTP Server
16/09/09 16:26:54 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/09/09 16:26:54 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:33301
16/09/09 16:26:54 INFO util.Utils: Successfully started service 'HTTP file server' on port 33301.
16/09/09 16:26:54 INFO spark.SparkEnv: Registering OutputCommitCoordinator
16/09/09 16:27:00 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/09/09 16:27:00 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
16/09/09 16:27:00 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
16/09/09 16:27:00 INFO ui.SparkUI: Started SparkUI at http://192.168.80.128:4040
16/09/09 16:27:02 WARN metrics.MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
16/09/09 16:27:02 INFO executor.Executor: Starting executor ID driver on host localhost
16/09/09 16:27:02 INFO executor.Executor: Using REPL class URI: http://192.168.80.128:41641
16/09/09 16:27:03 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 39805.
16/09/09 16:27:03 INFO netty.NettyBlockTransferService: Server created on 39805
16/09/09 16:27:03 INFO storage.BlockManagerMaster: Trying to register BlockManager
16/09/09 16:27:03 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:39805 with 534.5 MB RAM, BlockManagerId(driver, localhost, 39805)
16/09/09 16:27:04 INFO storage.BlockManagerMaster: Registered BlockManager
16/09/09 16:27:05 INFO repl.SparkILoop: Created spark context..
Spark context available as sc.
16/09/09 16:27:10 INFO hive.HiveContext: Initializing execution hive, version 1.2.1
16/09/09 16:27:13 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0
16/09/09 16:27:13 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
16/09/09 16:27:17 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
16/09/09 16:27:17 INFO metastore.ObjectStore: ObjectStore, initialize called
16/09/09 16:27:19 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
16/09/09 16:27:19 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
16/09/09 16:27:21 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/09/09 16:27:23 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/09/09 16:27:35 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
16/09/09 16:27:43 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/09/09 16:27:43 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/09/09 16:27:55 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/09/09 16:27:55 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/09/09 16:27:57 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
16/09/09 16:27:57 INFO metastore.ObjectStore: Initialized ObjectStore
16/09/09 16:27:58 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/09/09 16:27:59 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException
16/09/09 16:28:01 INFO metastore.HiveMetaStore: Added admin role in metastore
16/09/09 16:28:01 INFO metastore.HiveMetaStore: Added public role in metastore
16/09/09 16:28:02 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty
16/09/09 16:28:03 INFO metastore.HiveMetaStore: 0: get_all_databases
16/09/09 16:28:03 INFO HiveMetaStore.audit: ugi=spark ip=unknown-ip-addr cmd=get_all_databases
16/09/09 16:28:04 INFO metastore.HiveMetaStore: 0: get_functions: db=default pat=*
16/09/09 16:28:04 INFO HiveMetaStore.audit: ugi=spark ip=unknown-ip-addr cmd=get_functions: db=default pat=*
16/09/09 16:28:04 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
16/09/09 16:28:10 INFO session.SessionState: Created HDFS directory: /tmp/hive/spark
16/09/09 16:28:10 INFO session.SessionState: Created local directory: /tmp/spark
16/09/09 16:28:10 INFO session.SessionState: Created local directory: /tmp/2f4fd719-b203-418d-b492-8ed62d08a122_resources
16/09/09 16:28:11 INFO session.SessionState: Created HDFS directory: /tmp/hive/spark/2f4fd719-b203-418d-b492-8ed62d08a122
16/09/09 16:28:11 INFO session.SessionState: Created local directory: /tmp/spark/2f4fd719-b203-418d-b492-8ed62d08a122
16/09/09 16:28:11 INFO session.SessionState: Created HDFS directory: /tmp/hive/spark/2f4fd719-b203-418d-b492-8ed62d08a122/_tmp_space.db
16/09/09 16:28:11 INFO hive.HiveContext: default warehouse location is /user/hive/warehouse
16/09/09 16:28:11 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
16/09/09 16:28:12 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0
16/09/09 16:28:12 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
16/09/09 16:28:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/09/09 16:28:15 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
16/09/09 16:28:16 INFO metastore.ObjectStore: ObjectStore, initialize called
16/09/09 16:28:16 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
16/09/09 16:28:16 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
16/09/09 16:28:16 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/09/09 16:28:18 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/09/09 16:28:27 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
16/09/09 16:28:31 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/09/09 16:28:31 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/09/09 16:28:40 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/09/09 16:28:40 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/09/09 16:28:41 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
16/09/09 16:28:42 INFO metastore.ObjectStore: Initialized ObjectStore
16/09/09 16:28:42 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/09/09 16:28:43 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException
16/09/09 16:28:45 INFO metastore.HiveMetaStore: Added admin role in metastore
16/09/09 16:28:45 INFO metastore.HiveMetaStore: Added public role in metastore
16/09/09 16:28:45 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty
16/09/09 16:28:46 INFO metastore.HiveMetaStore: 0: get_all_databases
16/09/09 16:28:46 INFO HiveMetaStore.audit: ugi=spark ip=unknown-ip-addr cmd=get_all_databases
16/09/09 16:28:46 INFO metastore.HiveMetaStore: 0: get_functions: db=default pat=*
16/09/09 16:28:46 INFO HiveMetaStore.audit: ugi=spark ip=unknown-ip-addr cmd=get_functions: db=default pat=*
16/09/09 16:28:46 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
16/09/09 16:28:47 INFO session.SessionState: Created local directory: /tmp/64b097c1-1ac3-4f8b-94e5-6ce45fadf854_resources
16/09/09 16:28:47 INFO session.SessionState: Created HDFS directory: /tmp/hive/spark/64b097c1-1ac3-4f8b-94e5-6ce45fadf854
16/09/09 16:28:47 INFO session.SessionState: Created local directory: /tmp/spark/64b097c1-1ac3-4f8b-94e5-6ce45fadf854
16/09/09 16:28:47 INFO session.SessionState: Created HDFS directory: /tmp/hive/spark/64b097c1-1ac3-4f8b-94e5-6ce45fadf854/_tmp_space.db
16/09/09 16:28:47 INFO repl.SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.