4）spark集羣搭建

1.安裝spark

解壓縮(/opt)

tar zxvf spark-1.6.1-bin-hadoop2.6.tgz
mv spark-1.6.1-bin-hadoop2.6 spark

設置環境變量

vi ~/.bashrc

    export SPARK_HOME=/opt/spark

    export PATH=$SPARK_HOME/bin

    export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

source ~/.bashrc

2.修改spark-env.sh

cd /opt/spark/conf
mv spark-env.sh.template spark-env.sh
vi spark-env.sh

    export JAVA_HOME=/usr/jdk1.7.0_55

    export SCALA_HOME=/opt/scala

    export SPARK_MASTER_IP=192.168.252.164

    export SPARK_WORKER_MEMORY=1g

    export HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop

3.修改slaves

    hadoop001

    hadoop002

    hadoop003

4.二三節點

複製spark目錄到二三節點

cd /opt
scp -r spark hadoop002:/opt
scp -r spark hadoop003:/opt

複製環境變量到二三節點

scp ~/.bashrc hadoop002:~
scp ~/.bashrc hadoop003:~

5.啓動

啓動spark集羣

    start-all.sh

驗證

    jsp或8080端口驗證集羣啓動成功

    hadoop001:master

    hadoop002:worker

    hadoop003:worker

    進入spark-shell查看是否正常