Spark 從零到開發（二）Spark安裝和集羣搭建

準備工做

首先得安裝scala：CentOS7.x 安裝scala

僞集羣搭建沒作過的參考：CentOS7.x Hadoop集羣搭建

下載解壓

配置

1. 配置環境變量

/etc/profile

export SPARK_HOME=/home/fantj/spark
export PATH=$PATH:$SPARK_HOME/bin
export CLASSPAHT=.:$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
複製代碼

2. 配置/conf/spark-env.sh

cp spark-env.sh.template spark-env.sh

給尾部添加環境變量：

export JAVA_HOME=/home/fantj/jdk
export SCALA_HOME=/home/fantj/scala
export SPARK_MASTER_IP=s166
export SPARK_WORKER_MEMORY=1g
export HADOOP_CONF_DIR=/home/fantj/hadoop/etc/hadoop
複製代碼

3. 配置/conf/slaves.conf

cp slaves.template slaves.conf

新添數據：

spark2
spark3
spark4
複製代碼

同步配置到slave節點

將spark和scala 和配置文件拷貝到每一個slave節點。

1099  scp -r scala-2.11.7 spark-1.5.1-bin-hadoop2.4/ s168:/home/fantj/download/
 1100  scp -r scala-2.11.7 spark-1.5.1-bin-hadoop2.4/ s169:/home/fantj/download/

 1135  scp /etc/profile s167:/etc/profile
 1136  scp /etc/profile s168:/etc/profile
 1137  scp /etc/profile s169:/etc/profile
複製代碼

啓動spark

1. 首先得啓動hadoop或者只啓動hdfs。start-dfs.sh命令。

2. jps查看並確保主從機的hadoop的dfs都啓動後。（主：NameNode，從：DataNode）

3. 在spark的根目錄下執行./sbin/start-all.sh，若是想要slave節點也跟着啓動，須要作免密碼登陸。沒有作的話能夠用相同的命令一個一個節點去啓動。

[root@s166 spark]# ./sbin/start-all.sh 
starting org.apache.spark.deploy.master.Master, logging to /home/fantj/download/spark-1.5.1-bin-hadoop2.4/sbin/../logs/spark-root-org.apache.spark.deploy.master.Master-1-s166.out
localhost: starting org.apache.spark.deploy.worker.Worker, logging to /home/fantj/download/spark-1.5.1-bin-hadoop2.4/sbin/../logs/spark-root-org.apache.spark.deploy.worker.Worker-1-s166.out
localhost: starting org.apache.spark.deploy.worker.Worker, logging to /home/fantj/download/spark-1.5.1-bin-hadoop2.4/sbin/../logs/spark-root-org.apache.spark.deploy.worker.Worker-1-s167.out
localhost: starting org.apache.spark.deploy.worker.Worker, logging to /home/fantj/download/spark-1.5.1-bin-hadoop2.4/sbin/../logs/spark-root-org.apache.spark.deploy.worker.Worker-1-s168.out
localhost: starting org.apache.spark.deploy.worker.Worker, logging to /home/fantj/download/spark-1.5.1-bin-hadoop2.4/sbin/../logs/spark-root-org.apache.spark.deploy.worker.Worker-1-s169.out
複製代碼

4. 再查看jps

-------s166 jps -------
1397 NameNode
52854 Worker
1559 SecondaryNameNode
53671 Jps
52719 Master
-------s167 jps -------
1764 DataNode
29092 Jps
28414 Worker
-------s168 jps -------
33921 Worker
1756 DataNode
34063 Jps
-------s169 jps -------
27384 Jps
1754 DataNode
27242 Worker
複製代碼

能夠看到，一個Master三個Worker。 而後再訪問主節點ip的8080端口。

打開Spark-shell

[root@s166 bin]# spark-shell 
18/07/30 12:34:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/07/30 12:34:20 INFO spark.SecurityManager: Changing view acls to: root
18/07/30 12:34:20 INFO spark.SecurityManager: Changing modify acls to: root
18/07/30 12:34:20 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
18/07/30 12:34:22 INFO spark.HttpServer: Starting HTTP Server
18/07/30 12:34:23 INFO server.Server: jetty-8.y.z-SNAPSHOT
18/07/30 12:34:23 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:35005
18/07/30 12:34:23 INFO util.Utils: Successfully started service 'HTTP class server' on port 35005.
...
...
18/07/30 12:38:39 INFO session.SessionState: Created local directory: /tmp/2c350bb0-1297-40d8-a9bd-47446b116bf3_resources
18/07/30 12:38:39 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/2c350bb0-1297-40d8-a9bd-47446b116bf3
18/07/30 12:38:39 INFO session.SessionState: Created local directory: /tmp/root/2c350bb0-1297-40d8-a9bd-47446b116bf3
18/07/30 12:38:40 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/2c350bb0-1297-40d8-a9bd-47446b116bf3/_tmp_space.db
18/07/30 12:38:40 INFO repl.SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.

scala> 
複製代碼

這就證實開啓成功了，同理訪問4040端口。