Spark 1.4.1 Standalone 模式部署安裝配置

各節點執行以下操做（或在一個節點上操做完後 scp 到其它節點）：

一、 解壓spark安裝程序到程序目錄/bigdata/soft/spark-1.4.1,約定此目錄爲$SPARK_HOME

        tar –zxvf spark-1.4-bin-hadoop2.6.tar.gz

二、 配置spark，（scala 的安裝這裏就不講了）

• 配置文件vi $SPARK_HOME /conf/spark-env.sh

###添加以下內容：

export JAVA_HOME=/bigdata/soft/jdk1.7.0_79

export SCALA_HOME=/bigdata/soft/scala-2.10.5

export HADOOP_CONF_DIR=/bigdata/soft/hadoop-2.6.0/etc/hadoop

export SPARK_MASTER_IP=cloud-001

#export SPARK_MASTER_PORT=7077

export SPARK_WORKER_MEMORY=1g

export SPARK_WORKER_CORES=1

export SPARK_WORKER_INSTANCES=1

export SPARK_CLASSPATH=$SPARK_CLASSPATH:/bigdata/soft/spark-1.4.1/lib/mysql-connector-java-5.1.31.jar

• 配置vi $SPARK_HOME /conf/slaves

##根據集羣節點設置slave節點

cloud-002

cloud-003

• 配置vi $SPARK_HOME /conf/spark-defaults.conf

 ##先在hdfs上新建spark的日誌目錄

$Hadoop_HOME/bin/hadoop fs –mkdir /applogs

$Hadoop_HOME/bin/hadoop fs –mkdir /applogs/spark

 

##複製一個spark的配置文件

cp spark-defaults.conf.template spark-defaults.conf

##解注掉其中兩行

spark.master                    spark://cloud-001:7077

spark.eventLog.enabled          true

spark.eventLog.dir               hdfs://cloud-001:8020/applogs/spark

 

• 配置vi $SPARK_HOME /conf/hive-site.xml

###內容基本與hive的配置一致，詳見以下:

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

  <property>

    <name>javax.jdo.option.ConnectionURL</name>

    <value>jdbc:mysql://localhost:3306/hive_1_2_0?createDatabaseIfNotExist=true</value>

  </property>

  <property>

    <name>javax.jdo.option.ConnectionDriverName</name>

    <value>com.mysql.jdbc.Driver</value>

  </property>

  <property>

    <name>javax.jdo.PersistenceManagerFactoryClass</name>

    <value>org.datanucleus.api.jdo.JDOPersistenceManagerFactory</value>

  </property>

  <property>

    <name>javax.jdo.option.DetachAllOnCommit</name>

    <value>true</value>

  </property>

  <property>

    <name>javax.jdo.option.NonTransactionalRead</name>

    <value>true</value>

  </property>

  <property>

    <name>javax.jdo.option.ConnectionUserName</name>

    <value>root</value>

  </property>

  <property>

    <name>javax.jdo.option.ConnectionPassword</name>

    <value>abc123</value>

  </property>

  <property>

    <name>javax.jdo.option.Multithreaded</name>

    <value>true</value>

  </property>

  <property>

    <name>datanucleus.connectionPoolingType</name>

    <value>BoneCP</value>

  </property>

  <property>

    <name>hive.metastore.warehouse.dir</name>

    <value>/user/hive/warehouse</value>

  </property>

  <property>

      <name>fs.default.name</name>

      <value>hdfs://cloud-001:8020</value>

  </property>

  <property>

    <name>hive.server2.thrift.port</name>

    <value>10000</value>

  </property>

  <property>

    <name>hive.server2.thrift.bind.host</name>

    <value>cloud-001</value>

  </property>

</configuration>

• 複製一個mysql的jdbc驅動到$SPARK_HOME/lib

如cp $HIVE_HOME/lib/mysql-connector-java-5.1.31.jar $SPARK_HOME/lib

三、 standlone 模式啓動集羣

        啓動master和worker：

                    $SPARK_HOME/sbin/start-all.sh

        啓動spark的hive服務

                    $SPARK_HOME/sbin/start-thriftserver.sh --master spark://cloud-001:7077 --driver-memory 1g  --executor-memory 1g --total-executor-cores 2

四、 測試

測試spark-shell

    $SPARK_HOME/bin/spark-shell --master spark://cloud-001:7077 --driver-memory 1g  --executor-memory 1g --total-executor-cores 2

測試spark-submit

            $SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://cloud-001:7077 --executor-memory 1G --total-executor-cores 2 $SPARK_HOME/lib/spark-examples-1.4.1-hadoop2.6.0.jar 10000

    測試spark-sql

           $SPARK_HOME/bin/spark-sql --master spark://cloud-001:7077 --driver-memory 1g  --executor-memory 1g --total-executor-cores 2

    測試beeline

    $SPARK_HOME/bin/beeline -u jdbc:hive2://cloud-001:10000 -n hadoop