spark-submit

spark/bin/spark-submit \
--master yarn \
--deploy-mode cluster \
--class com.xp.wordcount.WordCount2 \
/usr/local/devtools/spark/testJar/sparkDemo.jar



spark/bin/spark-submit \
--class com.xp.wordcount.WordCount2 \
/usr/local/devtools/spark/testJar/sparkDemo.jar


#use --files ----driver-class-path for example
/usr/local/devtools/spark/spark/bin/spark-submit --class com.xp.sql1.DataFrame --files /usr/local/devTools/hive/hive-site.xml --driver-class-path /usr/local/devtools/spark/spark/test/jar/mysql-connector-java-5.2.35.jar  --master spark://xupan001:7070 /usr/local/devtools/spark/testJar/sparkDemo.jar

 /usr/local/devtools/spark/spark/bin/spark-submit \
 --class com.xp.sql1.DataFrame \ 
 --files /usr/local/devTools/hive/hive-site.xml \
 --driver-class-path /usr/local/devtools/spark/spark/test/jar/mysql-connector-java-5.2.35.jar \
 --master spark://xupan001:7070 \
 /usr/local/devtools/spark/testJar/sparkDemo.jar




/usr/local/devtools/spark/spark/bin/spark-submit --class com.xp.sql1.DataFrame --master spark://xupan001:7070 /usr/local/devtools/spark/testJar/sparkDemo.jar

 /usr/local/devtools/spark/spark/bin/spark-submit \
 --class com.xp.sql1.DataFrame \
 --master spark://xupan001:7070 \
 /usr/local/devtools/spark/testJar/sparkDemo.jar




#使用獨立集羣模式調用spark-submit
 /usr/local/devtools/spark/spark/bin/spark-submit \
 --master spark://xupan001:7070 \
 --deploy-model cluster \
 --class com.xp.sql1.DataFrame \
 --name "AppName" \
 --jars dep1.jar,dep1.jar,dep1.jar,dep1.jar \
 --total-executor-cores 4 \
 --executor-memory 2G
 --files /usr/local/devTools/hive/hive-site.xml \
 --driver-class-path /usr/local/devtools/spark/spark/test/jar/mysql-connector-java-5.2.35.jar \
 --master spark://xupan001:7070 \
 /usr/local/devtools/spark/testJar/sparkDemo.jar



/usr/local/devtools/spark/spark/bin/spark-submit --jars joda-time-2.9.jar,mysql-connector-java-5.1.34.jar --class broadcast.AccumulatorDemo --master spark://xupan001:7070 /usr/local/devtools/spark/testJar/sparkDemo.jar


-------------------------------------------------------------------------


Spark應用依賴第三方Jar
 ./bin/spark-shell  --help 
 ./bin/spark-submit --help

 spark-shell和spark-submit參數是同樣的

 Options:
  --master MASTER_URL         spark://host:port, mesos://host:port, yarn, or local.
  --deploy-mode DEPLOY_MODE   Whether to launch the driver program locally ("client") or
                              on one of the worker machines inside the cluster ("cluster")
                              (Default: client).
  --class CLASS_NAME          Your application's main class (for Java / Scala apps).
  --name NAME                 A name of your application.
  --jars JARS                 Comma-separated list of local jars to include on the driver
                              and executor classpaths.
  --packages                  Comma-separated list of maven coordinates of jars to include
                              on the driver and executor classpaths. Will search the local
                              maven repo, then maven central and any additional remote
                              repositories given by --repositories. The format for the
                              coordinates should be groupId:artifactId:version.
  --exclude-packages          Comma-separated list of groupId:artifactId, to exclude while
                              resolving the dependencies provided in --packages to avoid
                              dependency conflicts.
  --repositories              Comma-separated list of additional remote repositories to
                              search for the maven coordinates given with --packages.
  --py-files PY_FILES         Comma-separated list of .zip, .egg, or .py files to place
                              on the PYTHONPATH for Python apps.
  --files FILES               Comma-separated list of files to be placed in the working
                              directory of each executor.

  --conf PROP=VALUE           Arbitrary Spark configuration property.
  --properties-file FILE      Path to a file from which to load extra properties. If not
                              specified, this will look for conf/spark-defaults.conf.

  --driver-memory MEM         Memory for driver (e.g. 1000M, 2G) (Default: 1024M).memory  driver進程使用的內存數
  --driver-java-options       Extra Java options to pass to the driver.
  --driver-library-path       Extra library path entries to pass to the driver.
  --driver-class-path         Extra class path entries to pass to the driver. Note that
                              jars added with --jars are automatically included in the
                              classpath.

  --executor-memory MEM       Memory per executor (e.g. 1000M, 2G) (Default: 1G).

  --proxy-user NAME           User to impersonate when submitting the application.

  --help, -h                  Show this help message and exit
  --verbose, -v               Print additional debug output
  --version,                  Print the version of current Spark

 Spark standalone with cluster deploy mode only:
  --driver-cores NUM          Cores for driver (Default: 1).

 Spark standalone or Mesos with cluster deploy mode only:
  --supervise                 If given, restarts the driver on failure.
  --kill SUBMISSION_ID        If given, kills the driver specified.
  --status SUBMISSION_ID      If given, requests the status of the driver specified.

 Spark standalone and Mesos only:
  --total-executor-cores NUM  Total cores for all executors.

 Spark standalone and YARN only:
  --executor-cores NUM        Number of cores per executor. (Default: 1 in YARN mode,
                              or all available cores on the worker in standalone mode)

 YARN-only:
  --driver-cores NUM          Number of cores used by the driver, only in cluster mode  driver程序運行須要的cpu內核數
                              (Default: 1).
  --queue QUEUE_NAME          The YARN queue to submit to (Default: "default").
  --num-executors NUM         Number of executors to launch (Default: 2).
  --archives ARCHIVES         Comma separated list of archives to be extracted into the
                              working directory of each executor.
  --principal PRINCIPAL       Principal to be used to login to KDC, while running on
                              secure HDFS.
  --keytab KEYTAB             The full path to the file that contains the keytab for the
                              principal specified above. This keytab will be copied to
                              the node running the Application Master via the Secure
                              Distributed Cache, for renewing the login tickets and the
                              delegation tokens periodically.



主網址 含義
local 使用一個工做線程在本地運行Spark（即徹底沒有並行）。
local[K]  使用K工做線程本地運行Spark（理想狀況下，將其設置爲機器上的核心數）。
local[K,F]  使用K工做線程和F maxFailures在本地運行Spark（有關此變量的解釋，請參閱spark.task.maxFailures）
local[*]  使用與本機邏輯內核同樣多的工做線程在本地運行Spark。
local[*,F]  使用與本機上的邏輯內核和F maxFailures同樣多的工做線程在本地運行Spark。
spark://HOST:PORT 鏈接到給定的Spark獨立羣集主機。該端口必須是主設備配置使用的端口，默認爲7077。
spark://HOST1:PORT1,HOST2:PORT2 使用Zookeeper鏈接到具備備用主站的給定Spark獨立羣集。該列表必須包含使用Zookeeper設置的高可用性羣集中的全部主控主機。該端口必須是每一個主設備配置使用的，默認爲7077。



--jars JARS                  Comma-separated list of local jars to include on the driver and executor classpaths.【逗號分隔的jar文件列表】             


--driver-class-path         Extra class path entries to pass to the driver. Note that jars added with --jars are automatically included in the  classpath.


--files FILES               Comma-separated list of files to be placed in the working directory of each executor. 配置文件的路徑，通常爲hive-site.xml


--properties-file FILE      Path to a file from which to load extra properties. If not specified, this will look for conf/spark-defaults.conf.將配置屬性放到文件中，啓動