spark在yarn-cluster上面執行報錯

在單機模式下執行成功的spark程序，在yarn上面就報錯。異常信息以下：

 1 14/08/14 02:05:42 INFO DAGScheduler: Completed ResultTask(2, 0)
 2 14/08/14 02:05:42 INFO DAGScheduler: Stage 2 (saveAsTextFile at FileUtil.scala:114) finished in 0.179 s
 3 14/08/14 02:05:42 INFO SparkContext: Job finished: saveAsTextFile at FileUtil.scala:114, took 0.331739293 s
 4 14/08/14 02:05:42 INFO TaskSetManager: Finished TID 2 in 184 ms on localhost (progress: 1/1)
 5 14/08/14 02:05:42 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 
 6 14/08/14 02:05:42 INFO ApplicationMaster: finishApplicationMaster with SUCCEEDED
 7 Exception in thread "main" java.lang.AssertionError: assertion failed
 8     at scala.Predef$.assert(Predef.scala:165)
 9     at org.apache.spark.deploy.yarn.ApplicationMaster.waitForSparkContextInitialized(ApplicationMaster.scala:222)
10     at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:111)
11     at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:469)
12     at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:53)
13     at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:52)
14     at java.security.AccessController.doPrivileged(Native Method)
15     at javax.security.auth.Subject.doAs(Subject.java:415)
16     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
17     at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:52)
18     at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:468)
19     at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
20 14/08/14 02:05:44 INFO ApplicationMaster: AppMaster received a signal.
21 14/08/14 02:05:44 INFO ApplicationMaster: Deleting staging directory .sparkStaging/application_1408004797389_0007

從日誌上面分析，job執行成功了。但貌似是沒有獲取到yarn返回的結果信息。

debug後發現是下面的問題：

spark-submit --class org.andy.hadoop.ETL --master yarn-cluster  ../lib/rdbms-0.0.1-SNAPSHOT-jar-with-dependencies.jar /dest/ETL2

job以yarn-cluster形式執行，但代碼中初始化的爲：

1  var conf = new SparkConf().setAppName("testFilter").setMaster("local")
2  var sc = new SparkContext(conf)

以local的形式初始化的。因此接收不到yarn的返回結果。修改後：

1     var conf = new SparkConf().setAppName("testFilter").setMaster("yarn-cluster")
2     var sc = new SparkContext(conf)

執行成功！