【原創】大叔問題定位分享（2）spark任務必定概率報錯java.lang.NoSuchFieldError: HIVE_MOVE_FILES_THREAD_COUNT

最近用yarn cluster方式提交spark任務時，有時會報錯，報錯概率是40%，報錯以下：

18/03/15 21:50:36 116 ERROR ApplicationMaster91: User class threw exception: org.apache.spark.sql.AnalysisException: java.lang.NoSuchFieldError: HIVE_MOVE_FILES_THREAD_COUNT;

org.apache.spark.sql.AnalysisException: java.lang.NoSuchFieldError: HIVE_MOVE_FILES_THREAD_COUNT;

         at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)

         at org.apache.spark.sql.hive.HiveExternalCatalog.loadTable(HiveExternalCatalog.scala:766)

         at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult$lzycompute(InsertIntoHiveTable.scala:374)

         at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult(InsertIntoHiveTable.scala:221)

         at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.doExecute(InsertIntoHiveTable.scala:407)

         at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)

         at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)

         at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)

         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

         at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)

         at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)

         at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)

         at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)

         at org.apache.spark.sql.Dataset.<init>(Dataset.scala:185)

         at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)

         at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)

         at scala.util.control.Breaks.breakable(Breaks.scala:38)

         at app.package.APPClass$.main(APPClass.scala:177)

         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

         at java.lang.reflect.Method.invoke(Method.java:497)

         at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:637)

Caused by: java.lang.NoSuchFieldError: HIVE_MOVE_FILES_THREAD_COUNT

         at org.apache.hadoop.hive.ql.metadata.Hive.trashFilesUnderDir(Hive.java:1389)

         at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:2873)

         at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1621)

         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

         at java.lang.reflect.Method.invoke(Method.java:497)

         at org.apache.spark.sql.hive.client.Shim_v0_14.loadTable(HiveShim.scala:728)

         at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply$mcV$sp(HiveClientImpl.scala:676)

         at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply(HiveClientImpl.scala:676)

         at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply(HiveClientImpl.scala:676)

         at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:279)

         at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:226)

         at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:225)

         at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:268)

         at org.apache.spark.sql.hive.client.HiveClientImpl.loadTable(HiveClientImpl.scala:675)

         at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadTable$1.apply$mcV$sp(HiveExternalCatalog.scala:768)

         at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadTable$1.apply(HiveExternalCatalog.scala:766)

         at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadTable$1.apply(HiveExternalCatalog.scala:766)

         at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)

         ... 25 more

大概流程是spark sql在執行InsertIntoHiveTable時會調用loadTable，這個操做最終會經過反射調用hive代碼的loadTable方法

1. org.apache.spark.sql.hive.execution.InsertIntoHiveTable.doExecute(InsertIntoHiveTable.scala:407)
2. org.apache.spark.sql.hive.HiveExternalCatalog.loadTable(HiveExternalCatalog.scala:766)
3. org.apache.spark.sql.hive.client.Shim_v0_14.loadTable(HiveShim.scala:728)
4. java.lang.reflect.Method.invoke(Method.java:497)
5. org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1621)
6. org.apache.hadoop.hive.ql.metadata.Hive.trashFilesUnderDir(Hive.java:1389)

在第6步中報錯 java.lang.NoSuchFieldError: HIVE_MOVE_FILES_THREAD_COUNT

 

這個問題一般會認爲是hive-site.xml缺乏配置

  <property>

    <name>hive.mv.files.thread</name>

    <value>15</value>

  </property>

可是查看代碼會發現spark2.1.1依賴的是hive1.2.1，在hive1.2.1中是沒有hive.mv.files.thread這個配置的，這個配置從hive2纔開始出現，並且報錯的類org.apache.hadoop.hive.ql.metadata.Hive在hive1.2.1和hive2的相關代碼徹底不一樣，具體分寫以下：

 

在hive1.2.1的代碼是：（trashFilesUnderDir方法是FileUtils類的）

            if (FileUtils.isSubDir(oldPath, destf, fs2)) {

              FileUtils.trashFilesUnderDir(fs2, oldPath, conf);

            }

在hive2的代碼是：（trashFilesUnderDir方法是Hive類的）

  private boolean trashFilesUnderDir(final FileSystem fs, Path f, final Configuration conf)

      throws IOException {

    FileStatus[] statuses = fs.listStatus(f, FileUtils.HIDDEN_FILES_PATH_FILTER);

    boolean result = true;

    final List<Future<Boolean>> futures = new LinkedList<>();

    final ExecutorService pool = conf.getInt(ConfVars.HIVE_MOVE_FILES_THREAD_COUNT.varname, 25) > 0 ?

        Executors.newFixedThreadPool(conf.getInt(ConfVars.HIVE_MOVE_FILES_THREAD_COUNT.varname, 25),

        new ThreadFactoryBuilder().setDaemon(true).setNameFormat("Delete-Thread-%d").build()) : null;

 

因此第6步的報錯，執行的應該是hive2的代碼，因此猜想問題多是：

1）因爲jar包污染，啓動jvm的時候classpath裏同時有hive1和hive2的jar，有時加載類用到hive1的jar（能夠正常運行），有時用到hive2的jar（會報錯）；

2）集羣服務器環境配置差別，有的服務器classpath中沒有hive2的jar（能夠正常運行），有的服務器classpath有hive2的jar（可能報錯）；

 

對比正常和報錯的服務器的環境配置以及啓動命令發現都是同樣的，沒有發現hive2的jar，

經過在啓動任務時增長-verbose:class，發現正常和報錯的狀況下，Hive類都是從Hive1.2.1的jar加載出來的，

[Loaded org.apache.hadoop.hive.ql.metadata.Hive from file:/export/Data/tmp/hadoop-tmp/nm-local-dir/filecache/98/hive-exec-1.2.1.spark2.jar]

否認了上邊的兩種猜想；

 

分析提交任務命令發現，用到了spark.yarn.jars，避免每次都上傳spark的jar，這些jar會被做爲filecache緩存在yarn.nodemanager.local-dirs下，

反編譯正常和報錯服務器上filecache裏的hive-exec-1.2.1.spark2.jar最終發現問題，

正常服務器上Hive類代碼是：

    if (FileUtils.isSubDir(oldPath, destf, fs2))

        FileUtils.trashFilesUnderDir(fs2, oldPath, conf);

報錯服務器上的Hive類代碼是：

    private static boolean trashFilesUnderDir(final FileSystem fs, Path f, final Configuration conf) throws IOException {

        FileStatus[] statuses = fs.listStatus(f, FileUtils.HIDDEN_FILES_PATH_FILTER);

        boolean result = true;

        List<Future<Boolean>> futures = new LinkedList();

        ExecutorService pool = conf.getInt(ConfVars.HIVE_MOVE_FILES_THREAD_COUNT.varname, 25) > 0 ? Executors.newFixedThreadPool(conf.getInt(ConfVars.HIVE_MOVE_FILES_THREAD_COUNT.varname, 25), (new ThreadFactoryBuilder()).setDaemon(true).setNameFormat("Delete-Thread-%d").build()) : null;

報錯服務器上的Hive類用到ConfVars.HIVE_MOVE_FILES_THREAD_COUNT，可是在hive-common-1.2.1.jar中的ConfVars不存在這個屬性，因此報錯java.lang.NoSuchFieldError，

因此問題應該是hdfs上的hive-exec-1.2.1.spark2.jar一開始是對的，而後全部nodemanager下載到本地做爲filecache，後來這個jar被改錯了（使用hive2編譯spark），而後新加的nodemanager會下載有問題的jar做爲filecache，這樣結果就是有的服務器執行正常，有的服務器執行報錯；

 

yarn中的filecache清理有兩個配置

yarn.nodemanager.localizer.cache.cleanup.interval-ms：600000  Interval in between cache cleanups.

yarn.nodemanager.localizer.cache.target-size-mb：10240    Target size of localizer cache in MB, per local directory.

每隔cleanup.interval-ms會檢查本地filecache大小是否超過target-size-mb，超過才清理，不超過就一直使用filecache；