【原創】大叔問題定位分享（8）提交spark任務報錯 Caused by: java.lang.ClassNotFoundException: org.I0Itec.zkclient.exceptio

spark 2.1.1

 

一 問題重現

spark-submit --master local[*] --class app.package.AppClass --jars /jarpath/zkclient-0.3.jar --driver-memory 1g app.jar

報錯

Java HotSpot(TM) 64-Bit Server VM warning: Setting CompressedClassSpaceSize has no effect when compressed class pointers are not used
Java HotSpot(TM) 64-Bit Server VM warning: Cannot open file /export/Logs/hadoop/g1gc.log due to No such file or directory

18/09/14 16:24:38 336 WARN Utils66: Set SPARK_LOCAL_IP if you need to bind to another address
Exception in thread "main" java.lang.NoClassDefFoundError: org/I0Itec/zkclient/exception/ZkNoNodeException
at kafka.utils.ZKGroupDirs.consumerDir(ZkUtils.scala:727)
at kafka.utils.ZKGroupDirs.consumerGroupDir(ZkUtils.scala:728)
at kafka.utils.ZKGroupTopicDirs.consumerOffsetDir(ZkUtils.scala:733)
at app.package.AppClass.main(AppClass.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.I0Itec.zkclient.exception.ZkNoNodeException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 15 more

 

二 問題解析

ClassNotFoundException: org.I0Itec.zkclient.exception.ZkNoNodeException

1 --jars --jars /jarpath/zkclient-0.3.jar

/jarpath/zkclient-0.3.jar經過--jars參數傳遞給spark應用

2 類org.I0Itec.zkclient.exception.ZkNoNodeException確實存在/jarpath/zkclient-0.3.jar中

javap -cp /jarpath/zkclient-0.3.jar org.I0Itec.zkclient.exception.ZkNoNodeException
Compiled from "ZkNoNodeException.java"
public class org.I0Itec.zkclient.exception.ZkNoNodeException extends org.I0Itec.zkclient.exception.ZkException {
public org.I0Itec.zkclient.exception.ZkNoNodeException();
public org.I0Itec.zkclient.exception.ZkNoNodeException(org.apache.zookeeper.KeeperException);
public org.I0Itec.zkclient.exception.ZkNoNodeException(java.lang.String, org.apache.zookeeper.KeeperException);
public org.I0Itec.zkclient.exception.ZkNoNodeException(java.lang.String);
}

3 --verbose參數輸出

Main class:
app.pacakge.AppClass
Arguments:

System properties:
spark.executor.logs.rolling.maxSize -> 1073741824
spark.driver.memory -> 1g
spark.driver.extraLibraryPath -> /export/App/hadoop-2.6.1/lib/native
spark.eventLog.enabled -> true
spark.eventLog.compress -> true
spark.executor.logs.rolling.time.interval -> daily
SPARK_SUBMIT -> true
spark.app.name -> app.pacakge.AppClass
spark.driver.extraJavaOptions -> -XX:+PrintGCDetails -XX:+UseG1GC -XX:G1HeapRegionSize=32M -XX:+UseGCOverheadLimit -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-UseCompressedClassPointers -XX:CompressedClassSpaceSize=3G -XX:+PrintGCTimeStamps -Xloggc:/export/Logs/hadoop/g1gc.log
spark.jars -> file:/jarpath/zkclient-0.3.jar,file:/jarpath/app.jar
spark.sql.adaptive.enabled -> true
spark.submit.deployMode -> client
spark.executor.logs.rolling.maxRetainedFiles -> 10
spark.executor.extraClassPath -> /usr/lib/hadoop/lib/hadoop-lzo.jar:/export/App/spark/extlib/*
spark.eventLog.dir -> hdfs://hdfs_name/spark/history
spark.master -> local[*]
spark.sql.crossJoin.enabled -> true
spark.driver.extraClassPath -> /usr/lib/hadoop/lib/hadoop-lzo.jar:/export/App/spark/extlib/*
Classpath elements:
file:/jarpath/app.jar
file:/jarpath/zkclient-0.3.jar

可見/jarpath/zkclient-0.3.jar確實存在classpath中

綜上3點，爲何還會報ClassNotFoundException，這裏須要兩個背景知識：

一個是spark submit過程，詳見 http://www.javashuo.com/article/p-gzsdbzra-bo.html

一個是java的classloader，詳見 http://www.javashuo.com/article/p-ukrereus-hk.html

 

三 問題回放過程

1 spark-submit
2 spark-class
3 org.apache.spark.launcher.Main
4 org.apache.spark.deploy.SparkSubmit
4.1 初始化MutableURLClassLoader，將--jars和應用jar做爲classpath添加到MutableURLClassLoader中（注意此時zkclient-0.3.jar在這裏），而後經過MutableURLClassLoader來加載應用class，也就是AppClass，而後反射調用main
4.2 AppClass.main中間接調用kafka.utils.ZKGroupDirs（kafka jar位於spark.driver.extraClassPath中），ZKGroupDirs由MutableURLClassLoader的父classloader，也就是AppClassLoader來加載，kafka.utils.ZKGroupDirs中依賴org.I0Itec.zkclient.exception.ZkNoNodeException，也由AppClassLoader來加載，並非由MutableURLClassLoader來加載，因此報加載不到，雖然看起來zkclient-0.3.jar確實在classpath中；

問題的解決須要將zkclient-0.3.jar添加到AppClassLoader的classpath中（即spark.driver.extraClassPath）