記一次自定義Tomcat ClassLoader問題排查

問題描述

當前開發的項目須要隔離spark環境，所以自定義實現了SparkClassLoader。可是真正打包在服務器上運行的時候，應用須要初始化SparkSession，可是報出了以下錯誤：

20/04/09 16:57:02 ERROR SparkContext: Error initializing SparkContext.
java.lang.ClassCastException: org.apache.spark.serializer.JavaSerializer cannot be cast to org.apache.spark.serializer.Serializer
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:295)
    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:187)
    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:424)
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2523)
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
    at org.apache.spark.sql.SparderContext$$anonfun$initSpark$1$$anon$4.run(SparderContext.scala:128)
    at java.lang.Thread.run(Thread.java:748)

解決過程

1. 經過JVM -verbose:class參數查看兩個類加載的jar包，看一下是否是包衝突的問題

[Loaded org.apache.spark.serializer.Serializer from file:/root/wangrupeng/spark/jars/spark-core_2.11-2.4.1-os-kylin-r3.jar]
[Loaded org.apache.spark.serializer.JavaSerializer from file:/root/wangrupeng/spark/jars/spark-core_2.11-2.4.1-os-kylin-r3.jar]

結果發現兩個類都是從同一個jar包中加載的，排除依賴衝突的緣由。

1. 藉助阿里巴巴的強力工具Arthas

官方網址

curl -O https://alibaba.github.io/arthas/arthas-boot.jar
java -jar arthas-boot.jar
# 輸入項目的進程ID
sc classloader #發現咱們自定義的classloader有兩個實例
sc -d org.apache.spark.serializer.JavaSerializer # 不出意外，改類被兩個classloader實例分別加載了兩次

緣由找到了，是由於兩個類被兩個classloader實例加載了兩次，而後class cast的時候是兩個不一樣classloader加載的，因此致使了ClassCastException

1. 爲何classloader會初始化兩次呢？

因爲這個SparkClassLoader是咱們本身定義的，因此我在其構造函數中打印了一下Stack信息，這樣就可以看到這個類實例的初始化過程了

protected SparkClassLoader(ClassLoader parent) throws IOException {
        super(new URL[] {}, parent);
        init();
        Thread.dumpStack();
    }

最終日誌中相關輸出以下：

java.lang.Exception: Stack trace
        at java.lang.Thread.dumpStack(Thread.java:1336)
        at org.apache.kylin.spark.classloader.DebugTomcatClassLoader.<init>(DebugTomcatClassLoader.java:75)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.catalina.loader.WebappLoader.createClassLoader(WebappLoader.java:753)
        at org.apache.catalina.loader.WebappLoader.startInternal(WebappLoader.java:598)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5581)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1016)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:992)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:639)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1127)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:2020)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


java.lang.Exception: Stack trace
        at java.lang.Thread.dumpStack(Thread.java:1336)
        at org.apache.kylin.spark.classloader.DebugTomcatClassLoader.<init>(DebugTomcatClassLoader.java:75)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.catalina.loader.WebappLoader.createClassLoader(WebappLoader.java:753)
        at org.apache.catalina.loader.WebappLoader.startInternal(WebappLoader.java:598)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5581)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1016)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:992)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:639)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1296)
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2038)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

能夠看到是Tomcat在部署web實例的時候初始化的ClassLoader實例，而後重點關注，爲何deploy了兩次，分別是DeployDirectory和DeployWar各一次，可是經過查看Tomcat官方文檔能夠知道部署一個web應用這兩種方式只會選擇一個，但爲何出現了兩次？

1. 最終排查

最終發現是由於tomcat/webapp目錄下有兩個app目錄，刪掉一個沒有用的就能夠啦。