spark 報錯 InvalidClassException: no valid constructor

 1 2019-03-19 02:50:24 WARN  TaskSetManager:66 - Lost task 1.0 in stage 0.0 (TID 1, 1.2.3.4, executor 1): java.io.InvalidClassException: xxx.xxx.spark.xxx.xxx.Test; no valid constructor
 2         at java.io.ObjectStreamClass$ExceptionInfo.newInvalidClassException(ObjectStreamClass.java:157)
 3         at java.io.ObjectStreamClass.checkDeserialize(ObjectStreamClass.java:862)
 4         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2041)
 5         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
 6         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)
 7         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)
 8         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
 9         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
10         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)
11         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)
12         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
13         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
14         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)
15         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)
16         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
17         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
18         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)
19         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)
20         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
21         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
22         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)
23         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)
24         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
25         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
26         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
27         at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
28         at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
29         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:80)
30         at org.apache.spark.scheduler.Task.run(Task.scala:109)
31         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
32         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
33         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
34         at java.lang.Thread.run(Thread.java:748)

 

若是你執行spark的job時出現上面這個錯誤，同時知足如下條件

一、存在抽象父類

二、抽象父類中存在有參數的構造方法

三、子類實現了序列化接口，父類沒有實現

 

那麼恭喜你，來對地方了，解決方法很是簡單，實現抽象父類的默認無參數構造方法就能夠

下面代碼的第9行

 1 public abstract class AbBaseThread extends Thread {
 2 
 3     protected SparkSession session;
 4     
 5     protected JavaSparkContext jsc;
 6     
 7     private SparkParam sp;
 8 
 9     public AbBaseThread() {}
10     
11     public AbBaseThread(SparkSession session, SparkParam sp) {
12         this.session = session;
13         this.jsc = new JavaSparkContext(session.sparkContext());
14         this.sp = sp;
15     }
16     
17     public abstract String execute(SparkParam sp);
18     
19     public void run() {
20         String result = this.execute(sp);
21         Jedis jedis = RedisUtil.getJedis();
22         jedis.set(sp.getUuid(), result);
23         RedisUtil.returnResource(jedis);
24     }
25 
26 }

爲何會出現上面這個問題，我在這裏強行解釋一波：

一、子類序列化的時候，由於父類沒有實現序列化接口，因此序列化的流中不存在父類相關的信息

二、反序列化子類對象和直接實例化一個子類對象的原理應該差很少，因此在反序列子類的時候也是先有父類對象再有子類對象，父類怎麼來的，調用默認的無參構造函數來的。