【原創】大叔問題定位分享（9）oozie提交spark任務報 java.lang.NoClassDefFoundError: org/apache/kafka/clients/producer/Kaf

時間 2019-11-21

標籤原創問題定位分享 oozie 提交 spark 任務 java.lang.noclassdeffounderror java lang noclassdeffounderror org apache kafka clients producer kaf 欄目系統網絡简体版

原文原文鏈接

oozie中支持不少的action類型，好比spark、hive，對應的標籤爲：tcp

<spark xmlns="uri:oozie:spark-action:0.1"> oop

...spa

oozie中sharelib用於存放每一個action類型須要的依賴，能夠查看當前全部的action類型以及每一個action類型的依賴xml

oozie admin -shareliblist
[Available ShareLib]
hive
sparkbak
distcp
mapreduce-streaming
spark
oozie
hcatalog
hive2
sqoop
pigget

oozie admin -shareliblist sparkkafka

...spark

上述報錯是因爲oozie中spark依賴裏的kafka基礎依賴缺乏kafka-clients-0.8.2.2.jar，將jar上傳到sharelib目錄，而後手動更新便可io

oozie admin -sharelibupdatestream

更新以後可見：基礎

oozie admin -shareliblist spark|grep kafka
        hdfs://hdfs_name/oozie/share/lib_20180801121138/spark/kafka-clients-0.8.2.2.jar
        hdfs://hdfs_name/oozie/share/lib_20180801121138/spark/kafka_2.11-0.8.2.1.jar
        hdfs://hdfs_name/oozie/share/lib_20180801121138/spark/spark-streaming-kafka-0-8_2.11-2.1.0.jar

添加成功