Storm常見問題處理

時間 2019-11-13

原文原文鏈接

錯誤 1 ：發佈topologies到遠程集羣時，出現Nimbus host is not set異常。異常內容以下所示：

[root@xop-dev-a bin]# ./storm jar /home/clx/storm-starter.jar storm.starter.WordCountTopology wordcountjava

Running: export STORM_JAR=/home/clx/storm-starter.jar; java -client -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -cp /home/clx/storm-0.5.4/storm-0.5.4.jar:/home/clx/storm-0.5.4/lib/log4j-1.2.16.jar:/home/clx/storm-0.5.4/lib/tools.macro-0.1.0.jar:/home/clx/storm-0.5.4/lib/jline-0.9.94.jar:/home/clx/storm-0.5.4/lib/commons-lang-2.5.jar:/home/clx/storm-0.5.4/lib/core.incubator-0.1.0.jar:/home/clx/storm-0.5.4/lib/junit-3.8.1.jar:/home/clx/storm-0.5.4/lib/compojure-0.6.4.jar:/home/clx/storm-0.5.4/lib/zookeeper-3.3.2.jar:/home/clx/storm-0.5.4/lib/clojure-contrib-1.2.0.jar:/home/clx/storm-0.5.4/lib/httpcore-4.0.1.jar:/home/clx/storm-0.5.4/lib/commons-logging-1.1.1.jar:/home/clx/storm-0.5.4/lib/commons-io-1.4.jar:/home/clx/storm-0.5.4/lib/ring-core-0.3.10.jar:/home/clx/storm-0.5.4/lib/httpclient-4.0.1.jar:/home/clx/storm-0.5.4/lib/commons-codec-1.3.jar:/home/clx/storm-0.5.4/lib/jzmq-2.1.0.jar:/home/clx/storm-0.5.4/lib/jvyaml-1.0.0.jar:/home/clx/storm-0.5.4/lib/commons-fileupload-1.2.1.jar:/home/clx/storm-0.5.4/lib/slf4j-log4j12-1.5.8.jar:/home/clx/storm-0.5.4/lib/servlet-api-2.5.jar:/home/clx/storm-0.5.4/lib/json-simple-1.1.jar:/home/clx/storm-0.5.4/lib/ring-jetty-adapter-0.3.11.jar:/home/clx/storm-0.5.4/lib/slf4j-api-1.5.8.jar:/home/clx/storm-0.5.4/lib/jetty-util-6.1.26.jar:/home/clx/storm-0.5.4/lib/joda-time-1.6.jar:/home/clx/storm-0.5.4/lib/libthrift7-0.7.0.jar:/home/clx/storm-0.5.4/lib/commons-exec-1.1.jar:/home/clx/storm-0.5.4/lib/clojure-1.2.0.jar:/home/clx/storm-0.5.4/lib/ring-servlet-0.3.11.jar:/home/clx/storm-0.5.4/lib/clj-time-0.3.0.jar:/home/clx/storm-0.5.4/lib/hiccup-0.3.6.jar:/home/clx/storm-0.5.4/lib/clout-0.4.1.jar:/home/clx/storm-0.5.4/lib/jetty-6.1.26.jar:/home/clx/storm-0.5.4/lib/servlet-api-2.5-20081211.jar:/home/clx/storm-starter.jar:/root/.storm:/home/clx/storm-0.5.4/bin storm.starter.WordCountTopology wordcountjson

0 [main] INFO backtype.storm.StormSubmitter - Jar not uploaded to master yet. Submitting jar...api

Exception in thread "main" java.lang.IllegalArgumentException: Nimbus host is not set服務器

at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:30)

at backtype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:17)網絡

at backtype.storm.StormSubmitter.submitJar(StormSubmitter.java:78)app

at backtype.storm.StormSubmitter.submitJar(StormSubmitter.java:71)ui

at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:50)指針

at storm.starter.WordCountTopology.main(WordCountTopology.java:81)rest

解決方法：在~/.storm/目錄新建storm.yaml文件，~表明用戶主目錄。storm.yaml文件內容：nimbus.host: "10.0.0.24"。重啓nimbus後臺程序，異常消失。日誌

錯誤2：啓動Supervisor時，出現java.lang.UnsatisfiedLinkError: /usr/local/lib/libjzmq.so.0.0.0: libzmq.so.1: cannot open shared object file: No such file or directory異常。異常內容以下所示：

2011-12-01 11:58:47 worker [ERROR] Error on initialization of server mk-worker

java.lang.UnsatisfiedLinkError: /usr/local/lib/libjzmq.so.0.0.0: libzmq.so.1: cannot open shared object file: No such file or directory

at java.lang.ClassLoader$NativeLibrary.load(Native Method)

at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1803)

at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1728)

at java.lang.Runtime.loadLibrary0(Runtime.java:823)

at java.lang.System.loadLibrary(System.java:1028)

at org.zeromq.ZMQ.<clinit>(ZMQ.java:34)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:169)

at zilch.mq__init.load(Unknown Source)

at zilch.mq__init.<clinit>(Unknown Source)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:247)

at clojure.lang.RT.loadClassForName(RT.java:1578)

at clojure.lang.RT.load(RT.java:399)

at clojure.lang.RT.load(RT.java:381)

at clojure.core$load$fn__4511.invoke(core.clj:4905)

at clojure.core$load.doInvoke(core.clj:4904)

at clojure.lang.RestFn.invoke(RestFn.java:409)

at clojure.core$load_one.invoke(core.clj:4729)

at clojure.core$load_lib.doInvoke(core.clj:4766)

at clojure.lang.RestFn.applyTo(RestFn.java:143)

at clojure.core$apply.invoke(core.clj:542)

at clojure.core$load_libs.doInvoke(core.clj:4800)

at clojure.lang.RestFn.applyTo(RestFn.java:138)

at clojure.core$apply.invoke(core.clj:542)

at clojure.core$require.doInvoke(core.clj:4869)

at clojure.lang.RestFn.invoke(RestFn.java:422)

at backtype.storm.messaging.zmq$loading__4410__auto__.invoke(zmq.clj:1)

at backtype.storm.messaging.zmq__init.load(Unknown Source)

at backtype.storm.messaging.zmq__init.<clinit>(Unknown Source)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:247)

at clojure.lang.RT.loadClassForName(RT.java:1578)

at clojure.lang.RT.load(RT.java:399)

at clojure.lang.RT.load(RT.java:381)

at clojure.core$load$fn__4511.invoke(core.clj:4905)

at clojure.core$load.doInvoke(core.clj:4904)

at clojure.lang.RestFn.invoke(RestFn.java:409)

at clojure.core$load_one.invoke(core.clj:4729)

at clojure.core$load_lib.doInvoke(core.clj:4766)

at clojure.lang.RestFn.applyTo(RestFn.java:143)

at clojure.core$apply.invoke(core.clj:542)

at clojure.core$load_libs.doInvoke(core.clj:4800)

at clojure.lang.RestFn.applyTo(RestFn.java:138)

at clojure.core$apply.invoke(core.clj:542)

at clojure.core$require.doInvoke(core.clj:4869)

at clojure.lang.RestFn.invoke(RestFn.java:409)

at backtype.storm.messaging.loader$mk_zmq_context.doInvoke(loader.clj:8)

at clojure.lang.RestFn.invoke(RestFn.java:437)

at backtype.storm.daemon.worker$fn__3074$exec_fn__858__auto____3075.invoke(worker.clj:109)

at clojure.lang.AFn.applyToHelper(AFn.java:187)

at clojure.lang.AFn.applyTo(AFn.java:151)

at clojure.core$apply.invoke(core.clj:540)

at backtype.storm.daemon.worker$fn__3074$mk_worker__3216.doInvoke(worker.clj:78)

at clojure.lang.RestFn.invoke(RestFn.java:513)

at backtype.storm.daemon.worker$_main.invoke(worker.clj:247)

at clojure.lang.AFn.applyToHelper(AFn.java:174)

at clojure.lang.AFn.applyTo(AFn.java:151)

at backtype.storm.daemon.worker.main(Unknown Source)

解決方法 1 ：export LD_LIBRARY_PATH=/usr/local/lib

解決方法 2 ：編輯/etc/ld.so.conf文件，增長一行：/usr/local/lib。再執行sudo ldconfig命令，重啓Supervisor，異常消失。

錯誤 3 ：發佈drpc類型的topologies到遠程集羣時，出現空指針異常，鏈接drpc服務器失敗。

解決方法：在conf/storm.yaml文件中增長drpc服務器配置，啓動配置文件中指定的全部drpc服務。內容以下所示：

drpc.servers:

- "drpc服務器ip"

錯誤 4 ：客戶端調用drpc服務時，worker的日誌中出現Failing message，而bolt都未收到數據。日誌以下所示：

2011-12-02 09:59:16 task [INFO] Failing message backtype.storm.drpc.DRPCSpout$DRPCMessageId@3770bdf7: source: 1:27, stream: 1, id: {-5919451531315711689=-5919451531315711689}, [foo.com/blog/1, {"port":3772,"id":"5","host":"10.0.0.24"}]

解決方法：主機名，域名，hosts文件配置不正確會引發這類錯誤。檢查並修改storm相關機器的主機名，域名，hosts文件。重啓網絡服務：service network restart。重啓storm，再次調用drpc服務，成功。Hosts文件中必須包含以下內容：

【nimbus主機ip】【nimbus主機名】【nimbus主機別名】

【supervisor主機ip】【supervisor主機名】【supervisor主機別名】

【zookeeper主機ip】【zookeeper主機名】【zookeeper主機別名】

錯誤 5 ：發佈topologies時，出現不能序列化log4j.Logger的異常。

解決方法：使用slf4j代替log4j。

錯誤 6 ：bolt在處理消息時，worker的日誌中出現Failing message。

解決方法：提交Topology時設置適當的消息的超時時間，比消息本來的超時時間更長，以下所示：

conf.setMessageTimeoutSecs(60);//默認爲30秒。