redhat6.4上build storm 0.9.0.1

1.安裝mvn

2.下載源代碼

3.build

mvn package 

過程當中出現問題,clojars.org 訪問不了。經過私服映射clojars.org並在pom.xml中將dependency的地址改掉。

從新運行mvn package

發現問題:

java.lang.RuntimeException: Pipe to subprocess seems to be broken! No output read.
Shell Process Exception:
/tmp/a3a2aead-499f-4f93-8390-0650f2d75d0f/supervisor/stormdist/test-1-1392348739/resources/storm.rb:19:in `require': no such file to load -- rubygems (LoadError)
 from /tmp/a3a2aead-499f-4f93-8390-0650f2d75d0f/supervisor/stormdist/test-1-1392348739/resources/storm.rb:19
 from tester_spout.rb:19:in `require'
 from tester_spout.rb:19

 at backtype.storm.utils.ShellProcess.readString(ShellProcess.java:135) ~[classes/:na]
 at backtype.storm.utils.ShellProcess.readMessage(ShellProcess.java:81) ~[classes/:na]
 at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:62) ~[classes/:na]
 at backtype.storm.spout.ShellSpout.open(ShellSpout.java:53) ~[classes/:na]
 at backtype.storm.daemon.executor$fn__3103$fn__3118.invoke(executor.clj:519) ~[classes/:na]
 at backtype.storm.util$async_loop$fn__291.invoke(util.clj:431) ~[classes/:na]
 at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
 at java.lang.Thread.run(Thread.java:662) [na:1.6.0_31]

先把系統默認安裝的ruby1.8.7卸載掉

yum erase ruby

而後在下載一個ruby1.9.3下來

./configure

make

make install

重啓機器就行了

 

在運行storm-book上的第一二章的例子時，發現以下錯誤:

4237 [Thread-9-EventThread] WARN  backtype.storm.cluster - Received event :disconnected::none: with disconnected Zookeeper.
4901 [Thread-9-SendThread(localhost:2000)] WARN  org.apache.zookeeper.ClientCnxn - Session 0x1442eaea3c4000b for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.6.0_35]
 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) ~[na:1.6.0_35]
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119) ~[zookeeper-3.3.3.jar:3.3.3-1073969]

根據錯誤消息中看到，客戶端用的zookeeper版本是3.3.3，而集羣的zookeeper版本是3.4.5.從eclipse中的pom界面能夠搜到是由下列依賴傳遞致使的：

storm-0.9.0.1-->curator-framework-1.0.1-->curator-client-1.0.1-->zookeeper-3.3.3

將zookeeper集羣將爲3.3.3版本

發現仍是不行。後來想到我用的第2章的例子是LocalCluster,猜測應該是運行在一個節點上，而個人集羣是用4個節點搭建的。並且其中zk集羣爲3臺機器。
因而嘗試將conf/storm.yaml中的

storm.zookeeper.servers:

 - jfp3-1

 - jfp3-2

 - jfp3-3

改成只有一條jfp3-1

並重啓zk.再次啓動nimbus,ui,supervisor，而後提交任務。日誌以下:

[root@jfp3-1 ~]# storm jar storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar storm.starter.WordCountTopology mystorm
Running: java -client -Dstorm.options= -Dstorm.home=/opt/storm/storm-0.9.0.1 -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /opt/storm/storm-0.9.0.1/storm-core-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/storm-netty-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/storm-console-logging-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/lib/junit-3.8.1.jar:/opt/storm/storm-0.9.0.1/lib/commons-io-1.4.jar:/opt/storm/storm-0.9.0.1/lib/minlog-1.2.jar:/opt/storm/storm-0.9.0.1/lib/core.incubator-0.1.0.jar:/opt/storm/storm-0.9.0.1/lib/jetty-util-6.1.26.jar:/opt/storm/storm-0.9.0.1/lib/ring-core-1.1.5.jar:/opt/storm/storm-0.9.0.1/lib/clj-time-0.4.1.jar:/opt/storm/storm-0.9.0.1/lib/clout-1.0.1.jar:/opt/storm/storm-0.9.0.1/lib/json-simple-1.1.jar:/opt/storm/storm-0.9.0.1/lib/curator-client-1.0.1.jar:/opt/storm/storm-0.9.0.1/lib/reflectasm-1.07-shaded.jar:/opt/storm/storm-0.9.0.1/lib/disruptor-2.10.1.jar:/opt/storm/storm-0.9.0.1/lib/commons-logging-1.1.1.jar:/opt/storm/storm-0.9.0.1/lib/logback-classic-1.0.6.jar:/opt/storm/storm-0.9.0.1/lib/jgrapht-0.8.3.jar:/opt/storm/storm-0.9.0.1/lib/logback-core-1.0.6.jar:/opt/storm/storm-0.9.0.1/lib/libthrift7-0.7.0-2.jar:/opt/storm/storm-0.9.0.1/lib/httpcore-4.1.jar:/opt/storm/storm-0.9.0.1/lib/snakeyaml-1.11.jar:/opt/storm/storm-0.9.0.1/lib/commons-codec-1.4.jar:/opt/storm/storm-0.9.0.1/lib/clojure-1.4.0.jar:/opt/storm/storm-0.9.0.1/lib/servlet-api-2.5.jar:/opt/storm/storm-0.9.0.1/lib/mockito-all-1.9.5.jar:/opt/storm/storm-0.9.0.1/lib/hiccup-0.3.6.jar:/opt/storm/storm-0.9.0.1/lib/guava-13.0.jar:/opt/storm/storm-0.9.0.1/lib/commons-exec-1.1.jar:/opt/storm/storm-0.9.0.1/lib/tools.logging-0.2.3.jar:/opt/storm/storm-0.9.0.1/lib/carbonite-1.5.0.jar:/opt/storm/storm-0.9.0.1/lib/tools.macro-0.1.0.jar:/opt/storm/storm-0.9.0.1/lib/servlet-api-2.5-20081211.jar:/opt/storm/storm-0.9.0.1/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/storm-0.9.0.1/lib/clj-stacktrace-0.2.2.jar:/opt/storm/storm-0.9.0.1/lib/jline-0.9.94.jar:/opt/storm/storm-0.9.0.1/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/storm-0.9.0.1/lib/tools.nrepl-0.2.3.jar:/opt/storm/storm-0.9.0.1/lib/asm-4.0.jar:/opt/storm/storm-0.9.0.1/lib/slf4j-api-1.6.5.jar:/opt/storm/storm-0.9.0.1/lib/commons-lang-2.5.jar:/opt/storm/storm-0.9.0.1/lib/joda-time-2.0.jar:/opt/storm/storm-0.9.0.1/lib/compojure-1.1.3.jar:/opt/storm/storm-0.9.0.1/lib/ring-devel-0.3.11.jar:/opt/storm/storm-0.9.0.1/lib/curator-framework-1.0.1.jar:/opt/storm/storm-0.9.0.1/lib/httpclient-4.1.1.jar:/opt/storm/storm-0.9.0.1/lib/netty-3.6.3.Final.jar:/opt/storm/storm-0.9.0.1/lib/commons-fileupload-1.2.1.jar:/opt/storm/storm-0.9.0.1/lib/zookeeper-3.3.3.jar:/opt/storm/storm-0.9.0.1/lib/tools.cli-0.2.2.jar:/opt/storm/storm-0.9.0.1/lib/jetty-6.1.26.jar:/opt/storm/storm-0.9.0.1/lib/kryo-2.17.jar:/opt/storm/storm-0.9.0.1/lib/clojure-complete-0.2.3.jar:/opt/storm/storm-0.9.0.1/lib/math.numeric-tower-0.0.1.jar:/opt/storm/storm-0.9.0.1/lib/ring-servlet-0.3.11.jar:/opt/storm/storm-0.9.0.1/lib/objenesis-1.2.jar:/opt/storm/storm-0.9.0.1/lib/jzmq-2.1.0.jar:storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar:/opt/storm/storm-0.9.0.1/conf:/opt/storm/storm-0.9.0.1/bin -Dstorm.jar=storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar storm.starter.WordCountTopology mystorm
344  [main] INFO  backtype.storm.StormSubmitter - Jar not uploaded to master yet. Submitting jar...
357  [main] INFO  backtype.storm.StormSubmitter - Uploading topology jar storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar to assigned location: /var/storm/workdir/nimbus/inbox/stormjar-c00ff052-802b-40c1-bc72-83df035c2358.jar
608  [main] INFO  backtype.storm.StormSubmitter - Successfully uploaded topology jar to assigned location: /var/storm/workdir/nimbus/inbox/stormjar-c00ff052-802b-40c1-bc72-83df035c2358.jar
608  [main] INFO  backtype.storm.StormSubmitter - Submitting topology mystorm in distributed mode with conf {"topology.workers":3,"topology.debug":true}
1002 [main] INFO  backtype.storm.StormSubmitter - Finished submitting topology: mystorm 

 

並且還解決了一個原來的問題，web ui訪問不了。如今能夠訪問webui了。

 

運行第2章的例子：

mvn exec:java -Dexec.mainClass="TopologyMain" -Dexec.args="src/main/resources/words.txt"