機子配置信息
操做系統:CentOS release 6.5 (Final)
cpu:Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz
內存:32G
磁盤大小:1.5T
storm 軟件包
apache-storm-1.0.0.tar.gz
Python 版本
Centos 6.5 系統自帶有以下版本 Python,知足 storm
部署的要求
Python 2.6.6 (r266:84292, Nov 22 2013, 12:16:22)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
JDK
storm1.0.0
官方要求 JDK1.7
及以上,本次實驗使用 JDK1.8
java version "1.8.0_91"
zookeeper 版本
zookeeper-3.4.9.tar.gzhtml
因爲機子數量不足,一臺機子可能有多個職能。
zookeeper
172.24.132.173
172.24.132.143
172.24.132.142
nimbus
172.24.132.174
supbervisor
172.24.132.143
172.24.132.142java
編輯各個機子的 /etc/hosts
文件,將主機名稱與 IP
的對應關係加上,例如:linux
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 172.24.132.142 CLTQ-132-142
export JAVA_HOME=/usr/local/lifecycle/jdk1.8.0_91 export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
source /etc/profile
使修改生效注意點:git
/usr/local/lifecycle/jdk1.8.0_91
改成本機 jdk
安裝目錄:
來分隔路徑.
不能丟,把當前目錄丟掉也是常見的錯誤。zookeeper-3.4.9.tar.gz
解壓到:/usr/local/
github
sudo tar -C /usr/local -xzf zookeeper-3.4.9.tar.gz
配置 zookeeper
的環境變量,這一步是可選的,配置以後可以全局使用 zookeeper
相關的命令,沒有配置則須要到 zookeeper
的安裝目錄下執行命令
添加以下配置到 /etc/profile
文件的最後,並經過命令 source /etc/profile
命令使修改後的配置生效apache
#ZOOKEEPER ZOOKEEPER=/usr/local/zookeeper-3.4.9 PATH=$PATH:$ZOOKEEPER/bin
首先將 /usr/local/zookeeper-3.4.6/conf/zoo_sample.cfg
文件複製一份,並改名爲 zoo.cfg
。若是不須要配置集羣,則不修改修改 zoo.cfg
文件。要配置集羣,則須要將 zookeeper
集羣信息經過 server
配置。api
tickTime=2000 initLimit=10 syncLimit=5 dataDir=/usr/local/zookeeper-3.4.6/data dataLogDir=/usr/local/zookeeper-3.4.6/log clientPort=2181 server.1=172.24.132.173:2888:3888 server.2=172.24.132.143:2888:3888 server.3=172.24.132.142:2888:3888
server.A=B:C:D
:其中 A
是一個數字,表示這個是第幾號服務器;B
是這個服務器的 ip
地址;C
表示的是這個服務器與集羣中的 Leader
服務器交換信息的端口;D
表示的是萬一集羣中的 Leader
服務器掛了,須要一個端口來從新進行選舉,選出一個新的 Leader
,而這個端口就是用來執行選舉時服務器相互通訊的端口。若是是僞集羣的配置方式,因爲 B
都是同樣,因此不一樣的 zookeeper
實例通訊端口號不能同樣,因此要給它們分配不一樣的端口號。服務器
根據 dataDir
和 dataLogDir
變量建立相應的目錄,建議優先建立,由於有可能使用的 linux
帳戶權限不足,zookeeper
沒法自動建立這幾個目錄。session
在配置文件 zoo.cfg
中 dataDir
所指路徑 /usr/local/zookeeper-3.4.6/data
下,新建 myid
文件,並寫入 zoo.cfg
文件的 server.A
中 A
的數值,在不一樣機器上的該文件中填寫相應的值。本次部署中,172.24.132.142 的 myid
文件應該寫入數值 3
;172.24.132.143 的 myid
文件應該寫入數值 2
;172.24.132.173 的 myid
文件應該寫入數值 1
dom
執行命令 zkServer.sh start
將會啓動 zookeeper
。而執行命令 zkServer.sh stop
將會中止 zookeeper
。
經過 jps
命令,能夠看到 zookeeper
的進程名:QuorumPeerMain
。以及執行命令 zkServer.sh status
查看 zookeeper
集羣狀態,以下所示:
#172.24.132.142 ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper-3.4.9/bin/../conf/zoo.cfg Mode: follower #172.24.132.143 ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper-3.4.9/bin/../conf/zoo.cfg Mode: leader #172.24.132.173 ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper-3.4.9/bin/../conf/zoo.cfg Mode: follower
zookeeper
集羣逐臺啓動的過程當中,查閱 zookeeper.out
,會有以下異常:
2017-06-12 19:58:04,289 [myid:3] - WARN [WorkerSender[myid=3]:QuorumCnxManager@400] - Cannot open channel to 1 at election address CLTQ-132-173/172.24.132.173:3888 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433) at java.lang.Thread.run(Thread.java:745) 2017-06-12 19:58:04,289 [myid:3] - INFO [WorkerSender[myid=3]:QuorumPeer$QuorumServer@149] - Resolved hostname: CLTQ-132-173 to address: CLTQ-132-173/172.24.132.173
上述異常能夠忽略,由於集羣環境中某些子節點尚未啓動 zookeeper
先安裝 nc
:
yum install -y nc
返回 imok
則代表機子的狀態是正常的
echo ruok | nc 172.24.132.143 2181 imok
bin/zkCli.sh -server 172.24.132.143:2181
若是須要一次性鏈接 zookeeper
集羣的多臺機子,則可使用以下語法:
bin/zkCli.sh -server 172.24.132.142:2181,172.24.132.173:2181,172.24.132.143:2181
輸出以下結果:
[root@CLTQ-132-142 zookeeper-3.4.9]# bin/zkCli.sh -server 172.24.132.143:2181 Connecting to 172.24.132.143:2181 2017-06-08 16:25:07,196 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.9-1757313, built on 08/23/2016 06:50 GMT 2017-06-08 16:25:07,199 [myid:] - INFO [main:Environment@100] - Client environment:host.name=CLTQ-132-142 2017-06-08 16:25:07,199 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_91 2017-06-08 16:25:07,200 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation 2017-06-08 16:25:07,200 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/local/lifecycle/jdk1.8.0_91/jre 2017-06-08 16:25:07,200 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/usr/local/zookeeper-3.4.9/bin/../build/classes:/usr/local/zookeeper-3.4.9/bin/../build/lib/*.jar:/usr/local/zookeeper-3.4.9/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/local/zookeeper-3.4.9/bin/../lib/slf4j-api-1.6.1.jar:/usr/local/zookeeper-3.4.9/bin/../lib/netty-3.10.5.Final.jar:/usr/local/zookeeper-3.4.9/bin/../lib/log4j-1.2.16.jar:/usr/local/zookeeper-3.4.9/bin/../lib/jline-0.9.94.jar:/usr/local/zookeeper-3.4.9/bin/../zookeeper-3.4.9.jar:/usr/local/zookeeper-3.4.9/bin/../src/java/lib/*.jar:/usr/local/zookeeper-3.4.9/bin/../conf:.:/usr/local/lifecycle/jdk1.8.0_91/lib/dt.jar:/usr/local/lifecycle/jdk1.8.0_91/lib/tools.jar 2017-06-08 16:25:07,200 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2017-06-08 16:25:07,200 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp 2017-06-08 16:25:07,200 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA> 2017-06-08 16:25:07,201 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux 2017-06-08 16:25:07,201 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64 2017-06-08 16:25:07,201 [myid:] - INFO [main:Environment@100] - Client environment:os.version=2.6.32-431.el6.x86_64 2017-06-08 16:25:07,201 [myid:] - INFO [main:Environment@100] - Client environment:user.name=root 2017-06-08 16:25:07,201 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/root 2017-06-08 16:25:07,201 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/usr/local/zookeeper-3.4.9 2017-06-08 16:25:07,202 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=172.24.132.143:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@799f7e29 Welcome to ZooKeeper! 2017-06-08 16:25:07,219 [myid:] - INFO [main-SendThread(172.24.132.143:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server 172.24.132.143/172.24.132.143:2181. Will not attempt to authenticate using SASL (unknown error) JLine support is enabled 2017-06-08 16:25:07,266 [myid:] - INFO [main-SendThread(172.24.132.143:2181):ClientCnxn$SendThread@876] - Socket connection established to 172.24.132.143/172.24.132.143:2181, initiating session [zk: 172.24.132.143:2181(CONNECTING) 0] 2017-06-08 16:25:07,286 [myid:] - INFO [main-SendThread(172.24.132.143:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server 172.24.132.143/172.24.132.143:2181, sessionid = 0x25c86c7c3bf0000, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null
建立文件夾:
[zk: 172.24.132.143:2181(CONNECTED) 4] create /c1project c1project Created /c1project [zk: 172.24.132.143:2181(CONNECTED) 5] get /c1project c1project c1project cZxid = 0x100000002 ctime = Thu Jun 08 16:30:47 CST 2017 mZxid = 0x100000002 mtime = Thu Jun 08 16:30:47 CST 2017 pZxid = 0x100000002 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 9 numChildren = 0
在集羣的其餘機子上可以查詢到該文件,則說明 zookeeper
集羣是建立成功了的。
apache-storm-1.0.0.tar.gz
解壓到: /usr/local/
:
sudo tar -C /usr/local -xzf apache-storm-1.1.0.tar.gz
添加以下幾個主要參數就能夠
storm.zookeeper.servers: - "172.24.132.142" - "172.24.132.143" - "172.24.132.173" storm.zookeeper.port: 2181 storm.local.dir: "/usr/local/apache-storm-1.0.0/data" nimbus.seeds: ["172.24.132.174"] supervisor.slots.ports: - 6700 - 6701 - 6702 - 6703
storm.zookeeper.servers
:zookeeper
集羣的 IP
storm.zookeeper.port
:zookeeper
集羣的端口,若是不是默認端口 2181
則須要設置nimbus.seeds
:能夠做爲 nimbus
的機子將配置修改完成的軟件拷貝到其餘機子上:
scp -r apache-storm-1.0.0 172.24.132.142:/usr/local/
nimbus
與 supervisor
能夠部署在同一臺機子,可是建議分開,避免相互影響。
172.24.132.174 後臺運行 nimbus
bin/storm nimbus >/dev/null 2>&1 &
172.24.132.142 172.24.132.143後臺運行 supervisor
bin/storm supervisor >/dev/null 2>&1 &
172.24.132.174 後臺運行 storm ui
storm ui
得要在 nimbus
機子上運行,不可以在 supervisor
機子上運行
bin/storm ui >/dev/null 2>&1 &
使用 storm
自帶測試例子測試,從 github
下載工程到本地:
git clone https://github.com/apache/storm.git
拉取對應版本的分支,本次部署使用分支 1.0.x-branch
:
git checkout 1.0.x-branch
進入目錄 storm/examples/storm-starter
,打包:
mvn package
打包頗有可能失敗,每每是各類的 jar
包沒法下載,能夠藉助 IDE,將須要的依賴下載到本地。本次部署一直沒法下載的 kafka-avro-serializer 1.0
,能夠經過以下鏈接直接下載,kafka-avro-serializer 1.0,下載完成的 jar
包得要放到 .m2\repository\io\confluent\kafka-avro-serializer\1.0
下。
生成目錄:storm\examples\storm-starter\target
,其下的 storm-starter-1.0.2.jar
能夠做爲測試 jar
執行命令:
bin/storm jar /usr/local/apache-storm-1.0.0/storm-starter-1.0.2.jar org.apache.storm.starter.ExclamationTopology et
et
是 topology
的名字,任意,可是不可以省略。
[root@CLTQ-132-174 apache-storm-1.0.0]# bin/storm jar /usr/local/apache-storm-1.0.0/storm-starter-1.0.2.jar org.apache.storm.starter.ExclamationTopology et Running: /usr/local/lifecycle/jdk1.8.0_91/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/local/apache-storm-1.0.0 -Dstorm.log.dir=/usr/local/apache-storm-1.0.0/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/local/apache-storm-1.0.0/storm-starter-1.0.2.jar:/usr/local/apache-storm-1.0.0/lib/servlet-api-2.5.jar:/usr/local/apache-storm-1.0.0/lib/clojure-1.7.0.jar:/usr/local/apache-storm-1.0.0/lib/slf4j-api-1.7.7.jar:/usr/local/apache-storm-1.0.0/lib/asm-5.0.3.jar:/usr/local/apache-storm-1.0.0/lib/kryo-3.0.3.jar:/usr/local/apache-storm-1.0.0/lib/log4j-core-2.1.jar:/usr/local/apache-storm-1.0.0/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/apache-storm-1.0.0/lib/storm-rename-hack-1.0.0.jar:/usr/local/apache-storm-1.0.0/lib/objenesis-2.1.jar:/usr/local/apache-storm-1.0.0/lib/reflectasm-1.10.1.jar:/usr/local/apache-storm-1.0.0/lib/storm-core-1.0.0.jar:/usr/local/apache-storm-1.0.0/lib/minlog-1.3.0.jar:/usr/local/apache-storm-1.0.0/lib/log4j-slf4j-impl-2.1.jar:/usr/local/apache-storm-1.0.0/lib/log4j-api-2.1.jar:/usr/local/apache-storm-1.0.0/lib/disruptor-3.3.2.jar:/usr/local/apache-storm-1.0.0/storm-starter-1.0.2.jar:/usr/local/apache-storm-1.0.0/conf:/usr/local/apache-storm-1.0.0/bin -Dstorm.jar=/usr/local/apache-storm-1.0.0/storm-starter-1.0.2.jar org.apache.storm.starter.ExclamationTopology et 531 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -7107387084739244238:-5556405776831141841 583 [main] INFO o.a.s.s.a.AuthUtils - Got AutoCreds [] 624 [main] INFO o.a.s.StormSubmitter - Uploading topology jar /usr/local/apache-storm-1.0.0/storm-starter-1.0.2.jar to assigned location: /usr/local/apache-storm-1.0.0/data/nimbus/inbox/stormjar-1ee74fb2-7356-4e0c-8f97-f0c71ff4c884.jar Start uploading file '/usr/local/apache-storm-1.0.0/storm-starter-1.0.2.jar' to '/usr/local/apache-storm-1.0.0/data/nimbus/inbox/stormjar-1ee74fb2-7356-4e0c-8f97-f0c71ff4c884.jar' (73376663 bytes) [==================================================] 73376663 / 73376663 File '/usr/local/apache-storm-1.0.0/storm-starter-1.0.2.jar' uploaded to '/usr/local/apache-storm-1.0.0/data/nimbus/inbox/stormjar-1ee74fb2-7356-4e0c-8f97-f0c71ff4c884.jar' (73376663 bytes) 1029 [main] INFO o.a.s.StormSubmitter - Successfully uploaded topology jar to assigned location: /usr/local/apache-storm-1.0.0/data/nimbus/inbox/stormjar-1ee74fb2-7356-4e0c-8f97-f0c71ff4c884.jar 1029 [main] INFO o.a.s.StormSubmitter - Submitting topology et in distributed mode with conf {"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-7107387084739244238:-5556405776831141841","topology.workers":3,"topology.debug":true} 1471 [main] INFO o.a.s.StormSubmitter - Finished submitting topology: et
16373 [Thread-10] ERROR o.a.s.d.s.ReadClusterState - Failed to Sync Supervisor java.lang.RuntimeException: java.lang.InterruptedException at org.apache.storm.utils.Utils.wrapInRuntime(Utils.java:1531) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.zookeeper.zookeeper.getChildren(zookeeper.java:265) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.cluster.ZKStateStorage.get_children(ZKStateStorage.java:174) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.cluster.StormClusterStateImpl.assignments(StormClusterStateImpl.java:153) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.daemon.supervisor.ReadClusterState.run(ReadClusterState.java:126) [storm-core-1.1.0.jar:1.1.0] at org.apache.storm.event.EventManagerImp$1.run(EventManagerImp.java:54) [storm-core-1.1.0.jar:1.1.0] Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) ~[?:1.8.0_91] at java.lang.Object.wait(Object.java:502) ~[?:1.8.0_91] at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1588) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1625) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:226) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:219) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:216) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:207) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:40) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.zookeeper.zookeeper.getChildren(zookeeper.java:260) ~[storm-core-1.1.0.jar:1.1.0] ... 4 more
部署的 storm
服務器版本爲 1.1.0
,而提交給 storm
跑的 jar
包爲:storm-starter-1.0.2.jar
。版本不對應致使上面的問題。應該是 storm-core-1.1.0.jar
的代碼與 storm-core-1.0.2.jar
相差較大,或者修改了通訊的協議致使。將 storm
服務器版本修改成 1.0.0
便可解決問題
具體現象就是啓動了多個 supervisor
,單在 ui
上只顯示一個(也有多是多個 supervisor
中的某幾個看上去被「合併」了),kill
掉其中任意一個 supervisor
,另外一個就出現。
例如本例中有兩個 supervisor
,172.24.132.143
和 172.24.132.142
,可是經過接口請求,每次都只會顯示其中的一個,可是現實的機子是交替出現的:
curl -X GET \ http://172.24.132.174:8080/api/v1/supervisor/summary { "supervisors": [ { "totalMem": 3072, "host": "CLTQ-132-142", "id": "26cdf80b-394e-47e8-a82a-ea78f82e7c22", "uptime": "14h 16m 17s", "totalCpu": 400, "usedCpu": 0, "logLink": "http://CLTQ-132-142:8000/daemonlog?file=supervisor.log", "usedMem": 2496, "slotsUsed": 3, "version": "1.0.0", "slotsTotal": 4, "uptimeSeconds": 51377 } ], "schedulerDisplayResource": false, "logviewerPort": 8000 }
解決方案:storm.yaml
文件中有配置 storm.local.dir: "/usr/local/apache-storm-1.0.0/data"
,local.dir
所指目錄,重啓便可解決問題。緣由是因爲部署時經過 linux scp
命令直接分發軟件到其餘機子,殘留了 local.dir
的東西,而 storm
是根據 local.dir
中的某一個或一些文件計算出一個 supervisor id
的。刪除 local.dir
後,會從新生成 id
。
bin/storm jar /usr/local/apache-storm-1.0.0/storm-starter-1.0.2.jar org.apache.storm.starter.ExclamationTopology et
第一確保 storm-starter-1.0.2.jar
的路徑是正確的;第二保證 packagename.ExclamationTopology
,包名packagename
與 類名 ExclamationTopology
是正確的