Storm On Yarn 安裝部署

1. 安裝 JDK7 和 Maven

2. 部署Hadoop2集羣，並啓動yarn
http://my.oschina.net/zc741520/blog/362824

3. 下載 Storm on Yarn
[grid@hadoop4 ~]$ wget https://github.com/yahoo/storm-yarn/archive/master.zip

4. 編譯

[grid@hadoop4 ~]$ unzip master.zip
[grid@hadoop4 ~]$ cd storm-yarn-master
## 修改 pom.xml，將Hadoop的版本號改爲對應的版本號
[grid@hadoop4 storm-yarn-master]$ vim pom.xml
    <properties>
        <storm.version>0.9.0-wip21</storm.version>
        <hadoop.version>2.5.2</hadoop.version>
        <!--hadoop.version>2.1.0.2.0.5.0-67</hadoop.version-->
    </properties>
## 編譯
[grid@hadoop4 storm-yarn-master]$ mvn package -DskipTests

5. storm-yarn-master/lib/storm-0.9.0-wip21.zip 解壓到上層目錄storm-yarn-master中 

[grid@hadoop4 storm-yarn-master]$ cd lib
[grid@hadoop4 lib]$ unzip storm-0.9.0-wip21.zip -d ..
[grid@hadoop4 storm-yarn-master]$ ls
bin  CLA.pdf  create-tarball.sh  lib  LICENSE.txt  pom.xml  README.md  src  storm-0.9.0-wip21  target

6. 在HDFS中建立對應Storm目錄
[grid@hadoop4 storm-yarn-master]$ hadoop fs -mkdir -p /lib/storm/0.9.0-wip21

7. 將storm.zip放到HDFS
##  根據實際須要，添加Storm工程須要的額外Jar包到storm-0.9.0-wip21的lib下，從新壓縮成storm.zip文件，上傳至HDFS的指定目錄中（很是重要，集羣中經過訪問hdfs中的storm.zip獲取工做環境）

[grid@hadoop4 storm-yarn-master]$ hadoop fs -put ./lib/storm.zip /lib/storm/0.9.0-wip21
[grid@hadoop4 storm-yarn-master]$ hadoop fs -ls /lib/storm/0.9.0-wip21
Found 1 items
-rw-r--r--   2 grid supergroup   17141078 2015-05-24 19:43 /lib/storm/0.9.0-wip21/storm.zip

8. 在安裝Hadoop時已經設置好了hadoop的一些環境變量，如今再增長以下環境變量 

[grid@hadoop4 storm-yarn-master]$ vim ~/.bash_profile 
export PATH=$PATH:/home/grid/storm-yarn-master/storm-0.9.0-wip21/bin:/home/grid/storm-yarn-master/bin
[grid@hadoop4 storm-yarn-master]$ source ~/.bash_profile

9. 修改 storm-yarn-master/storm-0.9.0-wip21/conf/storm.yaml 配置文件，增長zookeeper的配置 

## zookeeper集羣的安裝與啓動，在介紹Storm集羣安裝（ http://my.oschina.net/zc741520/blog/399233）時已經有過詳細說明，這裏就再也不贅述

[grid@hadoop4 storm-yarn-master]$ vim storm-0.9.0-wip21/conf/storm.yaml 
storm.zookeeper.servers:
    - "hadoop4"
    - "hadoop5"
    - "hadoop6"

master.initial-num-supervisors: 1
master.container.size-mb: 1024

10. 啓動 storm on yarn 環境
[grid@hadoop4 storm-yarn-master]$ storm-yarn launch storm-0.9.0-wip21/conf/storm.yaml

由於storm是做爲一個yarn程序運行在集羣上的，因此在YARN的集羣管理頁面中會有一個AppId
PS：第一次啓動時失敗了，最終發現是內存不足致使的，解決辦法是在yarn-site.xml中設置yarn.nodemanager.vmem-check-enabled的值爲false



 11. 查找nimbus節點

[grid@hadoop4 storm-yarn-master]$ storm-yarn getStormConfig -appId application_1432484548277_0001 -output ~/.storm/storm.yaml
[grid@hadoop4 storm-yarn-master]$ cat ~/.storm/storm.yaml | grep nimbus.host
nimbus.host: 192.168.0.107

12. 提交Topology，這裏用Storm自帶的WordCount例子程序

[grid@hadoop4 storm-yarn-master]$ storm jar lib/storm-starter-0.0.1-SNAPSHOT.jar storm.starter.WordCountTopology WordCountTopology -c nimbus.host=192.168.0.107

 13. 查看storm的UI監控界面（nimbus.host:7070）