Storm on Yarn 安裝配置

1.背景知識

在不修改Storm任何源代碼的狀況下，讓Storm運行在YARN上，最簡單的實現方法是將Storm的各個服務組件（包括Nimbus和Supervisor），做爲單獨的任務運行在YARN上，當前比較有名的「Storm On YARN」實現是由yahoo！開源的，它基本實現了上述描述的功能，下面具體進行說明：
（1） YARN-Storm Client
提供了一系列Shell命令供用戶控制YARN上的Storm服務，好比構建一個Storm集羣命令以下：
storm-yarn launch <storm-yarn-config>
其中，<storm-yarn-config>是Storm配置信息，包括啓動的Supervisor個數、Storm ApplicationMaster佔用的內存等。
啓動Storm以後，可經過如下命令控制Storm：
storm-yarn [command] -appId [appId] -output [file] [-supervisors [n]]

「-supervisors」爲需增長的Supervisor服務個數，該參數只對命令「addSupervisors」有效。

Command	參數含義
setStormConfig	重置集羣配置，集羣從新啓動
getStormConfig	獲取當前集羣配置，Json格式。
addSupervisors	增長supervisor個數
startNimbus/stopNimbus	啓動、中止Nimbus
startUI/stopUI	啓動中止Web UI
startSupervisors/stopSupervisors	啓動中止全部supervisor
shutdown	關閉集羣

 (2) YARN-Storm ApplicationMaster

Storm ApplicationMaster初始化時，將在同一個Container中啓動Storm Nimbus和Storm Web UI兩個服務，而後根據待啓動的Supervisor數目向ResourceManager申請資源，在目前實現中，ApplicationMaster將請求一個節點上全部資源而後啓動Supervisor服務，也就是說，當前Supervisor將獨佔節點而不會與其餘服務共享節點資源，這種狀況下可避免其餘服務對Storm集羣的干擾。
除了運行Storm Nimbus和Web UI外，Storm ApplicationMaster還會啓動一個Thrift Server以處理來自YARN-Storm Client端的各類請求。

 

2.安裝環境

a. Hadoop 2.2.0
b. jdk1.7.0_60
c. apache-maven-3.0.5

 

3.Storm on Yarn的安裝準備

注意：全部節點上都須要安裝storm；能夠只安裝一個storm on yarn客戶端。

a. 從GitHub上下載Storm on Yarn
wget https://github.com/yahoo/storm-yarn/archive/master.zip
b. storm on yarn 須要編譯

unzip storm-yarn-master.zip

cd storm-yarn-master

c. Edit pom.xml，修改Hadoop的版本號，改爲對應的版本號

<properties>
        <storm.version>0.9.0-wip21</storm.version>
        <hadoop.version>2.2.0</hadoop.version>
        <!--hadoop.version>2.1.0.2.0.5.0-67</hadoop.version-->
</properties>

d.mvn編譯

mvn package -DskipTests

編譯好後解壓storm-yarn-master/lib/storm-0.9.0-wip21.zip，獲得 storm-0.9.0-wip21目錄。
將獲得 storm-0.9.0-wip21 目錄移動到 和 storm-yarn-master同級。

/home/ebupt/storm/
|-- storm-0.9.0-wip21
`-- storm-yarn-master

4.配置Storm的工做環境

a. 添加storm-0.9.0-wip21和storm-yarn-master的bin到Path環境變量
vi ~/.bash_profile

export STORM_HOME=$HOME/storm
export PATH=$PATH:$STORM_HOME/storm-yarn-master/bin:$STORM_HOME/storm-0.9.0-wip21/bin

b. 添加Storm工程須要的額外Jar包(storm.zip)。上傳至HDFS的指定目錄中（很是重要，集羣中經過訪問hdfs中的storm.zip獲取工做環境）

zip -r storm.zip storm-0.9.0-wip21/

hadoop fs –put storm.zip /lib/storm/0.9.0-wip21/

 

5. 安裝並運行Storm

a. 修改storm.yaml文件
vi storm-0.9.0-wip21/conf/storm.yaml

storm.zookeeper.servers:
- "eb170"
- "eb171"

storm.zookeeper.port: 2182
storm.local.dir: "/home/ebupt/storm/localstorm"

supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
- 6704

b. 提交運行storm on yarn，並獲得一個ApplicationId
storm-yarn launch ~/storm/storm-0.9.0-wip21/conf/storm.yaml 

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/ebupt/eb/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/ebupt/eb/storm-yarn/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
14/07/04 15:37:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/07/04 15:37:45 INFO client.RMProxy: Connecting to ResourceManager at eb170/10.1.69.170:8032
14/07/04 15:37:46 INFO yarn.StormOnYarn: Copy App Master jar from local filesystem and add to local environment
14/07/04 15:37:47 INFO yarn.StormOnYarn: Set the environment for the application master
14/07/04 15:37:47 INFO yarn.StormOnYarn: YARN CLASSPATH COMMAND = [[yarn, classpath]]
14/07/04 15:37:47 INFO yarn.StormOnYarn: YARN CLASSPATH = [/home/ebupt/eb/hadoop-2.2.0/etc/hadoop:/home/ebupt/eb/hadoop-2.2.0/etc/hadoop:/home/ebupt/eb/hadoop-2.2.0/etc/hadoop:/home/ebupt/eb/hadoop-2.2.0/share/hadoop/common/lib/*:/home/ebupt/eb/hadoop-2.2.0/share/hadoop/common/*:/home/ebupt/eb/hadoop-2.2.0/share/hadoop/hdfs:/home/ebupt/eb/hadoop-2.2.0/share/hadoop/hdfs/lib/*:/home/ebupt/eb/hadoop-2.2.0/share/hadoop/hdfs/*:/home/ebupt/eb/hadoop-2.2.0/share/hadoop/yarn/lib/*:/home/ebupt/eb/hadoop-2.2.0/share/hadoop/yarn/*:/home/ebupt/eb/hadoop-2.2.0/share/hadoop/mapreduce/lib/*:/home/ebupt/eb/hadoop-2.2.0/share/hadoop/mapreduce/*:/home/ebupt/hadoop/contrib/capacity-scheduler/*.jar:/home/ebupt/eb/hadoop-2.2.0/share/hadoop/yarn/*:/home/ebupt/eb/hadoop-2.2.0/share/hadoop/yarn/lib/*]
14/07/04 15:37:47 INFO yarn.StormOnYarn: Using JAVA_HOME = [/home/ebupt/eb/jdk1.7.0_60]
14/07/04 15:37:47 INFO yarn.StormOnYarn: Setting up app master command:[/home/ebupt/eb/jdk1.7.0_60/bin/java, -Dstorm.home=./storm/storm-0.9.0-wip21/, -Dlogfile.name=<LOG_DIR>/master.log, com.yahoo.storm.yarn.MasterServer, 1><LOG_DIR>/stdout, 2><LOG_DIR>/stderr]
14/07/04 15:37:47 INFO impl.YarnClientImpl: Submitted application application_1402648970753_0025 to ResourceManager at eb170/10.1.69.170:8032
application_1402648970753_0025

注：由於storm是做爲一個yarn程序運行在集羣上的，因此會有一個AppId，如上圖所示：application_1402648970753_0025。

 

6.storm提交任務

a.獲取集羣配置

storm-yarn getStormConfig -appId application_1402648970753_0025 -output ~/.storm/storm.yaml
b. 經過如下命令獲得Nimbus host
cat ~/.storm/storm.yaml | grep nimbus.host

c. 提交Topology
storm jar ~/storm/storm-yarn-master/lib/storm-starter-0.0.1-SNAPSHOT.jar storm.starter.WordCountTopology WordCountTopology -c nimbus.host=<your nimbus host>
d. 監控Topology
查看Storm ui，地址是：http://<your nimbus host>:7070

e. 關閉Topology
storm kill [Topology_name]
f. 關閉Storm on yarn集羣
storm-yarn shutdown -appId [applicationId]

g.查看storm進程狀態：nimbus、supervisor、core、logviewer、worker

[ebupt@eb171 ~]$ jps
8700 JournalNode
8939 NodeManager
8805 DFSZKFailoverController
31802 worker
8501 NameNode
5189 Jps
31616 supervisor
8592 DataNode
28865 logviewer
31793 worker
31475 MasterServer
31795 worker
5841 HRegionServer
31509 nimbus
31510 core
31577 QuorumPeerMain

 

7.他山之石

a.有時候發現supervisor啓動不來，會發現是內存資源不夠。在虛擬機環境的同志要注意這點。(現實中的環境也是如此)

b.nimbus.host是你提交storm到yarn後，yarn會給你分配一個地址，得本身找。

 

8.參考資料

https://github.com/yahoo/storm-yarn

http://dongxicheng.org/mapreduce-nextgen/storm-on-yarn/

http://blog.csdn.net/jiushuai/article/details/26693311

http://ghost-face.iteye.com/blog/2017374

 

9.遇到的問題及解決

① 沒法加載storm集羣

[ebupt@eb170 conf]$ storm-yarn launch storm.yaml
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/ebupt/eb/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/ebupt/eb/storm-yarn/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.UnsupportedClassVersionError: backtype/storm/utils/Utils : Unsupported major.minor version 51.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at com.yahoo.storm.yarn.Config.readStormConfig(Config.java:48)
        at com.yahoo.storm.yarn.LaunchCommand.process(LaunchCommand.java:59)
        at com.yahoo.storm.yarn.Client.execute(Client.java:142)
        at com.yahoo.storm.yarn.Client.main(Client.java:148)

緣由：java版本問題。（J2SE 7 = 51）實驗室環境的java環境是jdk1.6，storm on yarn要求jdk1.7。

解決：升級jdk版本爲1.7。

 

②yarn上提交storm on yarn，任務fail，日誌以下。

User:    huangq
Name:    Storm-on-Yarn
Application Type:    YARN
State:    FAILED
FinalStatus:    FAILED
Started:    4-Jul-2014 10:14:15
Elapsed:    4 sec
Tracking URL:    History
Diagnostics:    
Application application_1402648970753_0013 failed 2 times due to AM Container for appattempt_1402648970753_0013_000002 exited with exitCode: 126 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
.Failing this attempt.. Failing the application.

緣由：yarn給storm on yarn分配的運行機器的jdk版本不是1.7，修改jdk版本後無報錯。

解決：全部yarn部署的集羣的jdk版本都要統一成jdk1.7。

 

③ 加載storm集羣后，storm-yarn launch ~/storm/storm-0.9.0-wip21/conf/storm.yaml ，在storm UI界面沒法看到supervisor的進程。

緣由：爲何storm啓動後，加載了nimbus、core進程，未能加載supervisor進程。

這是因爲「Supervisor將獨佔節點而不會與其餘服務共享節點資源」，正好實驗室hadoop2.0測試集羣只有2臺機器，supervisor已經沒有能夠獨佔的節點，致使未能啓動。

解決：擴展集羣節點數量。另外一個測試折衷的辦法：手動啓動supervisor：storm supervisor & ；壞處是須要本身手動管理supervisor進程，本身kill釋放資源。

 

④mvn編譯的時候，報了域名沒法解析，下載pom失敗的問題。在240的maven上能夠編譯經過，170集羣上出現了這個問題。暫時沒解決。

緣由：原來懷疑是域名有問題，更改了maven倉庫的域名仍然沒有解決；第二次嘗試：將240的本地倉庫拷貝到170機器上，170上仍然沒解決。

解決：緣由是eb170的機器沒法鏈接外網，所以沒法下載外網資源。

 

⑤supervisor沒法啓動

緣由：

1.yarn集羣的配置有問題或者內存不夠用，致使沒法啓動

2.未可以解決，多是內存不夠用。有待研究清楚。

解決：todo。