storm on yarn安裝時 提交到yarn失敗 failed

   最近在部署storm on yarn ，部署參考文章

在安裝完zookeeper，配置好storm 和storm on yarn後，啓動zookeeper，其中zookeeper的port爲2181,
而後經過mvn package 編譯工程，發現會出現錯誤，而後使用mvn packet -DskipTests 從新編譯，跳過test
而後向yarn 提交storm任務，storm-yarn launch <path to your storm.yaml file>

提交後查看localhost:8088,發現任務failed，查看錯誤信息發現錯誤以下
http://www.tuicool.com/articles/BFr2Yv
http://blog.csdn.net/jiushuai/article/details/18729367

Application application_1411179375629_0005 failed 2 times due to AM Container for appattempt_1411179375629_0005_000002 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application. 

查看log信息發現具體錯誤以下

14/09/19 20:44:55 INFO yarn.MasterServer: Starting Master Thrift Server
14/09/19 20:44:55 ERROR auth.ThriftServer: ThriftServer is being stopped due to: org.apache.thrift7.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:9000.
org.apache.thrift7.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:9000.
        at org.apache.thrift7.transport.TNonblockingServerSocket.<init>(TNonblockingServerSocket.java:89)
        at org.apache.thrift7.transport.TNonblockingServerSocket.<init>(TNonblockingServerSocket.java:68)
        at org.apache.thrift7.transport.TNonblockingServerSocket.<init>(TNonblockingServerSocket.java:61)
        at backtype.storm.security.auth.SimpleTransportPlugin.getServer(SimpleTransportPlugin.java:47)
        at backtype.storm.security.auth.ThriftServer.serve(ThriftServer.java:52)
        at com.yahoo.storm.yarn.MasterServer.main(MasterServer.java:175)

 在啓動Master Thrift Server時發生錯誤，錯誤爲端口錯誤，由於9000端口已經被hdfs佔用監聽，所以不能建立ServerSocket

致謝google，搜到的解決辦法爲修改端口號，帖子以下

https://groups.google.com/forum/#!topic/storm-yarn/A1ds1M6qmN8

修改storm-yarn-master/src/main/resources/master_defaults.yaml，將其中的master.thrift.port修改成一個其餘的合適的值，我修改成9001

而後再編譯工程，從新提交，這時發現任務沒有failed，可是訪問localhost:7070，不能訪問，查看log後發現錯誤爲nimbus沒有啓動成功，錯誤以下：

15/07/15 03:36:06 ERROR yarn.MasterServer: Unhandled error in AM: 
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested virtual cores < 0, or requested virtual cores > max configured, requestedVirtualCores=130, maxVirtualCores=8
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:213)
    at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests(RMServerUtils.java:97)
    at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:502)
    ... ... ... ... ... ...
    at com.yahoo.storm.yarn.MasterServer$1.run(MasterServer.java:69)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException): Invalid resource request, requested virtual cores < 0, or requested virtual cores > max configured, requestedVirtualCores=130, maxVirtualCores=8
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:213)
    at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests(RMServerUtils.java:97)

    ... ... ... ... ... ...

 at com.sun.proxy.$Proxy7.allocate(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) ... 9 more 15/07/15 03:36:06 INFO yarn.StormMasterServerHandler: stopping supervisors... 15/07/15 03:36:06 INFO yarn.StormMasterServerHandler: stopping UI... 15/07/15 03:36:06 INFO yarn.StormMasterServerHandler: stopping nimbus...

意思是申請的virtual cores超過最大限制maxVirtualCores，google後找到帖子

hadoop - Why cannot more than 32 cores be requested from YARN to run a job? - Stack Overflow
http://stackoverflow.com/questions/29780401/why-cannot-more-than-32-cores-be-requested-from-yarn-to-run-a-job

所以修改yarn-site.xml以下

<property>
  <name>yarn.nodemanager.resource.cpu-vcores</name>
  <value>8</value>
</property>

從新啓動yarn，而後提交，成功！！！

若是發現如下錯誤

2015-07-15 03:19:28 o.a.z.ClientCnxn [INFO] Opening socket connection to server MMC/192.168.1.200:2181
2015-07-15 03:19:28 o.a.z.ClientCnxn [WARN] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: 拒絕鏈接

有多是zookeeper沒有啓動，啓動便可

時刻注意本身的防火牆有沒有關閉，有些不知名的緣由是由於防火牆沒有關閉形成的！

 

用了三四天時間才解決了這個問題，期間走了很多彎路，由於剛開始接觸hadoop storm on yarn，所以有時候出錯誤了不知道怎麼去查錯，開始時只是本身估計是哪出了問題，而後改一下從新跑一下，結果仍是不行，後來學會了去查錯誤log，在masterhost：8088/logs/下就是全部的錯誤日誌，而後查找相關的錯誤日誌排錯事半功倍

 

致謝：google

吐槽一下萬惡的網G絡F審W查￥制#度，我操我操我操我操我操！！！Internet上的閉關鎖國！！！