hadoop yarn 實戰錯誤彙總

1.hadoop yarn 運行wordcount時執行完成，可是返回錯誤

錯誤信息以下：

15/09/05 03:48:02 INFO mapreduce.Job: Job job_1441395011668_0001 failed with state FAILED due to: Application application_1441395011668_0001 failed 2 times due to AM Container for appattempt_1441395011668_0001_000002 exited with  exitCode: 1
For more detailed output, check application tracking page:http://macmaster.hadoop:8088/proxy/application_1441395011668_0001/AThen, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1441395011668_0001_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
        at org.apache.hadoop.util.Shell.run(Shell.java:455)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
15/09/05 03:48:02 INFO mapreduce.Job: Counters: 0

有多是mapreduce.jobhistory.address沒有配置，由於yarn要讀取jobhistory信息來獲取是否執行成功，能夠修改yarn-site.xml以下：

<configuration>
   <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>macmaster.hadoop:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>macmaster.hadoop:19888</value>
    </property>
</configuration>

 

2.60000 millis timeout while waiting for channel to be ready for read. ch

有多是讀寫等待超時引發的錯誤，我是執行randomtextwriter和randomwriter時引發的，因爲CPU和內存性能較差，而且計算數據量較大，引發了讀取hdfs時很慢致使超時，能夠添加hdfs-site.xml以下：

<property>
        <name>dfs.datanode.socket.write.timeout</name>
        <value>600000</value> #其中默認爲60000
    </property>
    <property>
        <name>dfs.socket.timeout</name>
        <value>600000</value>  #其中默認爲60000</property>