NodeManager OOM掛掉問題解決

博客原文
hackershell

在更換JDK1.625到JDK1.745後，集羣出現頻繁死掉NM，出現結果爲以下：

2015-08-12 16:35:06,662 FATAL org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[process reaper,10,system] threw an Error. Shutting down now...
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at java.lang.UNIXProcess$ProcessPipeInputStream.drainInputStream(UNIXProcess.java:267)
at java.lang.UNIXProcess$ProcessPipeInputStream.processExited(UNIXProcess.java:280)
at java.lang.UNIXProcess.processExited(UNIXProcess.java:187)
at java.lang.UNIXProcess$3.run(UNIXProcess.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

和相似的

2015-08-12 16:37:56,893 FATAL org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[process reaper,10,system] threw an Error. Shutting down now...
java.lang.OutOfMemoryError: Java heap space
at java.lang.UNIXProcess$ProcessPipeInputStream.drainInputStream(UNIXProcess.java:267)
at java.lang.UNIXProcess$ProcessPipeInputStream.processExited(UNIXProcess.java:280)
at java.lang.UNIXProcess.processExited(UNIXProcess.java:187)
at java.lang.UNIXProcess$3.run(UNIXProcess.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

在google搜索關鍵字hadoop UNIXProcess drainInputStream，找到關於JDK7的一些bug，在NM負載高的狀況下，出現OOM問題。 詳情請看HADOOP-10146

和一些相關解釋：

JDK-8027348

JDK-8024521

後來更換JDK1.7_67則沒出現OOM的問題