（2）YARN的工做流程

Writing YARN Applications

文檔中的啓動過程：

Application submission client向Yarn ResourceManager提交一個Application，RM、NM、AM處理流程。　

首先，建立一個YarnClient對象並start它，而後Client能夠設置ApplicationContext。爲app準備第一個container來contain ApplicationMaster,而後提交Application。

RM在已經指定的Container中啓動ApplicationMaster。AM與YARN集羣通訊，處理Application的執行。在app啓動的過程當中（app的啓動過程當中，AM與RM的通訊是異步的），AM的主要工做包括：

（1）與RM通訊，協商爲以後的Containers分配資源（經過AMRMClientAsync對象，AMRMClientAsync.CallbackHandler指定事件的處理方法）；

（2）Container分配以後，與NodeManagers通訊，啓動它們所在節點的app的Containers（啓動一個Runnable對象，當爲Containers分配資源以後，啓動containers。做爲啓動Container的一部分，AM須要指定帶有啓動信息的ContainerLaunchContext）。

在Application執行的過程當中，ApplicationMaster經過NMClientAsync對象與NodeManagers通訊。全部Containers的事件由NMClientAsync.CallbackHandler處理。

一個callback handler處理Client的start，stop，status update以及error。

 

 

（1） 建立一個YarnClient對象並start它，而後Client能夠設置ApplicationContext，而後向ResourceManager提交Application。

（2）RM向NM發出指令，爲該App啓動第一個Container，並在其中啓動ApplicationMaster

（3）AM向RM註冊

（4）AM採用輪詢的方式向RM的YARN Scheduler申請資源

（5）當AM申請到資源後（即獲取到了空閒節點的信息），與NodeManagers通訊（多個NodeManager），請求啓動計算任務

（6）NodeManagers根據資源量的大小、所需的運行環境，在Container中啓動任務。

（7）各個任務向AM彙報本身的狀態和進度，以便AM掌握各個任務的執行狀況

（8）APP運行完成後，AM向RM註銷並關閉本身。

 

 

 

 

 

原文：

The general concept is that an application submission client submits an application to the YARN ResourceManager (RM). This can be done through setting up a YarnClient object. After YarnClient is started, the client can then set up application context, prepare the very first container of the application that contains the ApplicationMaster (AM), and then submit the application. You need to provide information such as the details about the local files/jars that need to be available for your application to run, the actual command that needs to be executed (with the necessary command line arguments), any OS environment settings (optional), etc. Effectively, you need to describe the Unix process(es) that needs to be launched for your ApplicationMaster.

The YARN ResourceManager will then launch the ApplicationMaster (as specified) on an allocated container. The ApplicationMaster communicates with YARN cluster, and handles application execution. It performs operations in an asynchronous fashion. During application launch time, the main tasks of the ApplicationMaster are: a) communicating with the ResourceManager to negotiate and allocate resources for future containers, and b) after container allocation, communicating YARN *NodeManager*s (NMs) to launch application containers on them. Task a) can be performed asynchronously through an AMRMClientAsync object, with event handling methods specified in a AMRMClientAsync.CallbackHandler type of event handler. The event handler needs to be set to the client explicitly. Task b) can be performed by launching a runnable object that then launches containers when there are containers allocated. As part of launching this container, the AM has to specify the ContainerLaunchContext that has the launch information such as command line specification, environment, etc.

參考：

（1）《Hadoop The Definitive Guide 4th》

（2）http://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html