Hadoop之YARN命令

概述

YARN命令是調用bin/yarn腳本文件,若是運行yarn腳本沒有帶任何參數,則會打印yarn全部命令的描述。html

使用: yarn [--config confdir] COMMAND [--loglevel loglevel] [GENERIC_OPTIONS] [COMMAND_OPTIONS]node

YARN有一個參數解析框架,採用解析泛型參數以及運行類。web

命令參數 描述
--config confdir 指定一個默認的配置文件目錄,默認值是: ${HADOOP_PREFIX}/conf.
--loglevel loglevel 重載Log級別。有效的日誌級別包含:FATAL, ERROR, WARN, INFO, DEBUG, and TRACE。默認是INFO。
GENERIC_OPTIONS YARN支持表A的通用命令項。
COMMAND COMMAND_OPTIONS YARN分爲用戶命令和管理員命令。

 

表A:apache

通用項 Description
-archives <comma separated list of archives> 用逗號分隔計算中未歸檔的文件。 僅僅針對JOB。
-conf <configuration file> 制定應用程序的配置文件。
-D <property>=<value> 使用給定的屬性值。
-files <comma separated list of files> 用逗號分隔的文件,拷貝到Map reduce機器,僅僅針對JOB
-jt <local> or <resourcemanager:port> 指定一個ResourceManager. 僅僅針對JOB。
-libjars <comma seperated list of jars> 將用逗號分隔的jar路徑包含到classpath中去,僅僅針對JOB。

 

 

用戶命令:

 

對於Hadoop集羣用戶頗有用的命令:服務器

 

application

使用: yarn application [options]app

命令選項 描述
-appStates <States> 使用-list命令,基於應用程序的狀態來過濾應用程序。若是應用程序的狀態有多個,用逗號分隔。 有效的應用程序狀態包含
以下: ALL, NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED
-appTypes <Types> 使用-list命令,基於應用程序類型來過濾應用程序。若是應用程序的類型有多個,用逗號分隔。
-list 從RM返回的應用程序列表,使用-appTypes參數,支持基於應用程序類型的過濾,使用-appStates參數,支持對應用程序狀態的過濾。
-kill <ApplicationId> kill掉指定的應用程序。
-status <ApplicationId> 打印應用程序的狀態。

示例1:框架

 

[hduser@hadoop0 bin]$ ./yarn application -list -appStates ACCEPTED
15/08/10 11:48:43 INFO client.RMProxy: Connecting to ResourceManager at hadoop1/10.0.1.41:8032
Total number of applications (application-types: [] and states: [ACCEPTED]):1
Application-Id	                Application-Name Application-Type User	 Queue	 State	  Final-State Progress Tracking-URL
application_1438998625140_1703	MAC_STATUS	 MAPREDUCE	  hduser default ACCEPTED UNDEFINED   0%       N/A

示例2:webapp

 

 

[hduser@hadoop0 bin]$ ./yarn application -list
15/08/10 11:43:01 INFO client.RMProxy: Connecting to ResourceManager at hadoop1/10.0.1.41:8032
Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):1
Application-Id	               Application-Name	Application-Type  User   Queue   State    Final-State   Progress Tracking-URL
application_1438998625140_1701 MAC_STATUS	MAPREDUCE	  hduser default ACCEPTED UNDEFINED	0%	 N/A

 

示例3:ide

 

[hduser@hadoop0 bin]$ ./yarn application -kill application_1438998625140_1705
15/08/10 11:57:41 INFO client.RMProxy: Connecting to ResourceManager at hadoop1/10.0.1.41:8032
Killing application application_1438998625140_1705
15/08/10 11:57:42 INFO impl.YarnClientImpl: Killed application application_1438998625140_1705

 

 

 

applicationattempt

使用: yarn applicationattempt [options]oop

命令選項 描述
-help 幫助
-list <ApplicationId> 獲取到應用程序嘗試的列表,其返回值ApplicationAttempt-Id 等於 <Application Attempt Id>
-status <Application Attempt Id> 打印應用程序嘗試的狀態。

打印應用程序嘗試的報告。

示例1:

 

[hadoop@hadoopcluster78 bin]$ yarn applicationattempt -list application_1437364567082_0106
15/08/10 20:58:28 INFO client.RMProxy: Connecting to ResourceManager at hadoopcluster79/10.0.1.79:8032
Total number of application attempts :1
ApplicationAttempt-Id	               State	AM-Container-Id	                       Tracking-URL
appattempt_1437364567082_0106_000001   RUNNING	container_1437364567082_0106_01_000001 http://hadoopcluster79:8088/proxy/application_1437364567082_0106/

示例2:

 

 

[hadoop@hadoopcluster78 bin]$ yarn applicationattempt -status appattempt_1437364567082_0106_000001
15/08/10 21:01:41 INFO client.RMProxy: Connecting to ResourceManager at hadoopcluster79/10.0.1.79:8032
Application Attempt Report : 
	ApplicationAttempt-Id : appattempt_1437364567082_0106_000001
	State : FINISHED
	AMContainer : container_1437364567082_0106_01_000001
	Tracking-URL : http://hadoopcluster79:8088/proxy/application_1437364567082_0106/jobhistory/job/job_1437364567082_0106
	RPC Port : 51911
	AM Host : hadoopcluster80
	Diagnostics :

 

 

classpath

使用: yarn classpath

打印須要獲得Hadoop的jar和所須要的lib包路徑

 

[hadoop@hadoopcluster78 bin]$ yarn classpath
/home/hadoop/apache/hadoop-2.4.1/etc/hadoop:/home/hadoop/apache/hadoop-2.4.1/etc/hadoop:/home/hadoop/apache/hadoop-2.4.1/etc/hadoop:/home/hadoop/apache/hadoop-2.4.1/share/hadoop/common/lib/*:/home/hadoop/apache/hadoop-2.4.1/share/hadoop/common/*:/home/hadoop/apache/hadoop-2.4.1/share/hadoop/hdfs:/home/hadoop/apache/hadoop-2.4.1/share/hadoop/hdfs/lib/*:/home/hadoop/apache/hadoop-2.4.1/share/hadoop/hdfs/*:/home/hadoop/apache/hadoop-2.4.1/share/hadoop/yarn/lib/*:/home/hadoop/apache/hadoop-2.4.1/share/hadoop/yarn/*:/home/hadoop/apache/hadoop-2.4.1/share/hadoop/mapreduce/lib/*:/home/hadoop/apache/hadoop-2.4.1/share/hadoop/mapreduce/*:/home/hadoop/apache/hadoop-2.4.1/contrib/capacity-scheduler/*.jar:/home/hadoop/apache/hadoop-2.4.1/share/hadoop/yarn/*:/home/hadoop/apache/hadoop-2.4.1/share/hadoop/yarn/lib/*

 

 

 

container

使用: yarn container [options]

命令選項 描述
-help 幫助
-list <Application Attempt Id> 應用程序嘗試的Containers列表
-status <ContainerId> 打印Container的狀態

打印container(s)的報告

示例1:

 

[hadoop@hadoopcluster78 bin]$ yarn container -list appattempt_1437364567082_0106_01 
15/08/10 20:45:45 INFO client.RMProxy: Connecting to ResourceManager at hadoopcluster79/10.0.1.79:8032
Total number of containers :25
                  Container-Id	          Start Time	         Finish Time	               State	                Host	                            LOG-URL
container_1437364567082_0106_01_000028	       1439210458659	                   0	             RUNNING	hadoopcluster83:37140	//hadoopcluster83:8042/node/containerlogs/container_1437364567082_0106_01_000028/hadoop
container_1437364567082_0106_01_000016	       1439210314436	                   0	             RUNNING	hadoopcluster84:43818	//hadoopcluster84:8042/node/containerlogs/container_1437364567082_0106_01_000016/hadoop
container_1437364567082_0106_01_000019	       1439210338598	                   0	             RUNNING	hadoopcluster83:37140	//hadoopcluster83:8042/node/containerlogs/container_1437364567082_0106_01_000019/hadoop
container_1437364567082_0106_01_000004	       1439210314130	                   0	             RUNNING	hadoopcluster82:48622	//hadoopcluster82:8042/node/containerlogs/container_1437364567082_0106_01_000004/hadoop
container_1437364567082_0106_01_000008	       1439210314130	                   0	             RUNNING	hadoopcluster82:48622	//hadoopcluster82:8042/node/containerlogs/container_1437364567082_0106_01_000008/hadoop
container_1437364567082_0106_01_000031	       1439210718604	                   0	             RUNNING	hadoopcluster83:37140	//hadoopcluster83:8042/node/containerlogs/container_1437364567082_0106_01_000031/hadoop
container_1437364567082_0106_01_000020	       1439210339601	                   0	             RUNNING	hadoopcluster83:37140	//hadoopcluster83:8042/node/containerlogs/container_1437364567082_0106_01_000020/hadoop
container_1437364567082_0106_01_000005	       1439210314130	                   0	             RUNNING	hadoopcluster82:48622	//hadoopcluster82:8042/node/containerlogs/container_1437364567082_0106_01_000005/hadoop
container_1437364567082_0106_01_000013	       1439210314435	                   0	             RUNNING	hadoopcluster84:43818	//hadoopcluster84:8042/node/containerlogs/container_1437364567082_0106_01_000013/hadoop
container_1437364567082_0106_01_000022	       1439210368679	                   0	             RUNNING	hadoopcluster84:43818	//hadoopcluster84:8042/node/containerlogs/container_1437364567082_0106_01_000022/hadoop
container_1437364567082_0106_01_000021	       1439210353626	                   0	             RUNNING	hadoopcluster83:37140	//hadoopcluster83:8042/node/containerlogs/container_1437364567082_0106_01_000021/hadoop
container_1437364567082_0106_01_000014	       1439210314435	                   0	             RUNNING	hadoopcluster84:43818	//hadoopcluster84:8042/node/containerlogs/container_1437364567082_0106_01_000014/hadoop
container_1437364567082_0106_01_000029	       1439210473726	                   0	             RUNNING	hadoopcluster80:42366	//hadoopcluster80:8042/node/containerlogs/container_1437364567082_0106_01_000029/hadoop
container_1437364567082_0106_01_000006	       1439210314130	                   0	             RUNNING	hadoopcluster82:48622	//hadoopcluster82:8042/node/containerlogs/container_1437364567082_0106_01_000006/hadoop
container_1437364567082_0106_01_000003	       1439210314129	                   0	             RUNNING	hadoopcluster82:48622	//hadoopcluster82:8042/node/containerlogs/container_1437364567082_0106_01_000003/hadoop
container_1437364567082_0106_01_000015	       1439210314436	                   0	             RUNNING	hadoopcluster84:43818	//hadoopcluster84:8042/node/containerlogs/container_1437364567082_0106_01_000015/hadoop
container_1437364567082_0106_01_000009	       1439210314130	                   0	             RUNNING	hadoopcluster82:48622	//hadoopcluster82:8042/node/containerlogs/container_1437364567082_0106_01_000009/hadoop
container_1437364567082_0106_01_000030	       1439210708467	                   0	             RUNNING	hadoopcluster83:37140	//hadoopcluster83:8042/node/containerlogs/container_1437364567082_0106_01_000030/hadoop
container_1437364567082_0106_01_000012	       1439210314435	                   0	             RUNNING	hadoopcluster84:43818	//hadoopcluster84:8042/node/containerlogs/container_1437364567082_0106_01_000012/hadoop
container_1437364567082_0106_01_000027	       1439210444354	                   0	             RUNNING	hadoopcluster84:43818	//hadoopcluster84:8042/node/containerlogs/container_1437364567082_0106_01_000027/hadoop
container_1437364567082_0106_01_000026	       1439210428514	                   0	             RUNNING	hadoopcluster83:37140	//hadoopcluster83:8042/node/containerlogs/container_1437364567082_0106_01_000026/hadoop
container_1437364567082_0106_01_000017	       1439210314436	                   0	             RUNNING	hadoopcluster84:43818	//hadoopcluster84:8042/node/containerlogs/container_1437364567082_0106_01_000017/hadoop
container_1437364567082_0106_01_000001	       1439210306902	                   0	             RUNNING	hadoopcluster80:42366	//hadoopcluster80:8042/node/containerlogs/container_1437364567082_0106_01_000001/hadoop
container_1437364567082_0106_01_000002	       1439210314129	                   0	             RUNNING	hadoopcluster82:48622	//hadoopcluster82:8042/node/containerlogs/container_1437364567082_0106_01_000002/hadoop
container_1437364567082_0106_01_000025	       1439210414171	                   0	             RUNNING	hadoopcluster83:37140	//hadoopcluster83:8042/node/containerlogs/container_1437364567082_0106_01_000025/hadoop

 

 

示例2:

 

[hadoop@hadoopcluster78 bin]$ yarn container -status container_1437364567082_0105_01_000020 
15/08/10 20:28:00 INFO client.RMProxy: Connecting to ResourceManager at hadoopcluster79/10.0.1.79:8032
Container Report : 
	Container-Id : container_1437364567082_0105_01_000020
	Start-Time : 1439208779842
	Finish-Time : 0
	State : RUNNING
	LOG-URL : //hadoopcluster83:8042/node/containerlogs/container_1437364567082_0105_01_000020/hadoop
	Host : hadoopcluster83:37140
	Diagnostics : null

 

 

jar

使用: yarn jar <jar> [mainClass] args...

運行jar文件,用戶能夠將寫好的YARN代碼打包成jar文件,用這個命令去運行它。

 

logs

使用: yarn logs -applicationId <application ID> [options]

注:應用程序沒有完成,該命令是不能打印日誌的。

命令選項 描述
-applicationId <application ID> 指定應用程序ID,應用程序的ID能夠在yarn.resourcemanager.webapp.address配置的路徑查看(即:ID)
-appOwner <AppOwner> 應用的全部者(若是沒有指定就是當前用戶)應用程序的ID能夠在yarn.resourcemanager.webapp.address配置的路徑查看(即:User)
-containerId <ContainerId> Container Id
-help 幫助
-nodeAddress <NodeAddress> 節點地址的格式:nodename:port (端口是配置文件中:yarn.nodemanager.webapp.address參數指定)

轉存container的日誌。

示例:

 

[hadoop@hadoopcluster78 bin]$ yarn logs -applicationId application_1437364567082_0104  -appOwner hadoop
15/08/10 17:59:19 INFO client.RMProxy: Connecting to ResourceManager at hadoopcluster79/10.0.1.79:8032


Container: container_1437364567082_0104_01_000003 on hadoopcluster82_48622
============================================================================
LogType: stderr
LogLength: 0
Log Contents:

LogType: stdout
LogLength: 0
Log Contents:

LogType: syslog
LogLength: 3673
Log Contents:
2015-08-10 17:24:01,565 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-08-10 17:24:01,580 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
。。。。。。此處省略N萬個字符
// 下面的命令,根據APP的全部者查看LOG日誌,由於application_1437364567082_0104任務我是用hadoop用戶啓動的,因此打印的是以下信息:
[hadoop@hadoopcluster78 bin]$ yarn logs -applicationId application_1437364567082_0104  -appOwner root
15/08/10 17:59:25 INFO client.RMProxy: Connecting to ResourceManager at hadoopcluster79/10.0.1.79:8032
Logs not available at /tmp/logs/root/logs/application_1437364567082_0104
Log aggregation has not completed or is not enabled.

 

 

node

使用: yarn node [options]

命令選項 描述
-all 全部的節點,無論是什麼狀態的。
-list 列出全部RUNNING狀態的節點。支持-states選項過濾指定的狀態,節點的狀態包
含:NEW,RUNNING,UNHEALTHY,DECOMMISSIONED,LOST,REBOOTED。支持--all顯示全部的節點。
-states <States> 和-list配合使用,用逗號分隔節點狀態,只顯示這些狀態的節點信息。
-status <NodeId> 打印指定節點的狀態。

示例1:

 

[hadoop@hadoopcluster78 bin]$ ./yarn node -list -all
15/08/10 17:34:17 INFO client.RMProxy: Connecting to ResourceManager at hadoopcluster79/10.0.1.79:8032
Total Nodes:4
         Node-Id	     Node-State	Node-Http-Address	Number-of-Running-Containers
hadoopcluster82:48622	        RUNNING	hadoopcluster82:8042	                           0
hadoopcluster84:43818	        RUNNING	hadoopcluster84:8042	                           0
hadoopcluster83:37140	        RUNNING	hadoopcluster83:8042	                           0
hadoopcluster80:42366	        RUNNING	hadoopcluster80:8042	                           0

示例2:

 

 

[hadoop@hadoopcluster78 bin]$ ./yarn node -list -states RUNNING
15/08/10 17:39:55 INFO client.RMProxy: Connecting to ResourceManager at hadoopcluster79/10.0.1.79:8032
Total Nodes:4
         Node-Id	     Node-State	Node-Http-Address	Number-of-Running-Containers
hadoopcluster82:48622	        RUNNING	hadoopcluster82:8042	                           0
hadoopcluster84:43818	        RUNNING	hadoopcluster84:8042	                           0
hadoopcluster83:37140	        RUNNING	hadoopcluster83:8042	                           0
hadoopcluster80:42366	        RUNNING	hadoopcluster80:8042	                           0

示例3:

 

 

[hadoop@hadoopcluster78 bin]$ ./yarn node -status hadoopcluster82:48622
15/08/10 17:52:52 INFO client.RMProxy: Connecting to ResourceManager at hadoopcluster79/10.0.1.79:8032
Node Report : 
	Node-Id : hadoopcluster82:48622
	Rack : /default-rack
	Node-State : RUNNING
	Node-Http-Address : hadoopcluster82:8042
	Last-Health-Update : 星期一 10/八月/15 05:52:09:601CST
	Health-Report : 
	Containers : 0
	Memory-Used : 0MB
	Memory-Capacity : 10240MB
	CPU-Used : 0 vcores
	CPU-Capacity : 8 vcores

打印節點的報告。

 

 

queue

使用: yarn queue [options]

命令選項 描述
-help 幫助
-status <QueueName> 打印隊列的狀態

打印隊列信息。

 

version

使用: yarn version

打印hadoop的版本。

 

管理員命令:

 

下列這些命令對hadoop集羣的管理員是很是有用的。

daemonlog

使用:

yarn daemonlog -getlevel <host:httpport> <classname> 
   yarn daemonlog -setlevel <host:httpport> <classname> <level>
參數選項 描述
-getlevel <host:httpport><classname> 打印運行在<host:port>的守護進程的日誌級別。這個命令內部會鏈接http://<host:port>/logLevel?log=<name>
-setlevel <host:httpport> <classname> <level> 設置運行在<host:port>的守護進程的日誌級別。這個命令內部會鏈接http://<host:port>/logLevel?log=<name>

針對指定的守護進程,獲取/設置日誌級別.

示例1:

 

[root@hadoopcluster78 ~]# hadoop daemonlog -getlevel hadoopcluster82:50075 org.apache.hadoop.hdfs.server.datanode.DataNode
Connecting to http://hadoopcluster82:50075/logLevel?log=org.apache.hadoop.hdfs.server.datanode.DataNode
Submitted Log Name: org.apache.hadoop.hdfs.server.datanode.DataNode
Log Class: org.apache.commons.logging.impl.Log4JLogger
Effective level: INFO

[root@hadoopcluster78 ~]# yarn daemonlog -getlevel hadoopcluster79:8088 org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl
Connecting to http://hadoopcluster79:8088/logLevel?log=org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl
Submitted Log Name: org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl
Log Class: org.apache.commons.logging.impl.Log4JLogger
Effective level: INFO

[root@hadoopcluster78 ~]# yarn daemonlog -getlevel hadoopcluster78:19888 org.apache.hadoop.mapreduce.v2.hs.JobHistory
Connecting to http://hadoopcluster78:19888/logLevel?log=org.apache.hadoop.mapreduce.v2.hs.JobHistory
Submitted Log Name: org.apache.hadoop.mapreduce.v2.hs.JobHistory
Log Class: org.apache.commons.logging.impl.Log4JLogger
Effective level: INFO

 

 

nodemanager

使用: yarn nodemanager

啓動NodeManager

 

proxyserver

使用: yarn proxyserver

啓動web proxy server

 

resourcemanager

使用: yarn resourcemanager [-format-state-store]

參數選項 描述
-format-state-store RMStateStore的格式. 若是過去的應用程序再也不須要,則清理RMStateStore, RMStateStore僅僅在ResourceManager沒有運行的時候,才運行RMStateStore

啓動ResourceManager

 

rmadmin

使用:

yarn rmadmin [-refreshQueues]
               [-refreshNodes]
               [-refreshUserToGroupsMapping] 
               [-refreshSuperUserGroupsConfiguration]
               [-refreshAdminAcls] 
               [-refreshServiceAcl]
               [-getGroups [username]]
               [-transitionToActive [--forceactive] [--forcemanual] <serviceId>]
               [-transitionToStandby [--forcemanual] <serviceId>]
               [-failover [--forcefence] [--forceactive] <serviceId1> <serviceId2>]
               [-getServiceState <serviceId>]
               [-checkHealth <serviceId>]
               [-help [cmd]]
參數選項 描述
-refreshQueues 重載隊列的ACL,狀態和調度器特定的屬性,ResourceManager將重載mapred-queues配置文件
-refreshNodes 動態刷新dfs.hosts和dfs.hosts.exclude配置,無需重啓NameNode。
dfs.hosts:列出了容許連入NameNode的datanode清單(IP或者機器名)
dfs.hosts.exclude:列出了禁止連入NameNode的datanode清單(IP或者機器名)
從新讀取hosts和exclude文件,更新容許連到Namenode的或那些須要退出或入編的Datanode的集合。
-refreshUserToGroupsMappings 刷新用戶到組的映射。
-refreshSuperUserGroupsConfiguration 刷新用戶組的配置
-refreshAdminAcls 刷新ResourceManager的ACL管理
-refreshServiceAcl ResourceManager重載服務級別的受權文件。
-getGroups [username] 獲取指定用戶所屬的組。
-transitionToActive [–forceactive] [–forcemanual] <serviceId> 嘗試將目標服務轉爲 Active 狀態。若是使用了–forceactive選項,不須要覈對非Active節點。若是採用了自動故障轉移,這個命令不能使用。雖然你能夠重寫–forcemanual選項,你須要謹慎。
-transitionToStandby [–forcemanual] <serviceId> 將服務轉爲 Standby 狀態. 若是採用了自動故障轉移,這個命令不能使用。雖然你能夠重寫–forcemanual選項,你須要謹慎。
-failover [–forceactive] <serviceId1> <serviceId2> 啓動從serviceId1 到 serviceId2的故障轉移。若是使用了-forceactive選項,即便服務沒有準備,也會嘗試故障轉移到目標服務。若是採用了自動故障轉移,這個命令不能使用。
-getServiceState <serviceId> 返回服務的狀態。(注:ResourceManager不是HA的時候,時不能運行該命令的)
-checkHealth <serviceId> 請求服務器執行健康檢查,若是檢查失敗,RMAdmin將用一個非零標示退出。(注:ResourceManager不是HA的時候,時不能運行該命令的)
-help [cmd] 顯示指定命令的幫助,若是沒有指定,則顯示命令的幫助。

 

scmadmin

使用: yarn scmadmin [options]

參數選項 描述
-help Help
-runCleanerTask Runs the cleaner task

Runs Shared Cache Manager admin client

 

sharedcachemanager

使用: yarn sharedcachemanager

啓動Shared Cache Manager

 

timelineserver

以前yarn運行框架只有Job history server,這是hadoop2.4版本以後加的通用Job History Server,命令爲Application Timeline Server,詳情請看:The YARN Timeline Server

使用: yarn timelineserver

啓動TimeLineServer

相關文章
相關標籤/搜索