Hive運行Tez引擎任務出現always pending

問題描述

在Hive中使用Tez計算引擎執行SQL任務時，Map和Reduce都處於Pending狀態。

解決思路

查看任務的ApplicationMaster的container日誌以下：

2016-09-13 17:14:52,103 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: Releasing held container as either there are pending but  unmatched requests or this is not a session, containerId=container_e14_1473755328049_0004_01_000002, pendingTasks=1, isSession=true. isNew=true
2016-09-13 17:14:52,350 [WARN] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Held container expected to be not null for a non-AM-released container
2016-09-13 17:14:52,351 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Ignoring unknown container: container_e14_1473755328049_0004_01_000002

Hive執行Tez引擎任務中Cpu資源設置涉及到的主要參數以下所示：

• mapreduce.map.cpu.vcores
• mapreduce.reduce.cpu.vcores
• yarn.scheduler.minimum-allocation-vcores
• hive.tez.cpu.vcores

mapreduce.map.cpu.vcores
Default Value: 1
The number of virtual cores to request from the scheduler for each map task.

mapreduce.reduce.cpu.vcores
Default Value: 1
The number of virtual cores to request from the scheduler for each reduce task.

yarn.scheduler.minimum-allocation-vcores
Default Value: -1
The minimum allocation for every container request at the RM, in terms of virtual CPU cores. Requests lower than this won't take effect, and the specified value will get allocated the minimum.

hive.tez.cpu.vcores
Default Value: -1
Added In: Hive 0.14.0 with HIVE-8452
By default Tez will ask for however many CPUs MapReduce is configured to use per container. This can be used to overwrite the default.

解決

查看ambari上Hive的參數配置發現，參數hive.tez.cpu.vcores被設置爲10。

在Hive客戶端中set此參數爲默認值或者1便可。

進一步測試

• 當hive.tez.cpu.vcores設置小於1時，將被mapreduce.map.cpu.vcores值覆蓋。
• mapreduce.reduce.cpu.vcores對於Tez任務來講不起任何做用不起做用。
• yarn.scheduler.minimum-allocation-vcores對於Tez任務來講不起任何做用不起做用。

測試結果以下僞代碼：

if hive.tez.cpu.vcores = 1 then
   任務Running。
elseif hive.tez.cpu.vcores > 1 then
   任務Pending。
elseif hive.tez.cpu.vcores < 1 then
   if mapreduce.map.cpu.vcores = 1 or 0 then
      任務Running
   elseif mapreduce.map.cpu.vcores > 1 then
      任務Pending
   elseif mapreduce.map.cpu.vcores < 0 then
      任務Error