hadoop1 & hadoop2 fair-schduler 配置和使用

hadoop1

• 配置 mapred-site.xml,增長以下內容

<property>
        <name>mapred.jobtracker.taskScheduler</name>
        <value>org.apache.hadoop.mapred.FairScheduler</value>
    </property>
    <property>
        <name>mapred.fairscheduler.allocation.file</name>
        <value>/etc/hadoop/conf/pools.xml</value>
    </property>

• 配置 pools.xml，增長以下內容

<queue name="default」>
  <minResources>1024 mb,1vcores</minResources>
  <maxResources>61440 mb,20vcores</maxResources>
  <maxRunningApps>10</maxRunningApps>
  <weight>2.0</weight>
  <schedulingPolicy>fair</schedulingPolicy>
</queue>

<queue name=「hadoop」>
  <minResources>1024 mb,10vcores</minResources>
  <maxResources>3072000 mb,960vcores</maxResources>
  <maxRunningApps>60</maxRunningApps>
  <weight>5.0</weight>
  <schedulingPolicy>fair</schedulingPolicy>
  <aclSubmitApps>hadoop,yarn,spark</aclSubmitApps>
</queue>

<queue name="spark">
  <minResources>1024 mb,10vcores</minResources>
  <maxResources>61440 mb,20vcores</maxResources>
  <maxRunningApps>10</maxRunningApps>
  <weight>4.0</weight>
  <schedulingPolicy>fair</schedulingPolicy>
<aclSubmitApps>yarn,spark</aclSubmitApps>
</queue>

<userMaxAppsDefault>20</userMaxAppsDefault>

• 提交做業指定隊列方式

-Dmapred.job.queue.name=hadoop

hadoop2

• 配置 yarn-site.xml,增長以下內容

<property>
  <name>yarn.resourcemanager.scheduler.class</name>
                 <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>

<property>
   <name>yarn.scheduler.fair.allocation.file</name>
   <value>/home/cluster/conf/hadoop/fair-scheduler.xml</value>
</property>

<property>  
  <name>yarn.scheduler.fair.user-as-default-queue</name>
  //若是但願以用戶名做爲隊列，能夠將該屬性配置爲true，默認爲true，因此若是不想以用戶名爲隊列的，必須顯式的設置成false  
  <value>false</value>  
</property>

• 配置 fair-scheduler.xml，增長以下內容

<queue name="default」>
  <minResources>1024 mb,1vcores</minResources>
  <maxResources>61440 mb,20vcores</maxResources>
  <maxRunningApps>10</maxRunningApps>
  <weight>2.0</weight>
  <schedulingPolicy>fair</schedulingPolicy>
</queue>

<queue name=「hadoop」>
  <minResources>1024 mb,10vcores</minResources>
  <maxResources>3072000 mb,960vcores</maxResources>
  <maxRunningApps>60</maxRunningApps>
  <weight>5.0</weight>
  <schedulingPolicy>fair</schedulingPolicy>
  <aclSubmitApps>hadoop,yarn,spark</aclSubmitApps>
</queue>

<queue name="spark">
  <minResources>1024 mb,10vcores</minResources>
  <maxResources>61440 mb,20vcores</maxResources>
  <maxRunningApps>10</maxRunningApps>
  <weight>4.0</weight>
  <schedulingPolicy>fair</schedulingPolicy>
<aclSubmitApps>yarn,spark</aclSubmitApps>
</queue>

<userMaxAppsDefault>20</userMaxAppsDefault>

• 提交做業指定隊列方式

-Dmapreduce.job.queuename=root.hadoop

spark

• 提交做業指定隊列方式

 --queue=root.spark

版權聲明：本文爲博主原創文章，未經博主容許不得轉載。