yarn多租戶管理

時間  2019-11-18
標籤 yarn 租戶 管理 欄目 Hadoop 简体版
原文   原文鏈接

yarn多租戶配置管理(CapacityScheduler)java

hadoop的版本爲2.7node

一：  多租戶實現前，只有一個default隊列web

 

 

二 配置文件修改算法

yarn-site.xmlexpress

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<configuration>
    <property>
        <name>yarn.resourcemanager.scheduler.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>　　<!--設置資源調度器-->
    </property>
    <property>
                <name>yarn.resourcemanager.scheduler.monitor.enable</name>
                <value>true</value>
        </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>FoxLog17.engine.wx</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>4</value>
    </property>

    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>16000</value>
    </property>
    
        <property>
         <name>yarn.nodemanager.vmem-pmem-ratio</name>
        <value>2.1</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>yarn.resourcemanager.store.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore</value>
    </property>
    <property>
        <name>yarn.resourcemanager.fs.state-store.uri</name>
        <value>/tmp/yarn/system/rmstore</value>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>6192</value>
    </property>
    <property>
                <name>yarn.scheduler.minimum-allocation-mb</name>
                <value>1280</value>
        </property>
        <property>
                <name>yarn.resourcemanager.recovery.enabled</name>
                <value>true</value>
        </property>


    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>    
    <property>
          <name>yarn.log-aggregation.retain-seconds</name>
            <value>607800</value>
    </property>
    <property>
        <name>yarn.log-aggregation.retain-check-interval-seconds</name>
        <value>10800</value>                                                           </property>
    <property>
        <name>yarn.nodemanager.local-dirs</name>
        <value>/hdpdata/hadoop/tmp/nm-local-dirs</value>
    </property>
    <property>
        <name>yarn.nodemanager.log-dirs</name>
        <value>/hdpdata/hadoop/tmp/nm-log-dirs</value>
    </property>
    <property>
        <name>yarn.nodemanager.log.retain-seconds</name>
        <value>10800</value>
    </property>
        <property>
            <name>yarn.nodemanager.remote-app-log-dir</name>
           <value>/tmp/logs</value>
        </property>
        <property>
                <name>yarn.nodemanager.remote-app-log-dir-suffix</name>
               <value>logs</value>
        </property>
        <property>
               <name>yarn.nodemanager.delete.debug-delay-sec</name>
              <value>600</value>
        </property>



         <!--property>
               <name>yarn.nodemanager.localizer.cache.target-size-mb</name>
               <value>2048</value>
        </property-->    
        <!--多租戶配置，新增，用於隊列的權限管理，和mapred-site.xml配置文件配合使用，例如不一樣租戶不能隨意kill任務，只能kill屬於本身的隊列任務，超級用戶除外-->
        <property>
            <name>yarn.acl.enable</name>
            <value>true</value>
        </property>

　　　　　　<property>apache

　　　　　　　　<name>yarn.admin.acl</name>架構

　　　　　　　　<value>hadp</value>app

　　　　　　</property>less


</configuration>

 mapred-site.xmldom

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

    <property>
        <name>mapreduce.reduce.shuffle.parallelcopies</name>
        <value>10</value>
    </property>
    <!--property>
        <name>yarn.app.mapreduce.am.resource.mb</name>
        <value>1024</value>
    </property-->    
    <!--property>
        <name>mapreduce.task.io.sort.mb</name>
        <value>100</value>        
    </property-->
    <!--property>    
        <name>mapreduce.task.io.sort.factor</name>
        <value>10</value>
     </property-->

    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>FoxLog17.engine.wx:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>        
            <value>FoxLog17.engine.wx:19888</value>
    </property>


        <!--property>
             <name>mapred.child.java.opts</name>
              <value>-Xmx6000m</value> 
        </property>
         <property> 
          <name>mapred.map.child.java.opts</name> 
          <value>-Xmx2000</value> 
      </property>
      <property> 
           <name>mapred.reduce.child.java.opts</name> 
           <value>-Xmx3024m</value>                                              
     </property-->


             <property>
                     <name>mapreduce.map.memory.mb</name>
                             <value>2048</value>
                                 </property>

        <property>
                <name>mapreduce.map.java.opts</name>
                        <value>-Xmx1536M</value>
                            </property>

        <property>
                <name>mapreduce.reduce.memory.mb</name>
                        <value>4096</value>
                            </property>

        <property>
                <name>mapreduce.reduce.java.opts</name>
                        <value>-Xmx3584M</value>
                            </property>

    <property>
        <name>mapred.child.env</name>
        <value>LD_LIBRARY_PATH=/home/hadoop/lzo/lib</value>
    </property>
    
    <!--多租戶配置，新增，和yarn-site.xml中acl配合使用-->
    <property>
              <name>mapreduce.cluster.acls.enabled</name>
        <value>true</value>
    </property>


</configuration>

capacity-scheduler.xml

<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

  <property>
    <name>yarn.scheduler.capacity.maximum-applications</name>
    <value>10000</value>
    <description>
      Maximum number of applications that can be pending and running.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.1</value>
    <description>
      Maximum percent of resources in the cluster which can be used to run 
      application masters i.e. controls number of concurrent running
      applications.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.resource-calculator</name>
    <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>
    <description>
      The ResourceCalculator implementation to be used to compare 
      Resources in the scheduler.
      The default i.e. DefaultResourceCalculator only uses Memory while
      DominantResourceCalculator uses dominant-resource to compare 
      multi-dimensional resources such as Memory, CPU etc.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>default,analysis</value>  <!--增長新的隊列-->
    <description>
      The queues at the this level (root is the root queue).
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.capacity</name>
    <value>70</value>    <!--修改資源配置比-->
    <description>Default queue target capacity.</description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
    <value>1.4</value>    
    <description>
      Default queue user limit a percentage from 0.0 to 1.0.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
    <value>100</value>
    <description>
      The maximum capacity of the default queue. 
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.state</name>
    <value>RUNNING</value>
    <description>
      The state of the default queue. State can be one of RUNNING or STOPPED.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
    <value>*</value>
    <description>
      The ACL of who can submit jobs to the default queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
    <value>*</value>
    <description>
      The ACL of who can administer jobs on the default queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.node-locality-delay</name>
    <value>40</value>
    <description>
      Number of missed scheduling opportunities after which the CapacityScheduler 
      attempts to schedule rack-local containers. 
      Typically this should be set to number of nodes in the cluster, By default is setting 
      approximately number of nodes in one rack which is 40.
    </description>
  </property>
  
    <!--新添加配置,20191022新增-->
    <property>
        <name>yarn.scheduler.capacity.root.analysis.capacity</name>
        <value>30</value>       <!--理想資源配置比,全部隊列比之和爲100-->
    </property>

    <property>
        <name>yarn.scheduler.capacity.root.analysis.user-limit-factor</name>
        <value>1.9</value>　　　　<！--能夠配置爲容許單個用戶獲取更多資源的隊列容量的倍數。若是值小於1，那麼該用戶使用的資源僅限該隊列資源，而不會大量去佔用其餘隊列的閒暇資源。-->
    </property>

    <property>
        <name>yarn.scheduler.capacity.root.analysis.maximum-capacity</name>
        <value>50</value>　　　　　　<!--隊列使用的資源上線-->
    </property>

    <property>
        <name>yarn.scheduler.capacity.root.analysis.state</name>
        <value>RUNNING</value>     <!--STOOPED時，意思是該隊列再也不使用-->
    </property>

    <property>
        <name>yarn.scheduler.capacity.root.analysis.acl_submit_applications</name>
        <value>*</value>
    </property>

    <property>
        <name>yarn.scheduler.capacity.root.analysis.acl_administer_queue</name>
        <value>*</value>
    </property>


</configuration>

以上配置的解釋，僅我的理解的觀點，能夠結合官方文檔進行對比。

 

三： 更新yarn參數

yarn rmadmin -refreshQueues

查看ui：

 

 

四： 提交任務指定queue（推薦方法，也存在其餘方式，不進行列舉）

mapreduce：代碼中指定隊列；  

config.set("mapred.job.queue.name", "analysis");

hive：修改配置文件；由於hive通常是用於OLAP平臺，能夠把隊列限制死；

hive-site.xml
<property>
    <name>mapreduce.job.queuename</name>
    <value>analysis</value>
</property>

spark：運行腳本指定queue 或 代碼中指定

1- 腳本方式
--queue analysis

2- 代碼方式
saprkConf.set("yarn,spark.queue", "your_queue_name")

 

五：聲明點

1- 搶佔資源基於先等待後強制搶佔的原則。默認配置是不強制搶佔資源。隊列借出去的資源被回收後纔會歸還；也能夠強制搶回資源（通常會有延時，默認配置不會強制搶回資源）。以container爲單位。
2- 資源能夠共用，而且能夠達到最大限制資源，通常要配合user-limit-factor參數（默認爲1，因此不能佔用大量隊列外資源，因此該參數通常設置大些）；
3- CapacityScheduler資源調度器是沒辦法解決因資源短缺形成的任務等待問題。
4- 若是資源不足夠多，會啓動較少的container。（例如想啓動一個四個container的任務，因爲資源不足，只夠啓動兩個，那麼會先啓動兩個container，若是有資源釋放，則會啓動預期的四個container）

 

六： Fair Scheduler與Capacity Scheduler對比（來自Hadoop技術內幕深刻解析YARN架構設計與實現原理）

FairScheduler的優勢：提升小應用程序響應時間。因爲採用了最大最小公平算法，小做業能夠快速獲取資源並運行完成。
    隨着Hadoop版本的演化，Fair Scheduler和Capacity Scheduler的功能愈來愈完善，包括層級隊列組織方式、資源搶佔、批量調度等，也正因如此，兩個調度器同質化愈來愈嚴重。目前看來，兩個調度器在應用場景、支持的特性、內部實現等方面很是接近，而因爲Fair Scheduler支持多種調度策略，所以能夠認爲Fair Scheduler具有了Capacity Scheduler具備的全部功能。

 

建議擴展閱讀：董西城--Hadoop技術內幕深刻解析YARN架構設計與實現原理 6.5 和 6.6節。

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程