Spark的Yarn運行模式部署實戰案例

時間 2020-09-13

標籤 spark yarn 運行模式部署實戰案例欄目 Spark 简体版

原文原文鏈接

　　　　　　　　　　　Spark的Yarn運行模式部署實戰案例java

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　做者：尹正傑node

一.啓動Hadoop集羣c++

1>.修改yarn的配置文件git

[root@hadoop101.yinzhengjie.org.cn ~]# vim /yinzhengjie/softwares/ha/etc/hadoop/yarn-site.xml 
[root@hadoop101.yinzhengjie.org.cn ~]# 
[root@hadoop101.yinzhengjie.org.cn ~]# cat /yinzhengjie/softwares/ha/etc/hadoop/yarn-site.xml 
<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<configuration>

<!-- Site specific YARN configuration properties -->

    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
        <description>Reducer獲取數據的方式</description>
    </property>

    <property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
        <description>啓用resourcemanager的HA功能</description>
    </property>


    <property>
        <name>yarn.resourcemanager.cluster-id</name>
        <value>yinzhengjie-yarn-ha</value>
        <description>標識集羣,以確保RM不會接替另外一個羣集的活動狀態</description>
    </property>


    <property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm101,rm105</value>
        <description>ResourceManager的邏輯ID列表</description>
    </property>


    <property>
        <name>yarn.resourcemanager.hostname.rm101</name>
        <value>hadoop101.yinzhengjie.org.cn</value>
        <description>指定rm101邏輯別名對應的真實服務器地址,即您能夠理解添加映射關係.</description>
    </property>

    <property>
        <name>yarn.resourcemanager.hostname.rm105</name>
        <value>hadoop105.yinzhengjie.org.cn</value>
        <description>指定rm105邏輯別名對應的真實服務器地址,即您能夠理解添加映射關係.</description>
    </property>

    <property>
        <name>yarn.resourcemanager.zk-address</name>
        <value>hadoop101.yinzhengjie.org.cn:2181,hadoop102.yinzhengjie.org.cn:2181,hadoop103.yinzhengjie.org.cn:2181</value>
        <description>指定zookeeper集羣的地址</description>
    </property>

    <property>
        <name>yarn.resourcemanager.recovery.enabled</name>
        <value>true</value>
        <description>啓用自動故障轉移(即自動恢復功能)</description>
    </property>

    <property>
        <name>yarn.resourcemanager.store.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
        <description>指定resourcemanager的狀態信息存儲在zookeeper集羣</description>
    </property>

    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
        <description>啓用或禁用日誌聚合的配置,默認爲false,即禁用,將該值設置爲true,表示開啓日誌彙集功能使能</description>
    </property>

    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>604800</value>
        <description>刪除聚合日誌前要保留多長時間(默認單位是秒),默認值是"-1"表示禁用,請注意,將此值設置得過小,您將向Namenode發送垃圾郵件.</description>
    </property>


    <property>
        <name>yarn.log-aggregation.retain-check-interval-seconds</name>
        <value>3600</value>
        <description>單位爲秒,檢查聚合日誌保留之間的時間.若是設置爲0或負值,那麼該值將被計算爲聚合日誌保留時間的十分之一;請注意,將此值設置得過小,您將向名稱節點發送垃圾郵件.</description>
    </property>

    <!-- 配置Spark ON YARN-->
     <property>
         <name>yarn.nodemanager.pmem-check-enabled</name>
         <value>false</value>
        <description>是否啓動一個線程檢查每一個任務正使用的物理內存量，若是任務超出分配值，則直接將其殺掉，默認是true</description>
     </property>

     <property>
         <name>yarn.nodemanager.vmem-check-enabled</name>
         <value>false</value>
         <description>是否啓動一個線程檢查每一個任務正使用的虛擬內存量，若是任務超出分配值，則直接將其殺掉，默認是true</description>
     </property>

</configuration>
[root@hadoop101.yinzhengjie.org.cn ~]#

[root@hadoop101.yinzhengjie.org.cn ~]# vim /yinzhengjie/softwares/ha/etc/hadoop/yarn-site.xml

[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh /yinzhengjie/softwares/ha/etc/hadoop/yarn-site.xml
******* [hadoop102.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/ha/etc/hadoop/yarn-site.xml] *******
命令執行成功
******* [hadoop103.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/ha/etc/hadoop/yarn-site.xml] *******
命令執行成功
******* [hadoop104.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/ha/etc/hadoop/yarn-site.xml] *******
命令執行成功
******* [hadoop105.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/ha/etc/hadoop/yarn-site.xml] *******
命令執行成功
******* [hadoop106.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/ha/etc/hadoop/yarn-site.xml] *******
命令執行成功
[root@hadoop101.yinzhengjie.org.cn ~]# 
[root@hadoop101.yinzhengjie.org.cn ~]#

[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh /yinzhengjie/softwares/ha/etc/hadoop/yarn-site.xml　　　　　　　　#分發配置文件

2>.啓動zookeeper集羣golang

[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
8239 Jps

hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
8745 Jps

hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
9677 Jps

hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6486 Jps

hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
8208 Jps

hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6730 Jps

[root@hadoop101.yinzhengjie.org.cn ~]#