做業流調度框架 oozie 使用 （二）

oozie 任務調度處理

標籤（空格分隔）： 協做框架

---

• 一：oozie example 運行任務調度案例
• 二：oozie 運行自定的mapreduce 的jar 包
• 三：oozie 調度shell 腳本
• 四：oozie 的coordinator 週期性調度當前任務

---

一： 運行oozie example 案例

1.1 解壓exmaple包

解壓example 包
tar -zxvf oozie-examples.tar.gz

cd /home/hadoop/yangyang/oozie/examples/apps/map-reduce

job.properties      --定義job相關的屬性，好比目錄路徑、namenode節點等。
                    --定義workflow的位置

workflow.xml    --定義工做流相關的配置（start  --end   --kill）（action）
                --mapred.input.dir
                --mapred.output.dir

lib     --目錄，存放job任務須要的資源（jar包）

---

1.2 更改job.properties

nameNode=hdfs://namenode01.hadoop.com:8020
jobTracker=namenode01.hadoop.com:8032
queueName=default
examplesRoot=examples

oozie.wf.application.path=${nameNode}/user/hadoop/${examplesRoot}/apps/map-reduce/workflow.xml
outputDir=map-reduce

1.3 配置workflow.xml 文件：

<workflow-app xmlns="uri:oozie:workflow:0.2" name="map-reduce-wf">
    <start to="mr-node"/>
    <action name="mr-node">
        <map-reduce>
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
                <delete path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/${outputDir}"/>
            </prepare>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
                <property>
                    <name>mapred.mapper.class</name>
                    <value>org.apache.oozie.example.SampleMapper</value>
                </property>
                <property>
                    <name>mapred.reducer.class</name>
                    <value>org.apache.oozie.example.SampleReducer</value>
                </property>
                <property>
                    <name>mapred.map.tasks</name>
                    <value>1</value>
                </property>
                <property>
                    <name>mapred.input.dir</name>
                    <value>/user/${wf:user()}/${examplesRoot}/input-data/text</value>
                </property>
                <property>
                    <name>mapred.output.dir</name>
                    <value>/user/${wf:user()}/${examplesRoot}/output-data/${outputDir}</value>
                </property>
            </configuration>
        </map-reduce>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

1.3 上傳example 目錄到hdfs 上面

hdfs dfs -put example example

1.4 運行oozie 調度任務

bin/oozie job -oozie http://namenode01.hadoop.com:11000/oozie -config examples/apps/map-reduce/job.properties -run

查看狀態：

輸出目錄

二：oozie 運行自定的mapreduce 的jar 包

2.1 在hdfs 上建立上傳目錄

cd /home/hadoop/yangyang/oozie/
hdfs dfs -mkdir oozie-apps

2.2 新建本地的文件用做上傳的目錄

mkdir oozie-apps
cd /home/hadoop/yangyang/oozie/examples/apps
cp -ap map-reduce /home/hadoop/yangyang/oozie/oozie-apps/

cd /homme/hadoop/yangyang/oozie/oozie-appps/map-reduce
mkdir input-data

2.3 拷貝運行的jar包與要運行的job 任務的文件

cp -p mr-wordcount.jar yangyang/oozie/oozie-apps/map-reduce/lib/
cp -p /home/hadoop/wc.input ./input-data

2.4 配置job.properties 文件和workflow.xml

vim job.properties

nameNode=hdfs://namenode01.hadoop.com:8020
jobTracker=namenode01.hadoop.com:8032
queueName=default
examplesRoot=oozie-apps/map-reduce

oozie.wf.application.path=${nameNode}/user/hadoop/${examplesRoot}/workflow.xml
outputDir=oozie-reduce

vim workflow.xml

<workflow-app xmlns="uri:oozie:workflow:0.2" name="wc-map-reduce">
    <start to="mr-node"/>
    <action name="mr-node">
        <map-reduce>
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
                <delete path="${nameNode}/user/hadoop/${examplesRoot}/output-data/${outputDir}"/>
            </prepare>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
                <!--0 new API-->                
                <property>
                    <name>mapred.mapper.new-api</name>
                    <value>true</value>
                </property>
                <property>
                    <name>mapred.reducer.new-api</name>
                    <value>true</value>
                </property>

                <!--1 input-->
                <property>
                    <name>mapred.input.dir</name>
                    <value>/user/hadoop/${examplesRoot}/input-data</value>
                </property>         

                <!--2 mapper class -->              
                <property>
                    <name>mapreduce.job.map.class</name>
                    <value>org.apache.hadoop.wordcount.WordCountMapReduce$WordCountMapper</value>
                </property>
                <property>
                    <name>mapreduce.map.output.key.class</name>
                    <value>org.apache.hadoop.io.Text</value>
                </property>
                <property>
                    <name>mapreduce.map.output.value.class</name>
                    <value>org.apache.hadoop.io.IntWritable</value>
                </property> 

                <!--3 reduer class -->                  
                <property>
                    <name>mapreduce.job.reduce.class</name>
                    <value>org.apache.hadoop.wordcount.WordCountMapReduce$WordCountReducer</value>
                </property>
                <property>
                    <name>mapreduce.job.output.key.class</name>
                    <value>org.apache.hadoop.io.Text</value>
                </property>
                <property>
                    <name>mapreduce.job.output.value.class</name>
                    <value>org.apache.hadoop.io.IntWritable</value>
                </property>             

                <!--4 output -->    
                <property>
                    <name>mapred.output.dir</name>
                    <value>/user/hadoop/${examplesRoot}/output-data/${outputDir}</value>
                </property> 
            </configuration>
        </map-reduce>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

2.6 上傳文件到hdfs 上面：

hdfs dfs -put map-reduce oozie-apps

2.7 執行oozie 命令運行job 處理

bin/oozie job -oozie http://namenode01.hadoop.com:11000/oozie -config oozie-apps/map-reduce/job.properties -run

2.8 在瀏覽器上面查看測試結果

三：oozie 調度shell 腳本

3.1 生成配置文件：

cd /home/hadoop/yangyang/oozie/examples/apps
cp -ap shell/ ../../oozie-apps/
mv shell mem-shell

3.2 書寫shell 腳本：

cd /home/hadoop/yangyang/oozie/oozie-apps/mem-shell

vim meminfo.sh

#!/bin/bash
/usr/bin/free -m >> /tmp/meminfo

3.3 配置job.properties 文件和workflow.xml

vim job.properties

nameNode=hdfs://namenode01.hadoop.com:8020
jobTracker=namenode01.hadoop.com:8032
queueName=default
examplesRoot=oozie-apps/mem-shell

oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/workflow.xml
EXEC=meminfo.sh

vim workflow.xml

<workflow-app xmlns="uri:oozie:workflow:0.4" name="mem-shell-wf">
    <start to="shell-node"/>
    <action name="shell-node">
        <shell xmlns="uri:oozie:shell-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
            <exec>${EXEC}</exec>
            <file>/user/hadoop/oozie-apps/mem-shell/${EXEC}#${EXEC}</file>
        </shell>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

3.4 上傳配置文件到hdfs 上面

cd /home/hadoop/yangyang/oozie/oozie-apps
hdfs dfs  -put mem-shell oozie-apps

3.5 執行oozie 調度 shell腳本

bin/oozie job -oozie http://namenode01.hadoop.com:11000/oozie -config oozie-apps/mem-shell/job.properties -run

四：oozie 的coordinator 週期性調度當前任務

4.1 配置時區 更改oozie 的配置文件

cd /home/hadoop/yangyang/oozie/conf

vim oozie-site.xml 增長：

    <property>
        <name>oozie.processing.timezone</name>
        <value>GMT+0800</value>
    </property>
        <property>
        <name>oozie.service.coord.check.maximum.frequency</name>
        <value>false</value>
    </property>

4.2 更改本地 時間

使用root 帳戶 配置

 cp -p /etc/localtime /etc/localtime.bak 

 rm -rf /etc/localtime

 cd /usr/share/zoneinfo/Asia/

 cp -p Shanghai /etc/localtime

4.3 更改oozie-consle.js 文件

cd /home/hadoop/yangyang/oozie/oozie-server/webapps/oozie

vim oozie-console.js 

function getTimeZone() {
    Ext.state.Manager.setProvider(new Ext.state.CookieProvider());
    return Ext.state.Manager.get("TimezoneId","GMT+0800");
}

4.4 重新啓動oozie 服務

bin/oozie-stop.sh 
bin/oozie-start.sh

4.5 查看oozie 的當前時間

4.6 配置job.properties 文件和workflow.xml

cd /home/hadoop/yangyang/oozie/examples/apps

cp -ap cron ../../oozie-apps/

cd cron

rm -rf job.properties workflow.xml
cd /home/hadoop/yangyang/oozie/oozie-apps/mem-shell

cp -p * ../cron

配置job.properties

vim job.properties

---
nameNode=hdfs://namenode01.hadoop.com:8020
jobTracker=namenode01.hadoop.com:8032
queueName=default
examplesRoot=oozie-apps/cron

oozie.coord.application.path=${nameNode}/user/hadoop/${examplesRoot}/
start=2016-06-6T16:57+0800
end=2016-06-6T20:00+0800
workflowAppUri=${nameNode}/user/hadoop/${examplesRoot}/
EXEC=meminfo.sh

配置workflow.xml

vim workflow.xml

---

<workflow-app xmlns="uri:oozie:workflow:0.4" name="memcron-shell-wf">
    <start to="shell-node"/>
    <action name="shell-node">
        <shell xmlns="uri:oozie:shell-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
            <exec>${EXEC}</exec>
            <file>/user/hadoop/oozie-apps/cron/${EXEC}#${EXEC}</file>
        </shell>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

配置coordinator.xml

vim coordinator.xml

---

<coordinator-app name="cron-coord" frequency="${coord:minutes(2)}" start="${start}" end="${end}" timezone="GMT+0800"
                 xmlns="uri:oozie:coordinator:0.2">
        <action>
        <workflow>
            <app-path>${workflowAppUri}</app-path>
            <configuration>
                <property>
                    <name>jobTracker</name>
                    <value>${jobTracker}</value>
                </property>
                <property>
                    <name>nameNode</name>
                    <value>${nameNode}</value>
                </property>
                <property>
                    <name>queueName</name>
                    <value>${queueName}</value>
                </property>
                                <property>
                    <name>EXEC</name>
                    <value>${EXEC}</value>
                </property>
            </configuration>
        </workflow>
    </action>
</coordinator-app>

4.7 上傳配置文件到hdfs 上面：

hdfs dfs -put cron oozie-apps

4.8 執行 oozie 命令 運行job

bin/oozie job -oozie http://namenode01.hadoop.com:11000/oozie -config oozie-apps/cron/job.properties -run

4.9 從web瀏覽job的相關問題