利用eclipse編寫自定義hive udf函數

在作日誌分析的過程當中，用到了hadoop框架中的hive，不過有些日誌處理用hive中的函數處理顯得力不從心，就須要用udf來進行擴展處理了

1  在eclipse中新建java project   hiveudf   而後新建class  package(com.afan)  name(UDFLower)

2  添加jar library  hadoop-0.20.2-core.jar   hive-exec-0.7.0-cdh3u0.jar兩個文件到project

3  編寫代碼

1. package com.afan;  
2. import org.apache.hadoop.hive.ql.exec.UDF;  
3. import org.apache.hadoop.io.Text;  
4.   
5. public class UDFLower extends UDF{  
6.     public Text evaluate(final Text s){  
7.         if (null == s){  
8.             return null;  
9.         }  
10.         return new Text(s.toString().toLowerCase());  
11.     }  
12. }  

4  編譯輸出打包文件爲 udf_hive.jar

5 將udf_hive.jar放入配置好的linux系統的文件夾中路徑爲/home/udf/udf_hive.jar

6 打開hive命令行測試

   hive> add jar /home/udf/udf_hive.jar;

Added udf_hive.jar to class path
Added resource: udf_hive.jar

建立udf函數
hive> create temporary function my_lower as 'com.afan.UDFLower';

建立測試數據
hive> create table dual (info string);

導入數據文件data.txt

data.txt文件內容爲

WHO

AM

I

HELLO

hive> load data local inpath '/home/data/data.txt' into table dual;

hive> select info from dual;

Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201105150525_0003, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201105150525_0003
Kill Command = /usr/local/hadoop/bin/../bin/hadoop job  -Dmapred.job.tracker=localhost:9001 -kill job_201105150525_0003
2011-05-15 06:46:05,459 Stage-1 map = 0%,  reduce = 0%
2011-05-15 06:46:10,905 Stage-1 map = 100%,  reduce = 0%
2011-05-15 06:46:13,963 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201105150525_0003
OK
WHO
AM
I
HELLO

使用udf函數
hive> select my_lower(info) from dual;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201105150525_0002, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201105150525_0002
Kill Command = /usr/local/hadoop/bin/../bin/hadoop job  -Dmapred.job.tracker=localhost:9001 -kill job_201105150525_0002
2011-05-15 06:43:26,100 Stage-1 map = 0%,  reduce = 0%
2011-05-15 06:43:34,364 Stage-1 map = 100%,  reduce = 0%
2011-05-15 06:43:37,484 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201105150525_0002
OK
who
am
i
hello

經測試成功經過

參考文章http://landyer.iteye.com/blog/1070377

——————————————————————————————————

一、編寫函數

1. package com.example.hive.udf;  
2.   
3. import org.apache.hadoop.hive.ql.exec.UDF;  
4. import org.apache.hadoop.io.Text;  
5.   
6. public final class LowerCase extends UDF {  
7.   public Text evaluate(final Text s) {  
8.     if (s == null) { return null; }  
9.     return new Text(s.toString().toLowerCase());  
10.   } 
11. }  

二、用eclipse下的fatjar插件進行打包

先下載net.sf.fjep.fatjar_0.0.31.jar插件包，cp至eclipse/plugins目錄下，重啓eclipse，右擊項目選Export，選擇用fatjar導出(能夠刪掉沒用的包，否則導出的jar包很大)

三、將導出的hiveudf.jar複製到hdfs上

hadoop fs -copyFromLocal hiveudf.jar hiveudf.jar

四、進入hive，添加jar，

add jar hdfs://localhost:9000/user/root/hiveudf.jar

五、建立一個臨時函數

create temporary function my_lower as 'com.example.hive.udf.LowerCase';

六、調用

select LowerCase(name) from teacher;

注：這種方法只能添加臨時的函數，每次從新進入hive的時候都要再執行4-6，要使得這個函數永久生效，要將其註冊到hive的函數列表

添加函數文件$HIVE_HOME/src/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLowerCase.java

修改$HIVE_HOME/src/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java文件

import org.apache.hadoop.hive.ql.udf.UDFLowerCase;

registerUDF(「LowerCase」, UDFLowerCase.class,false);

(上面這個方法未測試成功)

爲了不每次都有add jar 能夠設置hive的'輔助jar路徑'
在hive-env.sh中 export HIVE_AUX_JARS_PATH=/home/ckl/workspace/mudf/mudf_fat.jar;

from：http://blog.chinaunix.net/uid-28194925-id-3453844.html