HUE中Oozie執行Sqoop

Oozie執行Sqoop，傳入參數（注意，在使用--query時，參數中必定不要帶有空格，不然出錯）
1. 新建一個workflow

 

2. 拖入一個sqoop

 

3. sqoop抽取命令以下（建議先在命令行試下，由於oozie的錯誤提示很委婉，基本看不出錯誤緣由）

sqoop import --connect jdbc:mysql://localhost:3306/spider_new --username root --password 1234qwer --target-dir /user/fengz/brand/spider_data/amac/amac_fund_${date} --delete-target-dir --fields-terminated-by '`' --query select * from amac_fund where update_time between '${start_time}' and '${end_time}' and \$CONDITIONS -m 1

這是在命令行可執行的命令，其中有三個動態參數，${date}、${start_time}、${end_time}，此處作了一個import的導入操做，把指定數據抽取到HDFS路徑下，用了--query寫SQL語句抽取想要的數據，此處須要注意，若是使用--query，必需要帶\$CONDITIONS，不然執行失敗。
4. 填入oozie-sqoop中，須要填入的內容以下

import --connect jdbc:mysql://localhost:3306/spider_new --username root --password 1234qwer --target-dir /user/fengz/brand/spider_data/amac/amac_fund_${date} --delete-target-dir --fields-terminated-by ` --query select * from amac_fund where update_time between '${start_time}' and '${end_time}' and \$CONDITIONS -m 1

注意：指定分隔符時，要把單引號去掉，下邊還有一個大坑
5. 添加參數

 

6. 啓動，填入參數

 

7. 那麼問題來了，報錯是必須的
oozie在解析sqoop語句的時候，是根據空格來分割的，那就噁心了，日期字段中間帶有空格，就像這樣

Setting up log4j2 log4j2 configuration file created at /data1/yarn/nm/usercache/fengz/appcache/application_1552354446915_0759/container_1552354446915_0759_01_000001/sqoop-log4j2.xml Sqoop command arguments : import --connect jdbc:mysql://localhost:3306/spider_new
--username root --password ********
--target-dir /user/fengz/brand/spider_data/amac/amac_fund_20190315 --delete-target-dir --fields-terminated-by ` --query "select
*
from amac_fund where update_time between '2019-03-14
10:57:08' and '2019-03-14
14:08:45' and \$CONDITIONS" -m 1 Fetching child yarn jobs tag id : oozie-3e829587c1f3a050998935d2c2ac3ed4

 

再看下oozie的錯誤提示，呵呵呵....發現一點鳥用都沒有

No child hadoop job is executed. java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:410) at org.apache.oozie.action.hadoop.LauncherAM.access$300(LauncherAM.java:55) at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:223) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685) at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:217) at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685) at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141) Caused by: java.lang.SecurityException: Intercepted System.exit(1) at org.apache.oozie.action.hadoop.security.LauncherSecurityManager.checkExit(LauncherSecurityManager.java:57) at java.lang.Runtime.exit(Runtime.java:107) at java.lang.System.exit(System.java:971) at org.apache.sqoop.Sqoop.main(Sqoop.java:251) at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:214) at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:199) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:101) at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:51) ... 16 more Intercepting System.exit(1) Failing Oozie Launcher, Main Class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

之後用的時候注意下就行了。那這個問題怎麼解決？用shell，shell腳本里邊寫sqoop抽取。