修改hive 默認fs爲s3 遇到的坑

問題： hive我修改了 默認的f <property> <name>fs.defaultFS</name> <value>hdfs://。。。:8020</value> </property> 修改成<property> <name>fs.defaultFS</name> <value>s3://。。。</value> </property> 修改後執行窗口函數的時候會執行失敗,若是不執行窗口函數的沒問題。

失敗hive命令簡化版：select * ,row_number() over (partition by id order by utime desc) as sort_num from mobi_warehouse.table ； 成功命令簡化版：select * from mobi_warehouse.table；

錯誤：FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask

查看hive運行日誌，發現報錯以下：

 INFO [main([])]: session.SessionState (SessionState.java:close(1573)) - Error closing tez session java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: Incomplete HDFS URI, no host: hdfs:///apps/tez/tez.tar.gz at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.isOpen(TezSessionState.java:163) ~[hive-exec-2.3.2-amzn-2.jar:2.3.2-amzn-2] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.toString(TezSessionState.java:125) ~[hive-exec-2.3.2-amzn-2.jar:2.3.2-amzn-2] .....

這個錯誤指明沒法找到 tez 執行引擎的 library 檔案，這個路徑 hdfs:///apps/tez/tez.tar.gz 表示它存儲於 HDFS 上。 因爲您已修改 fs.defaultFS 至 S3 桶，所以可行的解決方法是將 tez.tar.gz 也複製到您的 S3 桶中。 如下是個人操做步驟。

1. 將 tez.tar.gz 從 HDFS 上覆制到 master node 的本地磁盤 $ hdfs dfs -copyToLocal /apps/tez/tez.tar.gz ./

2. 將 tez.tar.gz 複製到 S3 桶中 $ aws s3 cp ./tez.tar.gz s3://S3桶名稱/

3. 修改 /etc/tez/conf/tez-site.xml, 找到 tez.lib.uris property, 並修改其值以下

<property>

<name>tez.lib.uris</name>

<value>s3://S3桶名稱/tez.tar.gz</value>

</property>

修改後執行，發現仍然報以下錯誤：

Caused by: java.util.concurrent.ExecutionException: java.io.IOException: The ownership on the staging directory s3://mobimagic-microl-risk/tmp/hive/hadoop/_tez_session _dir/5db8dd84-5154-460d-9a6d-f875adb6fd6b is not as expected. It is owned by . The directory must be owned by the submitter hadoop or by hadoop at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_121] at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[?:1.8.0_121] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.isOpen(TezSessionState.java:168) ~[hive-exec-2.1.1-amzn-0.jar:2.1.1-amzn-0] ... 14 more Caused by: java.io.IOException: The ownership on the staging directory s3://mobimagic-microl-risk/tmp/hive/hadoop/_tez_session_dir/5db8dd84-5154-460d-9a6d-f875adb6fd6b is not as expected. It is owned by . The directory must be owned by the submitter hadoop or by hadoop

最終發現，當前hadoop版本是亞馬遜的2.7.2，升級到2.8.3就根據如上配置就沒問題了，把emr版本提升到5.12以上就能夠了

若是幫到你了，請點下贊

轉載註明出處