hive從本地導入數據時出現「Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask」錯誤

現象

經過load data local導入本地文件時報沒法導入的錯誤

hive> load data local inpath '/home/hadoop/out/mid_test.txt' overwrite into table my_mid.mid_test partition (etl_date=20190101);
Loading data to table my_mid.mid_test partition (etl_date=20190101)
Failed with exception Unable to move source file:/home/hadoop/out/mid_test.txt to destination hdfs://namenode01.my.com/my_etl/dw/mid/mid_test/etl_date=20190101/mid_test.txt
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask

相關操做

第一次導入時時成功的，後來發現源文件行數有問題，從新複製了一個文本文件到本地導入目錄，後續再導入時就出現了上述問題。

解決辦法

查詢hive日誌得到更詳細的信息，日誌的常見位置爲/tmp/${user}/hive.log中

2019-02-19 09:38:21,503 ERROR [main]: exec.Task (SessionState.java:printError(960)) - Failed with exception Unable to move source file:/home/hadoop/out/mid_test.txt to destination hdfs://namenode01.my.com/my_etl/dw/mid/mid_test/etl_date=20190101/mid_test.txt
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source file:/home/hadoop/out/mid_test.txt to destination hdfs://namenode01.my.com/my_etl/dw/mid/mid_test/etl_date=20190101/mid_test.txt
	at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2644)
	at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:2911)
	at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1398)
	at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1324)
	at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:438)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.hadoop.fs.ChecksumException: Checksum error: file:/home/hadoop/out/mid_test.txt at 0 exp: -827044509 got: 624370567
	at org.apache.hadoop.fs.FSInputChecker.verifySums(FSInputChecker.java:323)
	at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:279)
	at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
	at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
	at java.io.DataInputStream.read(DataInputStream.java:100)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1965)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1933)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1898)
	at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2637)
	... 23 more
```shell

> 肯定是校驗失敗，找到實際緣由是舊文件crc文件與新拷貝文件的校驗值不一致。

## 檢查是否有crc文件

```shell
[hadoop@my-17 out]$ls -al
總用量 112720
drwxrwxr-x   2 hadoop hadoop     4096 2月  19 09:31 .
drwx------. 39 hadoop hadoop     4096 2月  19 09:57 ..
-rw-r--r--   1 hadoop hadoop  1595242 2月  19 09:12 mid_test.txt
-rw-r--r--   1 hadoop hadoop     3128 2月  19 08:22 .mid_test.txt.crc

確實有此文件，將crc文件刪除便可。

後續導入即成功。