Hadoop distcp 命令跨集羣複製文件

hadoop提供了Hadoop distcp命令在Hadoop不一樣集羣之間進行數據複製和copy。

使用格式爲：hadoop  distcp   -pbc  hdfs://namenode1/test hdfs://namenode2/test

distcp copy只有Map沒有Reduce

usage: distcp OPTIONS [source_path...] <target_path>

OPTIONS

-append        Reuse existing data in target files and append new

                      data to them if possible

-async           Should distcp execution be blocking

-atomic          Commit all changes or none

-bandwidth <arg> Specify bandwidth per map in MB

-delete          Delete from target, files missing in source

-diff <arg>    Use snapshot diff report to identify the

                    difference between source and target

-f <arg>        List of files that need to be copied

-filelimit <arg> (Deprecated!) Limit number of files copied to <= n

-i Ignore failures during copy

-log <arg> Folder on DFS where distcp execution logs are

saved

-m <arg> Max number of concurrent maps to use for copy

-mapredSslConf <arg> Configuration for ssl config file, to use with

                               hftps://

-overwrite Choose to overwrite target files unconditionally,

even if they exist.

-p <arg> preserve status (rbugpcaxt)(replication,

              block-size, user, group, permission,

             checksum-type, ACL, XATTR, timestamps). If -p is

             specified with no <arg>, then preserves

             replication, block size, user, group, permission,

             checksum type and timestamps. raw.* xattrs are

             preserved when both the source and destination

             paths are in the /.reserved/raw hierarchy (HDFS

             only). raw.* xattrpreservation is independent of

             the -p flag. Refer to the DistCp documentation for

            more details.

-sizelimit <arg> (Deprecated!) Limit number of files copied to <= n

                          bytes

-skipcrccheck Whether to skip CRC checks between source and

                      target paths.

-strategy <arg> Copy strategy to use. Default is dividing work

                         based on file sizes

-tmp <arg> Intermediate work path to be used for atomic

                  commit

-update Update target, copying only missingfiles or

             directories

不一樣版本的Hadoop集羣因爲RPC協議版本不同不能直接使用命令 hadoop distcp hdfs://namenode1/test hdfs://namenode2/test

對於不一樣Hadoop版本間的拷貝，用戶應該使用HftpFileSystem。 這是一個只讀文件系統，因此DistCp必須運行在目標端集羣上（更確切的說是在可以寫入目標集羣的TaskTracker上）。 源的格式是hftp://<dfs.http.address>/<path> （默認狀況dfs.http.address是 <namenode>:50070）。