Linux 如何快速複製文件

http://www.cnblogs.com/paul8339/p/7802450.html

1，在須要對大量小文件進行移動或複製時，用cp、mv都會顯得很沒有效率，能夠用tar先壓縮再解壓縮的方式。 
　　

2，在網絡環境中傳輸時，能夠再結合nc命令，經過管道和tcp端口進行傳輸。 
　nc和tar能夠用來快速的在兩臺機器之間傳輸文件和目錄，比ftp和scp要來得簡單的多。 
　因爲nc是一個超輕量的命令，因此通常busybox都會集成它。當一個linux終端，好比linux pda, 
　經過usblan的方式鏈接到另外一臺linux主機的時候，這樣的嵌入式終端上通常不會集成ftp server, ssh server 
　這樣比較笨重的服務，這個時候， nc可能成爲惟一的上傳手段。 
　　

好比將機器A上的mytest目錄上傳到到機器 B（192.168.0.11）上，只須要： 
　　

在機器B上，用nc來監聽一個端口，隨便就好，只要不被佔用；而且將收到的數據用tar展開。-l表明監聽模式。 

nc -l 6666 |tar -C /target_dir -zxf -

 

而後，在A上經過nc和 tar發送test目錄。使用一致的6666的端口。 

tar -zcvf - mytest |nc 192.168.0.11 6666

Sometimes a simple cp -a command is a very painful and slow process. It's true that -v (verbose) option can give you some information on the details of the copy process, but not normally the progress of it. In fact, cp -a is a quite slow process that sometimes is faster (and safer) implemented by tar, for example:

 

$ tar cf - . | (cd /dst; tar xvf -)

 

Usually faster, and more verbose. Another commands such as pv can help you too, to monitor the progress of a copy between two directories, for example:

 

$ tar cf - . | pv | (cd /dst; tar xf -)
2,06GB 0:00:09 [ 194MB/s] [  <=>                     ]

 

But copying several gigabytes/terabytes of data and many files between quite old NFS disks is painful via cp. Let's see two alternatives for:

•  Monitoring the progress of the copy and the copied files.
•  Skipping to next file before an error (gcp)
•  Syncing directories (rsync)
•  Copying files via network (rsync)

One of the better commands for doing copies is rsync, that allows you to synchronize two directories, and in this sense src/ can have live data, that incrementally is synced to dst/ in several executions of the command

 

$ rsync --info=progress2 -auvz ~/Music/ /data/music/

 

giving a result like this:

 

Jake Bugg - Jake Bugg Album 2012/
Jake Bugg - Jake Bugg Album 2012/01 - Lighting Bolt.mp3
  1,913,897,967  15%   22.79MB/s    0:01:20 (xfr#277, ir-chk=1019/1825)
Jake Bugg - Jake Bugg Album 2012/05 - Simple As This.mp3
  1,936,698,070  15%   22.80MB/s    0:01:21 (xfr#281, ir-chk=1015/1825)

 

You can also use it with -n option to perform a dry run (this is more used than the skype test call), that checks and lists the differences between the two given directories. You can use it too with "-e ssh" user@host:dst/ or without --info option in older versions of rsync. It is slower for copying but it does a lot of useful things such syncing, checkings md5sums.... You will remember rsync if something goes bad.

 

Another fantastic command for copy is gcp. Besides of progress estimation, gcp does not copy when the file exists, skips to the next file if occurs an error, and all the fails are written to a journal file. 

 

$ gcp -rv ~/Music/* /data/music/
Copying 13.53 GiB   2% |#                                  | 165.50 MB/s ETA:  0:01:25

 

Please check journal: /home/cesar/.gcp/journal

 

$ cat /home/cesar/.gcp/journal

/home/cesar/Music/Alabama Shakes-Boys & Girls (2014)/01 - Alabama Shakes - Hold On.mp3
FAILED: already exists
/home/cesar/Music/Alabama Shakes-Boys & Girls (2014)/03 - Alabama Shakes - Hang Loose.mp3
FAILED: already exists

 

In an Alfresco context, many simple migrations (or restoring processes) are tracked via CIFS or Webdav drives. In these cases the above commands are useful. Even they can be useful, if you are doing a local copy in an Alfresco instance, for performing a later Filesystem Bulk process in Alfresco. From a system administrator point of view, when restoring huge contentstores or Lucene / SOLR indices, or moving backups, these commands can save you so much time.

 

Another day we took some time in alternatives for scp copies between two machines.

 

Some useful links for reading and just patience for copying:

• http://www.linuxtecnico.es/2014/04/benchmark-cp-vs-tar-vs-cpio-vs-rsync.html
• http://askubuntu.com/questions/17275/progress-and-speed-with-cp

NOTE: ~/Music and /data/music are simple tests on a local SSD disk.