linux下多線程壓縮命令pigz

官網地址：
http://www.zlib.net/pigz/

一.命令安裝

yum install zlib-devel  gcc gcc-c++   -y
wget http://www.zlib.net/pigz/pigz-2.4.tar.gz
tar xf pigz-2.4.tar.gz
cd  pigz-2.4
[root@VM_82_178_centos pigz-2.4]# make 
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas   -c -o pigz.o pigz.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas   -c -o yarn.o yarn.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas   -c -o try.o try.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -c zopfli/src/zopfli/deflate.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -c zopfli/src/zopfli/blocksplitter.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -c zopfli/src/zopfli/tree.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -c zopfli/src/zopfli/lz77.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -c zopfli/src/zopfli/cache.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -c zopfli/src/zopfli/hash.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -c zopfli/src/zopfli/util.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -c zopfli/src/zopfli/squeeze.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -c zopfli/src/zopfli/katajainen.c
gcc  -o pigz pigz.o yarn.o try.o deflate.o blocksplitter.o tree.o lz77.o cache.o hash.o util.o squeeze.o katajainen.o -lm -lpthread -lz
ln -f pigz unpigz
[root@VM_82_178_centos pigz-2.4]# echo $?
0

安裝成功

二.參數幫助說明

[root@VM_82_178_centos pigz-2.4]# find / -name "pigz"
/root/pigz-2.4/pigz

[root@VM_82_178_centos pigz-2.4]# /root/pigz-2.4/pigz --help
Usage: pigz [options] [files ...]
  will compress files in place, adding the suffix '.gz'. If no files are
  specified, stdin will be compressed to stdout. pigz does what gzip does,
  but spreads the work over multiple processors and cores when compressing.

Options:
  -0 to -9, -11        Compression level (level 11, zopfli, is much slower)
  --fast, --best       Compression levels 1 and 9 respectively
  -b, --blocksize mmm  Set compression block size to mmmK (default 128K)
  -c, --stdout         Write all processed output to stdout (won't delete)
  -d, --decompress     Decompress the compressed input
  -f, --force          Force overwrite, compress .gz, links, and to terminal
  -F  --first          Do iterations first, before block split for -11
  -h, --help           Display a help screen and quit
  -i, --independent    Compress blocks independently for damage recovery
  -I, --iterations n   Number of iterations for -11 optimization
  -J, --maxsplits n    Maximum number of split blocks for -11
  -k, --keep           Do not delete original file after processing
  -K, --zip            Compress to PKWare zip (.zip) single entry format
  -l, --list           List the contents of the compressed input
  -L, --license        Display the pigz license and quit
  -m, --no-time        Do not store or restore mod time
  -M, --time           Store or restore mod time
  -n, --no-name        Do not store or restore file name or mod time
  -N, --name           Store or restore file name and mod time
  -O  --oneblock       Do not split into smaller blocks for -11
  -p, --processes n    Allow up to n compression threads (default is the
                       number of online processors, or 8 if unknown)
  -q, --quiet          Print no messages, even on error
  -r, --recursive      Process the contents of all subdirectories
  -R, --rsyncable      Input-determined block locations for rsync
  -S, --suffix .sss    Use suffix .sss instead of .gz (for compression)
  -t, --test           Test the integrity of the compressed input
  -v, --verbose        Provide more verbose output
  -V  --version        Show the version of pigz
  -Y  --synchronous    Force output file write to permanent storage
  -z, --zlib           Compress to zlib (.zz) instead of gzip format
  --                   All arguments after "--" are treated as files
[root@VM_82_178_centos pigz-2.4]#

三.實例演示

說明此命令主要是結合tar命令一塊兒使用的

3.1物理服務器一測試

在4核，16G內存的物理服務器上打包測試
tar 結合pigz壓縮使用時間46s：

[root@localhost ~]# time tar cvf - /usr/local/mysql-5.7.24-linux-glibc2.12-x86_64 |/root/pigz-2.4/pigz -p 4 >mysql.tar.gz
tar: 從成員名中刪除開頭的「/」
real    0m46.406s
user    2m50.791s
sys 0m7.436s

tar 壓縮使用時間2分37秒：

[root@localhost ~]# time tar zcf mysql1.tar.gz  /usr/local/mysql-5.7.24-linux-glibc2.12-x86_64 
tar: 從成員名中刪除開頭的「/」

real    2m37.492s
user    2m36.530s
sys 0m7.730s

壓縮出來的結果:

[root@localhost ~]# du -sh mysql1.tar.gz  mysql.tar.gz
617M    mysql1.tar.gz
616M    mysql.tar.gz

3.2物理服務器二測試

在2顆cpu,單顆核心12，內存48G物理機器上測試

[root@localhost pigz-2.4]# cd /data/iplog/
[root@localhost iplog]# du -sh *
3.4G    2018-12-25
3.2G    2018-12-26
3.0G    2018-12-27
2.8G    2018-12-28
2.0G    2018-12-29
2.0G    2018-12-30
445M    2018-12-31

tar命令結合pigz命令指定24核心cpu,進行壓縮工做

[root@localhost iplog]# time tar cf - 2018-12-25  |/opt/pigz-2.4/pigz -p 24 > /tmp/2018-12-25.iplog.tar.gz
real    0m3.753s
user    1m31.944s
sys 0m3.950s

使用的時間不到4秒
壓縮一樣的文件，tar命令結合pigz命令指定6核心cpu,進行壓縮工做

[root@localhost iplog]# time tar cf - 2018-12-25  |/opt/pigz-2.4/pigz -p 6 > /tmp/12-25.iplog.tar.gz

real    0m13.462s
user    1m22.719s
sys 0m3.552s

使用的時間不到14秒

使用單獨tar命令進行壓縮，壓縮的時間爲1分16秒

[root@localhost iplog]# time tar zcf  /tmp/iplog.tar.gz 2018-12-25

real    1m15.811s
user    1m15.124s
sys 0m3.962s

壓縮出來的結果：

[root@localhost iplog]# du -sh /tmp/12-25.iplog.tar.gz /tmp/2018-12-25.iplog.tar.gz /tmp/iplog.tar.gz 
436M    /tmp/12-25.iplog.tar.gz
436M    /tmp/2018-12-25.iplog.tar.gz
435M    /tmp/iplog.tar.gz

pigz命令使用多cpu線程亞縮效率仍是很是的高的

指定壓縮級別的參數:

Options:
  -0 to -9, -11        Compression level (level 11, zopfli, is much slower)

測試：

[root@localhost iplog]# time tar cf - 2018-12-25  |/opt/pigz-2.4/pigz -4 -p 24 > /tmp/01.iplog.tar.gz

real    0m2.504s
user    1m1.557s
sys 0m3.775s
[root@localhost iplog]# du -sh /tmp/01.iplog.tar.gz
477M    /tmp/01.iplog.tar.gz
[root@localhost iplog]# time tar cf - 2018-12-25  |/opt/pigz-2.4/pigz -8 -p 24 > /tmp/02.iplog.tar.gz
real    0m7.569s
user    3m3.478s
sys 0m3.924s
[root@localhost iplog]# du -sh /tmp/02.iplog.tar.gz
415M    /tmp/02.iplog.tar.gz

[root@localhost iplog]# time tar cf - 2018-12-25  |/opt/pigz-2.4/pigz -9 -p 24 > /tmp/03.iplog.tar.gz
real    0m11.060s
user    4m27.083s
sys 0m4.148s

[root@localhost iplog]# du -sh /tmp/03.iplog.tar.gz
411M    03.iplog.tar.gz

下面的壓縮最耗時，並且也很是消耗CPU,同時壓縮比能在tar zcf壓縮後的基礎上提升16%。因此不建議在有負載的生產機器上採用指定壓縮級別的這種方式進行壓縮。

[root@localhost iplog]# time tar cf - 2018-12-25  |/opt/pigz-2.4/pigz -11 -p 24 > /tmp/04.iplog.tar.gz
real    18m52.560s
user    445m34.734s
sys 7m22.034s
[root@localhost iplog]# du -sh /tmp/04.iplog.tar.gz
367M    /tmp/04.iplog.tar.gz

比較壓縮結果：

[root@localhost ~]# du -sh /tmp/*.iplog.tar.gz 
477M    /tmp/01.iplog.tar.gz
415M    /tmp/02.iplog.tar.gz
411M    /tmp/03.iplog.tar.gz
367M    /tmp/04.iplog.tar.gz
436M    /tmp/12-25.iplog.tar.gz
436M    /tmp/2018-12-25.iplog.tar.gz

上述的壓縮文件都採用tar xf解壓

[root@localhost tmp]# tar xf 04.iplog.tar.gz
[root@localhost tmp]# du -sh *
3.4G    2018-12-25

[root@localhost tmp]# time tar cf - 2018-12-25  |/opt/pigz-2.4/pigz -9 -K -p 24 > /tmp/iplog.tgz
[root@localhost tmp]# /opt/pigz-2.4/pigz -d iplog.tgz

解壓出來的文件是tar包，因此在採用tar xf 來解壓

[root@localhost tmp]# /opt/pigz-2.4/pigz -d iplog.tgz
[root@localhost tmp]# ll iplog.tar
-rw-r--r-- 1 root root 3619717120 12月 31 09:33 iplog.tar
[root@localhost tmp]# du -sh iplog.tar
3.4G    iplog.tar
[root@localhost tmp]# tar xf iplog.tar
[root@localhost tmp]# du -sh *
3.4G    2018-12-25
3.4G    iplog.tar

到此簡單演示完畢