是一款開源的 多功能 快速的全量和增量備份軟件算法
a 等價於本地複製命令(cp)vim
b 等價於遠程複製命令(scp)服務器
c 等價於數據刪除命令(rm)網絡
d 等價於數據查看命令(ls)ssh
1. 能夠實現排除同步數據socket
2. 能夠實現加密傳輸數據(藉助ssh協議隧道)tcp
3. 能夠實現數據傳輸認證機制ide
4. 能夠實現socket守護進程方式,傳輸數據(重點)測試
5. 能夠保持數據傳輸屬性信息不變(-p)網站
6. 能夠傳輸普通文件,也能夠傳輸特殊文件(連接文件 或者 設備文件)
7. 能夠數據增量傳輸
基於一種特殊的算法「quick check」,實現增量複製傳輸數據
1. 先實現兩臺主機之間網絡互通
2. 對要傳輸的數據信息,進行屬性信息比對(好比 文件大小 時間信息 屬主 屬組 權限屬性等)
3. 只傳輸變化的數據信息
1. 實現本地數據傳輸備份
2. 實現遠程數據傳輸備份
3. 實現守護進程方式數據傳輸備份(重點)
第一步:檢查軟件是否安裝
第二步:編寫rsync軟件配置文件
第三步:建立備份目錄的管理用戶
第四步:建立認證時所須要密碼文件(對密碼文件權限進行修改成600權限)
第五步:建立數據備份目錄,修改備份目錄屬主和屬組權限爲管理用戶(rsync)
第六步:啓動rsync服務(rsync --daemon)
進行數據傳輸測試
說明:若是想實現免交互方式傳輸數據,客戶端要完成如下操做步驟
第一步:建立密碼文件,並進行修改權限爲600
echo "wuhuang123" >>/etc/rsync.password
chmod 600 /etc/rsync.password
[root@nfs01 ~]# echo "wuhuang123" >>/etc/rsync.password [root@nfs01 ~]# chmod 600 /etc/rsync.password [root@nfs01 ~]# ll /etc/rsync.password -rw------- 1 root root 10 Jan 23 14:45 /etc/rsync.password |
第二步:進行免交互傳輸數據測試
rsync -avz /etc/hosts rsync_backup@backup::backup --password-file=/etc/rsync.password
[root@nfs01 ~]# rsync -avz /etc/hosts rsync_backup@backup::backup --password-file=/etc/rsync.password sending incremental file list hosts
sent 201 bytes received 27 bytes 456.00 bytes/sec total size is 372 speedup is 1.63 [root@backup ~]# ll /backup/ total 4 -rw-r--r-- 1 rsync rsync 372 Jan 19 11:55 hosts |
1.7 rsync守護進程方式傳輸數據原理過程
在備份服務器上部署rsync軟件服務,採用推的方式(push:即從本地機器拷貝文件到遠程rsync服務器中)進行數據備份傳輸。
採用定時備份:備份的數據信息大部分是內部人員產生的數據
採用實時備份:備份的數據信息大部分是外部人員產生的數據
命令參數 |
參數 |
-v, --verbose |
詳細模式輸出,傳輸時的信息。 |
-z,--compress |
傳輸時進行壓縮提供傳輸效率,--compress-level=NUM 可按級別壓縮,局域網能夠不用壓縮 |
-a, --archive 重要 |
歸檔模式,表示以遞歸方式傳輸文件,並保持全部文件屬性,等於-rlptgoD。 |
-r, --recursive 歸類於-a參數 |
對子目錄以遞歸模式處理,即目錄下的全部目錄都一樣傳輸,注意是小寫 |
-t, --times 歸類於-a參數 |
保持文件時間信息。 |
-o, --owner 歸類於-a參數 |
保持文件屬主信息。 |
-p, --perms 歸類於-a參數 |
保持文件權限。 |
-g, --group 歸類於-a參數 |
保持文件屬組信息。 |
-P --progress |
等同於 --partial,顯示備份過程。 |
-D, --devices 歸類於-a參數 |
保持設備文件信息。 |
-l, --links 歸類於-a參數 |
保留軟鏈結。(小寫字母) |
-e, --rsh=command |
指定使用rsh、ssh方式進行數據同步 |
--exclude=PATTETN |
用來指定排除那些不但願傳輸的文件(和tar參數同樣) |
--exclude-from=FILE |
文件名所在的目錄文件,便可以實現排除多個文件(和tar參數同樣) |
--bwlimit=RATE |
limit I/O bandwidth: KBytes per second limit socket I/O bandwidth限速功能 案例:某DBA作數據同步,帶寬佔滿,致使用戶沒法訪問網站 |
--delete |
讓目標目錄SRC和源目錄數據DST一致,即無差別同步數據 |
保持同步目錄及文件屬性: 這裏的-avzP 至關於 -vzrtopgDIP(還多了DI功能),生產環境經常使用的參數選項爲 -avzP 或 -vzrtopgP 若是是放入腳本中,也能夠把 -v 和 -P去掉。這裏的 --progress 能夠用 -P代替。 |
例:-e, --rsh=command 指定使用rsh、ssh方式進行數據同步
[root@nfs01 ~]# rsync -avz -e "ssh -p22" /wuhuang/a 172.16.1.41:/backup The authenticity of host '172.16.1.41 (172.16.1.41)' can't be established. RSA key fingerprint is 57:3f:64:68:95:4d:99:54:01:33:ab:47:a0:72:da:bf. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '172.16.1.41' (RSA) to the list of known hosts. root@172.16.1.41's password: sending incremental file list a/ a/1 a/2 a/3
sent 171 bytes received 73 bytes 28.71 bytes/sec total size is 0 speedup is 0.00 [root@backup ~]# ll /backup/ total 8 drwxr-xr-x 2 root root 4096 Jan 23 17:54 a |
第4章 rsync服務常見錯誤
【客戶端的錯誤】 No route to host 【錯誤演示過程】 [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup rsync: failed to connect to 172.16.1.41: No route to host (113) rsync error: error in socket IO (code 10) at clientserver.c(124) [sender=3.0.6] 【異常問題解決】 關閉rsync服務端的防火牆服務(iptables) 也可作防火牆設置,此處很少講 [root@backup mnt]# /etc/init.d/iptables stop iptables: Setting chains to policy ACCEPT: filter [ OK ] iptables: Flushing firewall rules: [ OK ] iptables: Unloading modules: [ OK ] [root@backup mnt]# /etc/init.d/iptables status iptables: Firewall is not running. |
客戶端的錯誤現象: [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::/backup ERROR: The remote path must start with a module name not a / rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] 異常問題解決: rsync命令語法理解錯誤,::/backup是錯誤的語法,應該爲::backup(rsync模塊) |
客戶端的錯誤現象: [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup Password: @ERROR: auth failed on module backup rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] 異常問題解決: 1. 密碼真的輸入錯誤,用戶名真的錯誤 2. secrets file = /etc/rsync.password指定的密碼文件和實際密碼文件名稱不一致 3. /etc/rsync.password文件權限不是600 4. rsync_backup:123456密碼配置文件後面注意不要有空格 5. rsync客戶端密碼文件中只輸入密碼信息便可,不要輸入虛擬認證用戶名稱 |
[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup @ERROR: Unknown module 'backup' rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] 異常問題解決: 1. /etc/rsyncd.conf配置文件模塊名稱書寫錯誤 2. 模塊對應的目錄建立有誤 |
[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup Password: sending incremental file list hosts rsync: mkstemp ".hosts.5z3AOA" (in backup) failed: Permission denied (13)
sent 196 bytes received 27 bytes 63.71 bytes/sec total size is 349 speedup is 1.57 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6] 異常問題解決: 1. 共享目錄的屬主和屬組不正確,不是rsync 2. 共享目錄的權限不正確,不是755 |
[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup Password: @ERROR: chdir failed rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] 異常問題解決: 1. 備份存儲目錄沒有創建 2. 創建的備份存儲目錄和配置文件定義不一致 說明:若是沒有備份存儲目錄 |
[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup Password: @ERROR: invalid uid rsync rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] 異常問題解決: rsync服務對應rsync虛擬用戶不存在了 |
password file must not be other-accessible
[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password password file must not be other-accessible continuing without password file Password: sending incremental file list
sent 26 bytes received 8 bytes 5.23 bytes/sec total size is 349 speedup is 10.26 異常問題解決: rsync客戶端的祕鑰文件也必須是600權限 |
4.9 rsync客戶端鏈接慢問題
錯誤日誌輸出 2017/03/08 20:14:43 [3422] params.c:Parameter() - Ignoring badly formed line in configuration file: ignore errors 2017/03/08 20:14:43 [3422] name lookup failed for 172.16.1.31: Name or service not known 2017/03/08 20:14:43 [3422] connect from UNKNOWN (172.16.1.31) 2017/03/08 20:14:43 [3422] rsync to backup/ from rsync_backup@unknown (172.16.1.31) 2017/03/08 20:14:43 [3422] receiving file list 2017/03/08 20:14:43 [3422] sent 76 bytes received 83 bytes total size 349 正確日誌輸出 2017/03/08 20:16:45 [3443] params.c:Parameter() - Ignoring badly formed line in configuration file: ignore errors 2017/03/08 20:16:45 [3443] connect from nfs02 (172.16.1.31) 2017/03/08 20:16:45 [3443] rsync to backup/ from rsync_backup@nfs02 (172.16.1.31) 2017/03/08 20:16:45 [3443] receiving file list 2017/03/08 20:16:45 [3443] sent 76 bytes received 83 bytes total size 349 客戶端鏈接慢緣由:主機名未被解析 異常問題解決: 查看日誌進行分析,配置/etc/hosts文件,寫入對應主機名與IP的映射關係 |
4.10 rsync服務沒有正確啓動
Connection refused (111)
[root@wuhuang-muban ~]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup rsync: failed to connect to 172.16.1.41: Connection refused (111) rsync error: error in socket IO (code 10) at clientserver.c(124) [sender=3.0.6] 解決 rsync服務沒開啓 [root@wuhuang-muban ~]# rsync --daemon [root@wuhuang-muban ~]# ss -lntup |grep rsync tcp LISTEN 0 5 :::873 :::* users:(("rsync",1434,5)) tcp LISTEN 0 5 *:873 *:* users:(("rsync",1434,4)) [root@wuhuang-muban ~]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup Password: sending incremental file list hosts
sent 196 bytes received 27 bytes 49.56 bytes/sec total size is 349 speedup is 1.57 |
第5章 rsync擴展應用說明
1. 讓rsync守護進程開機自啓動
echo "rsync --daemon" >>/etc/rc.local
利用xinetd服務啓動rsync守護進程服務流程
rpm -qa|grep xinetd
yum install -y xinetd
[root@backup ~]# rpm -qa|grep xinetd [root@backup ~]# yum install xinetd -y |
5.1.2 第二個里程碑:配置xinetd文件,讓rsync服務容許被超級守護進程管理
vim /etc/xinetd.d/rsync
disable = yes ===> disable = no
[root@backup ~]# vim /etc/xinetd.d/rsync # default: off # description: The rsync server is a good addition to an ftp server, as it \ # allows crc checksumming etc. service rsync { disable =no flags = IPv6 socket_type = stream wait = no user = root server = /usr/bin/rsync server_args = --daemon log_on_failure += USERID } |
5.1.3 第三個里程碑:確保rsync守護進程服務關閉了
[root@backup ~]# netstat -lntup|grep rsync tcp 0 0 0.0.0.0:873 0.0.0.0:* LISTEN 1294/rsync tcp 0 0 :::873 :::* LISTEN 1294/rsync [root@backup ~]# killall rsync [root@backup ~]# killall rsync rsync: no process killed |
5.1.4 第四個里程碑:啓動xinetd服務
[root@backup ~]# /etc/init.d/xinetd start Starting xinetd: [ OK ] [root@backup ~]# netstat -lntup|grep 873 tcp 0 0 :::873 :::* LISTEN 1353/xinetd |
5.2 守護進程多模塊功能配置
[root@backup ~]# cat /etc/rsyncd.conf #rsync_config #created by HQ at 2017 ##rsyncd.conf start##
uid = rsync gid = rsync use chroot = no max connections = 200 timeout = 300 pid file = /var/run/rsyncd.pid lock file = /var/run/rsync.lock log file = /var/log/rsyncd.log ignore errors read only = false list = false hosts allow = 172.16.1.0/24
hosts deny = 0.0.0.0/32 auth users = rsync_backup secrets file = /etc/rsync.password [backup] comment = "backup dir by wuhuang" path = /backup [devbackup] comment = "backup dir by wuhuang" path = /devbackup [sabackup] comment = "backup dir by wuhuang" path = /sabackup |
[root@backup ~]# mkdir /{backup,devbackup,sabackup}/ -p [root@backup ~]# chown -R rsync.rsync /{backup,devbackup,sabackup}/ [root@backup ~]# ll /backup/ -d drwxr-xr-x 3 rsync rsync 4096 Jan 23 21:08 /backup/ [root@backup ~]# ll /devbackup/ -d drwxr-xr-x 2 rsync rsync 4096 Jan 23 23:17 /devbackup/ [root@backup ~]# ll /sabackup/ -d drwxr-xr-x 2 rsync rsync 4096 Jan 23 23:17 /sabackup/ |
[root@backup ~]# /etc/init.d/xinetd restart Stopping xinetd: [ OK ] Starting xinetd: [ OK ] |
1. 若是是由rsync --daemon啓動的,重啓時使用 killall rsync && rsync --daemon
2. 若是是由 /etc/init.d/xinetd start啓動的,重啓時使用 /etc/init.d/xinetd restart
[root@nfs01 ~]# rsync --delete -avz /etc/hosts rsync_backup@172.16.1.41::devbackup --password-file=/etc/rsync.passwordsending incremental file list hosts
sent 205 bytes received 27 bytes 464.00 bytes/sec total size is 372 speedup is 1.60 [root@nfs01 ~]# rsync --delete -avz /etc/hosts rsync_backup@172.16.1.41::sabackup --password-file=/etc/rsync.password sending incremental file list hosts
sent 205 bytes received 27 bytes 464.00 bytes/sec total size is 372 speedup is 1.60 [root@nfs01 ~]# rsync --delete -avz /etc/hosts rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password sending incremental file list hosts
sent 205 bytes received 27 bytes 464.00 bytes/sec total size is 372 speedup is 1.60 [root@backup ~]# ll /backup/ /devbackup/ /sabackup/ /backup/: total 4 -rw-r--r-- 1 rsync rsync 372 Jan 19 11:55 hosts /devbackup/: total 4 -rw-r--r-- 1 rsync rsync 372 Jan 19 11:55 hosts /sabackup/: total 4 -rw-r--r-- 1 rsync rsync 372 Jan 19 11:55 hosts |
建立一個測試環境:nfs服務器(客戶端)上操做
[root@nfs01 ~]# tree /wuhuang/ /wuhuang/ ├── a │ ├── 1 │ ├── 2 │ └── 3 ├── b │ ├── 1 │ ├── 2 │ └── 3 ├── c │ ├── 1 │ ├── 2 │ └── 3 └── d ├── 1 ├── 2 └── 3 4 directories, 12 files
|
實現數據同步傳輸排除方法
1. --exclude 參數指定排除的文件或目錄信息
2. --exclude-from 參數指定排除的多個數據信息文件
--exclude實現排除需求:不要a和b,只要c目錄全部數據,d目錄中只要1和3文件
[root@nfs01 ~]# rsync -avz /wuhuang/ --exclude=a --exclude=b --exclude=d/2 rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password sending incremental file list ./ c/ c/1 c/2 c/3 d/ d/1 d/3
sent 292 bytes received 114 bytes 270.67 bytes/sec total size is 0 speedup is 0.00 [root@backup ~]# ll -R /backup/ /backup/: total 8 drwxr-xr-x 2 rsync rsync 4096 Jan 23 17:54 c drwxr-xr-x 2 rsync rsync 4096 Jan 23 17:54 d
/backup/c: total 0 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 1 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 2 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 3
/backup/d: total 0 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 1 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 3 |
以上命令精簡化:
rsync -avz /wuhuang/ --exclude={a..b} --exclude=d/2 rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password 或 rsync -avz /wuhuang/ --exclude={a,b} --exclude=d/2 rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password 5.3.2 --exclude-from=FILE |
--exclude-from實現排除需求:不要a和b 只要c目錄全部數據 d目錄中只要1和3文件
[root@nfs01 wuhuang]# pwd /wuhuang [root@nfs01 wuhuang]# vim exclude.txt [root@nfs01 wuhuang]# cat exclude.txt a b d/2 [root@nfs01 wuhuang]# rsync -avz /wuhuang/ --exclude-from=/wuhuang/exclude.txt rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password sending incremental file list ./ exclude.txt c/ c/1 c/2 c/3 d/ d/1 d/3
sent 368 bytes received 133 bytes 1002.00 bytes/sec total size is 8 speedup is 0.02 [root@backup ~]# ll -R /backup/ /backup/: total 12 drwxr-xr-x 2 rsync rsync 4096 Jan 23 17:54 c drwxr-xr-x 2 rsync rsync 4096 Jan 23 17:54 d -rw-r--r-- 1 rsync rsync 8 Jan 23 23:54 exclude.txt
/backup/c: total 0 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 1 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 2 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 3 /backup/d: total 0 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 1 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 3 |
說明:若是在排除同步過程當中,出現了指定的exclude.txt文件,兩種方法解決:
01. 將exclude.txt文件從同步數據目錄中移出
02. 在exclude.txt文件中寫入排除自身文件信息
[root@backup ~]# ll /backup/ total 0 [root@nfs01 ~]# rsync -av /wuhuang/a rsync_backup@172.16.1.41::backup/01/ --password-file=/etc/rsync.password sending incremental file list created directory 01 a/ a/1 a/2 a/3
sent 176 bytes received 69 bytes 163.33 bytes/sec total size is 0 speedup is 0.00 [root@nfs01 ~]# rsync -avz /wuhuang/b rsync_backup@172.16.1.41::backup/02/ --password-file=/etc/rsync.password sending incremental file list created directory 02 b/ b/1 b/2 b/3
sent 167 bytes received 69 bytes 472.00 bytes/sec total size is 0 speedup is 0.00 [root@backup ~]# tree /backup/ /backup/ ├── 01 │ └── a │ ├── 1 │ ├── 2 │ └── 3 └── 02 └── b ├── 1 ├── 2 └── 3
4 directories, 6 files
|
[root@nfs01 ~]# rsync -avz /wuhuang/c rsync_backup@172.16.1.41::backup/03/w/h --password-file=/etc/rsync.password sending incremental file list rsync: mkdir "03/w/h" (in backup) failed: No such file or directory (2) rsync error: error in file IO (code 11) at main.c(576) [receiver=3.0.6] rsync: connection unexpectedly closed (5 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6] |
[root@backup ~]# cat /etc/rsyncd.conf #rsync_config #created by HQ at 2017 ##rsyncd.conf start##
uid = rsync gid = rsync use chroot = no max connections = 200 timeout = 300 pid file = /var/run/rsyncd.pid lock file = /var/run/rsync.lock log file = /var/log/rsyncd.log ignore errors read only = false list = false hosts allow = 172.16.1.0/24 hosts deny = 0.0.0.0/32 auth users = rsync_backup secrets file = /etc/rsync.password [backup] comment = "backup dir by wuhuang" path = /backup [devbackup] comment = "backup dir by wuhuang" path = /devbackup [sabackup] comment = "backup dir by wuhuang" path = /sabackup |
第三種狀況測試
[root@nfs01 ~]# rsync --delete -avz /etc/hosts rsync_backup@10.0.0.41::backup --password-file=/etc/rsync.password sending incremental file list hosts
sent 205 bytes received 27 bytes 154.67 bytes/sec total size is 372 speedup is 1.60 10.0.0.41網段能夠實現傳輸 |
5.4 守護進程無差別同步配置
無差別同步通俗的講:
1. 我有的數據,你也有;我沒有的數據,你也不能有(刪除)
2. 存儲與備份服務數據徹底一致(如出一轍)
我有的數據,你也有:實踐過程
rsync客戶端操做命令: [root@nfs01 ~]# rsync -avz --delete /wuhuang/ rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password sending incremental file list ./ exclude.txt a/ a/1 a/2 a/3 b/ b/1 b/2 b/3 c/ c/1 c/2 c/3 d/ d/1 d/2 d/3
sent 705 bytes received 274 bytes 1958.00 bytes/sec total size is 8 speedup is 0.01 rsync服務端檢查: [root@backup ~]# tree /backup/ /backup/ ├── a │ ├── 1 │ ├── 2 │ └── 3 ├── b │ ├── 1 │ ├── 2 │ └── 3 ├── c │ ├── 1 │ ├── 2 │ └── 3 ├── d │ ├── 1 │ ├── 2 │ └── 3 └── exclude.txt
4 directories, 13 files |
我沒有的數據,你也不能有(刪除):實踐過程
[root@nfs01 ~]# cd /wuhuang/ [root@nfs01 wuhuang]# rm -rf a [root@nfs01 wuhuang]# ll total 16 drwxr-xr-x 2 root root 4096 Jan 23 17:54 b drwxr-xr-x 2 root root 4096 Jan 23 17:54 c drwxr-xr-x 2 root root 4096 Jan 23 17:54 d -rw-r--r-- 1 root root 8 Jan 23 23:54 exclude.txt [root@nfs01 wuhuang]# rsync -avz --delete /wuhuang/ rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password sending incremental file list ./ deleting a/3 deleting a/2 deleting a/1 deleting a/
sent 177 bytes received 14 bytes 382.00 bytes/sec total size is 8 speedup is 0.04 rsync服務端檢查: [root@backup ~]# tree /backup/ /backup/ ├── b │ ├── 1 │ ├── 2 │ └── 3 ├── c │ ├── 1 │ ├── 2 │ └── 3 ├── d │ ├── 1 │ ├── 2 │ └── 3 └── exclude.txt
3 directories, 10 files |