Rsync and Inotify

Rsync

1.  rsync intruction

1.1 DESCRIPTION

rsync is a fast, versatile, remote (and local) file-copy tools.

 

Rsync is a fast and extraordinarily versatile file copying tool.  It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon.  It offers a large number  of  options  that  control  every aspect  of  its  behavior and permit very flexible specification of the set of files to be copied.  It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network  by  sending  only  the differences  between the source files and the existing files in the destination.  Rsync is widely used for back-ups and mirroring and as an improved copy command for everyday use.

 

       Rsync finds files that need to be transferred using a quick check algorithm (by default) that looks for files that have changed in size or in last-modified time.  Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the files data does

       not need to be updated.

 

       Some of the additional features of rsync are:

 

       o      support for copying links, devices, owners, groups, and permissions

 

       o      exclude and exclude-from options similar to GNU tar

 

       o      a CVS exclude mode for ignoring the same files that CVS would ignore

 

       o      can use any transparent remote shell, including ssh or rsh

 

       o      does not require super-user privileges

 

       o      pipelining of file transfers to minimize latency costs

 

       o      support for anonymous or authenticated rsync daemons (ideal for mirroring)

 

1.2  SYNOPSIS

Local:  rsync [OPTION...] SRC... [DEST]

 

       Access via remote shell:

         Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]

         Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

 

       Access via rsync daemon:

         Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST]

               rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]

         Push: rsync [OPTION...] SRC... [USER@]HOST::DEST

               rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST

 

       Usages with just one SRC arg and no DEST arg will list the source files instead of copying.

 

In a word, rsync is tools that can move data from one driver to another driver

 

2 setting and using

2.1 check and use it in local

use this command 「rpm -qa rsync」 then you will see

rsync-3.0.6-9.el6_4.1.x86_64

rsync is default install when preparing Linux system.

so we use it to copy file in local

The common usage is:

rsync [OPTION...] SRC... [DEST]

examples:

rsync /etc/hosts /tmp/

if you want copy directory, you can use 「-r」 options

rsync -r /tmp /root/

 

this usage is same as the 「copy」

 

2.2 using based on remote shell 

We also can use it by SSH pipelining, so first you should set SSH pipelining

The common usage is:

Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]

       examples:

       rsync -r -e 'ssh -p 52113' ygh_ssh@192.168.98.132:~ /backup/

       -e --rsh=COMMAND specify the remote shell to use

Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

       examples:

rsync -r -e 'ssh -p 52113' /backup/demo.txt  ygh_ssh@192.168.98.132:~

 

 

2.3 using based on a rsync daemon(this is a very important way)

2.3.1 preparation works

Before use this usage, we need know some important options

-v --verbose increase verbosity

-z --compress     compress file date during the transfer

-a --archive  archive mode;equals -rlptgoD (no -H,-A,-X)

 

-r --recursive recurse into directories

-t --times preserve modification times

-o --owner preserve owner(super user only)

-p --perms preserve permissions

-P --progress show progress during transfer

-D --devices special files

-l --links copy symlinks as symlinks

-a is contain above options

 

-e --rsh=COMMAND specify the remote shell to use

--exclude=PATTERN       exclude files matching PATTERN

--exclude-from=FILE     read exclude patterns from FILE

--bwlimit=KBPS          limit I/O bandwidth; KBytes per second

--delete                delete extraneous files from dest dirs

 

 

the options 「avz」 is usually used

If you want more, you can reference this command 「man rsync」

 

Now new need two machines, one as the client, another is as service to run rsync

daemon

 

The details information is following:

 

Hostname	Network card	Gateway	Functions
NFS_SERVICE	192.168.98.132	255.255.255.0	Run rsync daemon
NFS_CLIENT_1	192.168.98.135	255.255.255.0	Pull or push data
NFS_CLIENT_2	192.168.98.134	255.255.255.0	Pull or push data

 

2.3.2 Setting daemon in service

Firstly, create 「/etc/rsyncd.conf」 file, the file is not existing default, you need to create it..

use this command 「vim /etc/rsyncd.conf」,then writing following content

 

#rsync configuration

#user and group

uid = rsync

gid = rsync

use chroot = no

max connections = 200

timeout = 300

#the file put pid of rsync

pid file = /var/run/rsyncd.pid

#the file put lock information of rsync

lock file = /var/run/rsync.lock

#the logs file

log file = /var/log/rsyncd.log

ignore errors

read only = false

list = false

hosts allow = 255.255.255.0/24

hosts deny = 0.0.0.0/32

#the vitual user

auth users = rsync_backup

##the password is put in this

secrets file = /etc/rsync.password

#exclude file

#exclude= a b

#this is mode

[oldboy]

path = /oldboy/

 

you can get more information by 「man rsyncd.conf」

 

Secondly, create password file 「/etc/rsync.password」 then write following content,

then set it’s mod is 600,othwise rsync will give exception.

#username:password

rsync_backup:oldboy

 

Thirdly, run rsync daemon

use this command 「[root@nfs_service ~]# rsync --daemon」 to run rsync daemon

then use following command to view rsync daemon status

 [root@nfs_service ~]# ps -ef|grep rsync

root      7700     1  0 15:43 ?        00:00:00 rsync --daemon

root      7702  1472  0 15:43 pts/0    00:00:00 egrep --color=auto rsync

 

Above status indicate you rsync daemon is successfully running

now you can use COMMAND in client to transfer data

 

2.3.3 Test rsync daemon in client

In client, we need to set a password file for rsync COMMAND

echo 「oldboy」 >>/etc/rsync.password

the 「oldboy」 is same as the password in service

The USE is 「/etc/rsyncd.conf」 auth users = rsync_backup and the shared directory is the path = /oldboy/  by oldboy to build connection from auth_users to shared directory

 

The common usage is following:

Access via rsync daemon:

Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST]

       example:

                     rsync -avz rsync_backup@192.168.98.132::oldboy /rysnc --password-file=/etc/rsync.password

              這裏的oldboy不是目錄，是服務器rsync配置文件的一個模塊，經過模塊創建共享目錄和虛擬用戶直接的聯繫

rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]

       examples:

              rsync -avz rsync://rsync_backup@192.168.98.132/oldboy /rsync  --password-file=/etc/rsync.password

 

Push: rsync [OPTION...] SRC... [USER@]HOST::DEST

       examples:

              rsync -avz /tmp rsync_backup@192.168.98.132::oldboy --password-file=/etc/rsync.password

 

rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST

examples:

              rsync -avz /etc/ rsync://rsync_backup@192.168.98.132/oldboy --password-file=/etc/rsync.password

 

2.3.4 some other method

exclude copy：exclude some file you want not to copy

--exclude=PATTERN       exclude files matching PATTERN

--exclude-from=FILE     read exclude patterns from FILE

Examples:

       one file exclude:

              rsync -avz --exclude=a /rsync/ rsync_backup@192.168.98.132::oldboy --password-file=/etc/rsync.password

 

       more files exclude:

              rsync -avz --exclude={a,b} /rsync/ rsync_backup@192.168.98.132::oldboy --password-file=/etc/rsync.password

 

       more files exclude sortly:

              rsync -avz --exclude={a..g} /rsync/ rsync_backup@192.168.98.132::oldboy --password-file=/etc/rsync.password

              rsync -avz --exclude={1..10} /rsync/ rsync_backup@192.168.98.132::oldboy --password-file=/etc/rsync.password

 

       more files names are put in exclude.log

              seq 10 > exclude.log

              cat exclude.log

              rsync -avz --exclude-from=/rsync/exclude.log /rsync/ rsync_backup@192.168.98.132::oldboy --password-file=/etc/

 

the same copy keep client and service is same, the file is more in service will be delete：

       rsync -avz --delete --exclude=a /rsync/ rsync_backup@192.168.98.132::oldboy --password-file=/etc/rsync.password

 

2.4  set more shared directory

If you want to create more shared directory, just set in 「/etc/rsyncd.conf」

just like:

 

#rsync configuration

#user and group

uid = rsync

gid = rsync

use chroot = no

max connections = 200

timeout = 300

#the file put pid of rsync

pid file = /var/run/rsyncd.pid

#the file put lock information of rsync

lock file = /var/run/rsync.lock

#the logs file

log file = /var/log/rsyncd.log

ignore errors

read only = false

list = false

hosts allow = 255.255.255.0/24

hosts deny = 0.0.0.0/32

#the vitual user

auth users = rsync_backup

##the password is put in this

secrets file = /etc/rsync.password

#exclude file

#exclude= a b

[oldboy]

path = /oldboy/

[ygh]

path = /ygh/

#self setting

read only = true

[backup]

path = /backup/

 

If the set information is public, you can set it at top

if you need special setting at your mode, you can setting in self mode

 

 

3 Example

某公司有一臺Web服務器，裏面數據很重要，可是若是硬盤壞了，數據就會丟失，

如今領導要求你對數據作一下備份這樣的Web服務器數據丟失就能夠恢復，要求以下：

天天晚上00:00在web服務A上打包備份系統配置文件，網站程序目錄以及訪問日誌並經過rsync命令推送到B

服務器上進行備份（備份的思路是:如今本地按日期打包，而後在推到B服務器上）

 

具體要求以下：

       1）web服務器A和備份服務器B的備份目錄都爲/backup

       2)系統的配置文件包括，可是不限於以下

              a.定時任務服務的配置文件(/var/spool/cron/root)

              b.開機自啓的配置文件(/ect/rc.local)

              c.平常的腳本目錄(/server/scripts)

              d.防火牆iptables的配置文件(/etc/sysconfig/iptables)

       3)Web服務器的站點目的假定爲(/var/html/www)

       4)Web服務器A的訪問日誌的假定路徑爲(/app/log)

       5)Web服務器保留打包7天的備份數據便可(本地留存不能多餘7天，由於太多硬盤會滿，準備服務器上要保留6個月以上的數據副本)

       6)備份服務器上要按照備份數據服務的IP爲目錄保存，打包的文件按照時間名字保存

 

we solve it by a shell script and a crontab in every you can use this scripts

The shell scripts is following:

 

#!/bin/sh

#backup scripts

dir=/backup

ip=$(ifconfig|awk -F '[ :]+' 'NR==2{print $4}')

#if [ ! -d $dir/$ip ]

#then

 mkdir -p /backup/$ip

#fi

cd /backup && cd $ip

tar -zcf backup_$(date +%F).tar.gz /var/www/html/ /app/log/ /server/scripts/ /var/spool/cron/ /etc/

rsync -avz /backup/ rsync_backup@192.168.98.132::backup --password-file=/etc/rsync.password && touch ${ip}-flag-$(date +%F)

rsync -avz /backup/ rsync_backup@192.168.98.132::backup --password-file=/etc/rsync.password

find /backup/ -type f -name "backup*.tar.gz" -mtime +7 |xargs rm -f

 

then set it in crontab

 

#backup data by rsync

00 00 * * *  /bin/sh /server/scripts/backup/backup.sh > /dev/null 2>&1

 

Inotify

1 inotify instruction

inotify is a monitoring file system events, In a directory, if a file or directory is created or updated or deleted, it will tell you what file is change, then you can use rsync to deal with this file or directory.

 

Default, out machine don’t install it, so we need to install it at client

 

The steps of install is following:

 

install inotify: 

       一、dowmload inotify-tools-3.14.tar.gz

       二、tar -zxvf inotify-tools-3.14.tar.gz

       ll

       三、cd inotify-tools-3.14

       ll

       四、./configure --prefix=/usr/local/inotify-tools-3.14

       五、make && make install

       六、cd ../

       七、ln -s /usr/local/inotify-tools-3.14/ /usr/local/inotify

 

then, you can use this command 「」

2 Testing and Using at Client

view inotify help

       一、cd /usr/local/inotify

       二、./bin/inotifywait --help

 

 

              inotifywait 3.14

              Wait for a particular event on a file or set of files.

              Usage: inotifywait [ options ] file1 [ file2 ] [ file3 ] [ ... ]

              Options:

                      -h|--help       Show this help text.

                      @<file>         Exclude the specified file from being watched.

                      --exclude <pattern>

                                      Exclude all events on files matching the

                                      extended regular expression <pattern>.

                      --excludei <pattern>

                                      Like --exclude but case insensitive.

                      -m|--monitor    Keep listening for events forever.  Without

                                      this option, inotifywait will exit after one

                                      event is received.

                      -d|--daemon     Same as --monitor, except run in the background

                                      logging events to a file specified by --outfile.

                                      Implies --syslog.

                      -r|--recursive  Watch directories recursively.

                      --fromfile <file>

                                      Read files to watch from <file> or `-' for stdin.

                      -o|--outfile <file>

                                      Print events to <file> rather than stdout.

                      -s|--syslog     Send errors to syslog rather than stderr.

                      -q|--quiet      Print less (only print events).

                      -qq             Print nothing (not even events).

                      --format <fmt>  Print using a specified printf-like format

                                      string; read the man page for more details.

                      --timefmt <fmt> strftime-compatible format string for use with

                                      %T in --format string.

                      -c|--csv        Print events in CSV format.

                      -t|--timeout <seconds>

                                      When listening for a single event, time out after

                                      waiting for an event for <seconds> seconds.

                                      If <seconds> is 0, inotifywait will never time out.

                      -e|--event <event1> [ -e|--event <event2> ... ]

                              Listen for specific event(s).  If omitted, all events are

                              listened for.

 

              Exit status:

                      0  -  An event you asked to watch for was received.

                      1  -  An event you did not ask to watch for was received

                            (usually delete_self or unmount), or some error occurred.

                      2  -  The --timeout option was given and no events occurred

                            in the specified interval of time.

 

              Events:

                      access          file or directory contents were read

                      modify          file or directory contents were written

                      attrib          file or directory attributes changed

                      close_write     file or directory closed, after being opened in

                                      writeable mode

                      close_nowrite   file or directory closed, after being opened in

                                      read-only mode

                      close           file or directory closed, regardless of read/write mode

                      open            file or directory opened

                      moved_to        file or directory moved to watched directory

                      moved_from      file or directory moved from watched directory

                      move            file or directory moved to or from watched directory

                      create          file or directory created within watched directory

                      delete          file or directory deleted within watched directory

                      delete_self     file or directory was deleted

                      unmount         file system containing file or directory unmounted

 

 

some example:

       Number 1

              inotify listening command:

                                   /usr/local/inotify/bin/inotifywait -mrq --timefmt '%d-%m-%y %H:%M' --format '%T %w%f' -e create /backup

                     -m -r and -q and -e you can retrieve above

              execute following commands:

                                   touch b.txt

              listening result:     

                            17-01-17 09:43 /backup/b.txt

       Number 2

              inotify listening command:

                                   /usr/local/inotify/bin/inotifywait -mrq --timefmt '%d-%m-%y %H:%M' --format '%T %w%f' -e create,delete /backup

                     if want to listen more evenet, you can use "," as the seperator,%T is to print time %w is to print directory name

                     %f is to print filename

              execute following commands:

                                   mkdir t1

                                   touch t2

                                   rm t2

                                   rm -rf t1

              the listening results:

                                   17-01-17 09:46 /backup/t1

                                   17-01-17 09:46 /backup/t2

                                   17-01-17 09:48 /backup/t2

                                   17-01-17 09:48 /backup/t1

 

       Number 3:

              inotify listening command:

                                   /usr/local/inotify/bin/inotifywait -mrq --timefmt '%d-%m-%y %H:%M' --format '%T %w%f' -e create,delete,close_write /backup

              execute following commands:

                                   echo "sdsada

                                   touch c.txt

              the listening results:    

                                   17-01-17 09:54 /backup/a.txt

                                   17-01-17 09:55 /backup/c.txt

                                   17-01-17 09:55 /backup/c.txt

              the touch.txt will be listened by create and close_write

 

 

3 Examples

In some important situation, you should to backup data constantly, so you will use

inotify and rsync

you can use following shell scripts to solve it.

 

#!/bin/sh

#use inotify and rsync to backup data all time

host=192.168.98.132

src=/backup

dst=oldboy

user=rsync_backup

rsync_passfile=/etc/rsync.password

inotify_home=/usr/local/inotify-tools/

${inotify_home}bin/inotifywait -mrq --timefmt '%d-%m-%y %H:%M' --format '%w%f' -e close_write,delete,create,attrib $src \

|while read line

do

         rsync -avz $line $user@$host::$dst --password-file=${rsync_passfile} > /dev/null 2>&1

done

exit 0

 then you can run this script at  behind

/bin/sh /server/scripts/inotify/inotify1.sh &