mysqldump全量和增量備份,經過最近一次備份刷新產生binlog來定位執行增量。node
腳本下載地址mysql
場景一:git
天天執行一次全量備份.github
場景二:sql
每週日執行一次全量備份,而後天天3點執行增量備份.shell
應用場景:數據庫
增量備份在週一到週六凌晨3點,會使用mysqlbinlog 導出sql並使用gzip壓縮到指定目錄bash
全量備份則使用mysqldump將全部的數據庫導出,每週日凌晨3點執行,並會刪除N天以前的目錄和文件。參數以下:session
find $BASE_DIR -mtime + $DELETE_DAYS -type d -name "full*" -exec rm -rf {} \;
)MY_USER="gcdb" --備份賬號 MY_PASSWORD="iforgot" --備份密碼 MY_IP="192.168.49.247" --本機ip,例如從庫ip MY_MASTER_USER="gcdb" --master賬號 MY_MASTER_PASSWORD="iforgot" --master密碼 MY_MASTER_IP="192.168.49.246" --指主庫ip BINLOG_FILE=/r2/mysqldata --binlog文件所在的目錄,增量時須要用到 BASE_DIR=/mybak --備份基礎目錄 DELETE_DAYS=15 --備份保存天數,即刪除N天以前的備份,例如一週一個全備、每一天一個增量,該值必須大於配置爲7, FILTER="information_schema|test|mysql|sys|performance_schema" --過濾指定數據庫,也就是不備份的數據庫(注意mysql庫)
[root@node02 scripts]# tree /mybak/ /mybak/ ├── full │ ├── full_20180419 │ │ ├── backup.log │ │ ├── dbname │ │ ├── fullbak.sql.gz │ │ ├── grants.sql │ │ ├── master_grants.sql │ │ ├── master_users.sql │ │ ├── position │ │ └── users.sql │ └── full_20180420 │ ├── backup_full.log │ ├── dbname │ ├── fullbak.sql.gz │ ├── grants.sql │ ├── master_grants.sql │ ├── master_users.sql │ ├── position │ └── users.sql ├── incr │ ├── incr_20180420130001 │ │ ├── backup_incr.log │ │ ├── incr.sql.gz │ │ ├── new_binlogs_list │ │ ├── old_binlogs_list │ │ └── tmp_binlog_name │ └── incr_20180420140001 │ ├── backup_incr.log │ ├── incr.sql.gz │ ├── new_binlogs_list │ ├── old_binlogs_list │ └── tmp_binlog_name ├── public_backup.log --記錄備份是否成功 └── public_position --保存最新binlog文件名
備份命令app
備份命令
計劃任務
使用參考以下:
+-----------------------------------------------------------------------------+ |Usage : ./bak_mysql.sh (full|incr|oemu) | +-----------------------------------------------------------------------------+ |全備 :./bak_mysql.sh full | |增量 :./bak_mysql.sh incr | |只導出master權限 :./bak_mysql.sh oemu | +-----------------------------------------------------------------------------+ 計劃任務參考 +-----------------------------------------------------------------------------+ |全備 :30 0 * * * /bin/sh /scripts/bak_mysql.sh full >/dev/null 2>&1 | |增量 :30 2-23/2 * * * /bin/sh /scripts/bak_mysql.sh incr >/dev/null 2>&1 | +-----------------------------------------------------------------------------+ [root@node02 scripts]#
[root@node01 scripts]# sh bak_mysql.sh full 2>/dev/null +------------------+ | Backup_Host | +------------------+ | node01.mysql.com | +------------------+ Backup_Host 鏈接正常 +------------------+ | MY_Host | +------------------+ | node01.mysql.com | +------------------+ 192.168.49.245開始導出賬號和權限信息 192.168.49.245成功導出 10 個用戶權限 192.168.49.245成功導出 10 個用戶賬號 一、20180425 16:51:25 開始備份...... 二、備份如下數據庫: mysql percona 三、20180425 16:51:26 備份成功...... 四、備份用時: 1 秒 五、備份數據量大小: 6.9M 六、記錄最新的binlog文件名! +-------------+ | Master_Host | +-------------+ | slave7 | +-------------+ master 192.168.101.137開始導出賬號和權限信息 master 192.168.101.137成功導出 9 個用戶權限 master 192.168.101.137成功導出 9 個用戶賬號 全備成功 [root@node01 scripts]#
[root@node01 scripts]# cat /mybak/public_position binlog.000023 [root@node01 scripts]# cat /mybak/public_backup.log Backup_Host 鏈接正常 全備成功 刪除 /mybak/full 目錄下 7 天以前的備份! full_bakcup_ok [root@node01 scripts]# tree /mybak/full/full_20180425/ /mybak/full/full_20180425/ ├── backup_full.log ├── dbname --備份的庫名 ├── fullbak.sql.gz --備份文件 ├── grants.sql --本機受權文件(mysql5.7以後權限和賬號分開) ├── master_grants.sql --master受權文件 ├── master_users.sql --master賬號文件 ├── position --GTID和binlog文件名信息 └── users.sql --本機受權文件(mysql5.7以後權限和賬號分開) [root@node02 scripts]# cat /mybak/full/full_20180420/position -- GTID state at the beginning of the backup SET @@GLOBAL.GTID_PURGED='7debec7f-4797-11e8-9274-0050569d16ce:1-3, -- CHANGE MASTER TO MASTER_LOG_FILE='binlog.000023', MASTER_LOG_POS=234; [root@node01 scripts]# cat /mybak/full/full_20180425/position -- GTID state at the beginning of the backup SET @@GLOBAL.GTID_PURGED='7debec7f-4797-11e8-9274-0050569d16ce:1-3, -- CHANGE MASTER TO MASTER_LOG_FILE='binlog.000023', MASTER_LOG_POS=234; [root@node01 scripts]# cat /mybak/full/full_20180425/backup_full.log 一、20180425 16:51:25 開始備份...... 二、備份如下數據庫: mysql percona 三、20180425 16:51:26 備份成功...... 四、備份用時: 1 秒 五、備份數據量大小: 6.9M 六、記錄最新的binlog文件名! binlog.000023 master 192.168.101.137開始導出賬號和權限信息 master 192.168.101.137成功導出 9 個用戶權限 master 192.168.101.137成功導出 9 個用戶賬號 [root@node01 scripts]#
[2018-04-20 15:32:17.838][192.168.49.247-node02][000220][MYSQL] UPDATE `ttt`.`t1` SET `name` = 'rrrrrssss' WHERE `id` = 3 Time: 0.001s [2018-04-20 17:10:02.925][192.168.49.246-mycat][016413][MYSQL] UPDATE `ttt`.`t1` SET `name` = 'xiaowen' WHERE `id` = 3 Time: 0.002s [2018-04-20 17:11:42.657][192.168.49.246-mycat][016413][MYSQL] insert into t1 values(8,'xiaomi') Time: 0.001s
執行中
[root@node02 scripts]# ./bak_mysql.sh incr 2>/dev/null +------------------+ | Backup_Host | +------------------+ | node02.mysql.com | +------------------+ mysql鏈接正常 建立INCR_BACKUP_DIR目錄 /mybak/incr/incr_20180420171334 建立/mybak/incr/incr_20180420171334/backup_incr.log 000051 : PUBLIC_POSITION 有獲取到數值 循環寫入binlog名執行成功 mysqlbinlog 執行成功...... 寫入最新的binlog名到公共文件中 增量備份成功 刪除 /mybak/incr 目錄下 7 天以前的備份!
[root@node02 scripts]# ll /mybak/incr/incr_20180420171334 total 20 -rw-r--r-- 1 root root 2470 Apr 20 17:13 backup_incr.log -rw-r--r-- 1 root root 1488 Apr 20 17:13 incr.sql.gz -rw-r--r-- 1 root root 728 Apr 20 17:13 new_binlogs_list -rw-r--r-- 1 root root 714 Apr 20 17:13 old_binlogs_list -rw-r--r-- 1 root root 14 Apr 20 17:13 tmp_binlog_name [root@node02 scripts]# cat /mybak/incr/incr_20180420171334/backup_incr.log 不須要備份,後綴爲 000001 binlog文件 不須要備份,後綴爲 000002 binlog文件 ---------省略--------- 不須要備份,後綴爲 000049 binlog文件 不須要備份,後綴爲 000050 binlog文件 --全量備份到binlog.000050,flush logs生成了binlog.000051 需備份後綴爲 000051 binlog文件 mysqlbinlog 執行成功...... [root@node02 scripts]# cat /mybak/public_backup.log Backup_Host 鏈接正常 建立 /mybak/incr/incr_20180420171334/backup_incr.log incr_bakcup_ok 增量備份成功 刪除 /mybak/incr 目錄下 7 天以前的備份! [root@node02 scripts]# cat /mybak/incr/incr_20180420153720/tmp_binlog_name binlog.000051 [root@node02 scripts]# gunzip < /mybak/incr/incr_20180420171334/incr.sql.gz |more /*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; # at 4 #180420 17:02:35 server id 49247 end_log_pos 123 CRC32 0xdf164269 Start: binlog v 4, server v 5.7.18-log created 180420 17:02:35 BINLOG ' q6zZWg9fwAAAdwAAAHsAAAAAAAQANS43LjE4LWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAEzgNAAgAEgAEBAQEEgAAXwAEGggAAAAICAgCAAAACgoKKioAEjQA AWlCFt8= '/*!*/; # at 123 #180420 17:02:35 server id 49247 end_log_pos 234 CRC32 0x3cf6d5c4 Previous-GTIDs # 8a5dd931-42cc-11e8-aa39-0050569dc4ab:1-4, # fda7506d-33ea-11e8-b187-000c298b03f2:28759-28763 # at 234 #180420 17:09:48 server id 49246 end_log_pos 299 CRC32 0xe827d4c9 GTID last_committed=0 sequence_number=1 SET @@SESSION.GTID_NEXT= 'fda7506d-33ea-11e8-b187-000c298b03f2:28764'/*!*/; # at 299 #180420 17:09:48 server id 49246 end_log_pos 362 CRC32 0x52db9f29 Query thread_id=16413 exec_time=4294967295 error_ code=0 SET TIMESTAMP=1524215388/*!*/; SET @@session.pseudo_thread_id=16413/*!*/; SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=0, @@session.unique_checks=1, @@session.autocommit=1/*!*/; SET @@session.sql_mode=524288/*!*/; SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/; /*!\C utf8mb4 *//*!*/; SET @@session.character_set_client=45,@@session.collation_connection=45,@@session.collation_server=192/*!*/; SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; BEGIN /*!*/; # at 362 #180420 17:09:48 server id 49246 end_log_pos 409 CRC32 0x330344f1 Table_map: `ttt`.`t1` mapped to number 100172 # at 409 #180420 17:09:48 server id 49246 end_log_pos 473 CRC32 0x7a37b361 Update_rows: table id 100172 flags: STMT_END_F BINLOG ' XK7ZWhNewAAALwAAAJkBAAAAAEyHAQAAAAEAA3R0dAACdDEAAgMPAjwAAvFEAzM= XK7ZWh9ewAAAQAAAANkBAAAAAEyHAQAAAAEAAgAC///8AwAAAAlycnJycnNzc3P8AwAAAAd4aWFv d2VuYbM3eg== '/*!*/; ### UPDATE `ttt`.`t1` ### WHERE ### @1=3 /* INT meta=0 nullable=0 is_null=0 */ ### @2='rrrrrssss' /* VARSTRING(60) meta=60 nullable=1 is_null=0 */ ### SET ### @1=3 /* INT meta=0 nullable=0 is_null=0 */ ### @2='xiaowen' /* VARSTRING(60) meta=60 nullable=1 is_null=0 */ # at 473 #180420 17:09:48 server id 49246 end_log_pos 504 CRC32 0x3b8f41b5 Xid = 397443 COMMIT/*!*/; # at 504 #180420 17:09:48 server id 49246 end_log_pos 569 CRC32 0xc05fd7e0 GTID last_committed=1 sequence_number=2 SET @@SESSION.GTID_NEXT= 'fda7506d-33ea-11e8-b187-000c298b03f2:28765'/*!*/; # at 569 #180420 17:09:48 server id 49246 end_log_pos 632 CRC32 0x03de76ae Query thread_id=16413 exec_time=4294967295 error_ code=0 SET TIMESTAMP=1524215388/*!*/; BEGIN /*!*/; # at 632 #180420 17:09:48 server id 49246 end_log_pos 679 CRC32 0xb60b3faa Table_map: `ttt`.`t1` mapped to number 100172 # at 679 #180420 17:09:48 server id 49246 end_log_pos 726 CRC32 0xa2092f94 Write_rows: table id 100172 flags: STMT_END_F BINLOG ' XK7ZWhNewAAALwAAAKcCAAAAAEyHAQAAAAEAA3R0dAACdDEAAgMPAjwAAqo/C7Y= XK7ZWh5ewAAALwAAANYCAAAAAEyHAQAAAAEAAgAC//wFAAAABnhpYW9taZQvCaI= '/*!*/; ### INSERT INTO `ttt`.`t1` ### SET ### @1=5 /* INT meta=0 nullable=0 is_null=0 */ ### @2='xiaomi' /* VARSTRING(60) meta=60 nullable=1 is_null=0 */ # at 726 #180420 17:09:48 server id 49246 end_log_pos 757 CRC32 0xf7d054a7 Xid = 397445 COMMIT/*!*/; # at 757 #180420 17:11:28 server id 49246 end_log_pos 822 CRC32 0xb24fe15b GTID last_committed=2 sequence_number=3 SET @@SESSION.GTID_NEXT= 'fda7506d-33ea-11e8-b187-000c298b03f2:28766'/*!*/; # at 822 #180420 17:11:28 server id 49246 end_log_pos 885 CRC32 0xdb150b55 Query thread_id=16413 exec_time=4294967295 error_ code=0 SET TIMESTAMP=1524215488/*!*/; BEGIN /*!*/; # at 885 #180420 17:11:28 server id 49246 end_log_pos 932 CRC32 0xf91a1b61 Table_map: `ttt`.`t1` mapped to number 100172 # at 932 #180420 17:11:28 server id 49246 end_log_pos 979 CRC32 0xafce0c68 Write_rows: table id 100172 flags: STMT_END_F BINLOG ' wK7ZWhNewAAALwAAAKQDAAAAAEyHAQAAAAEAA3R0dAACdDEAAgMPAjwAAmEbGvk= wK7ZWh5ewAAALwAAANMDAAAAAEyHAQAAAAEAAgAC//wIAAAABnhpYW9taWgMzq8= '/*!*/; ### INSERT INTO `ttt`.`t1` ### SET ### @1=8 /* INT meta=0 nullable=0 is_null=0 */ ### @2='xiaomi' /* VARSTRING(60) meta=60 nullable=1 is_null=0 */ # at 979 #180420 17:11:28 server id 49246 end_log_pos 1010 CRC32 0x6e2f666e Xid = 397447 COMMIT/*!*/; # at 1010 #180420 17:13:34 server id 49247 end_log_pos 1054 CRC32 0x02255fe1 Rotate to binlog.000052 pos: 4 SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/; DELIMITER ; # End of log file /*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/; /*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/; [root@node02 scripts]#
[root@node02 scripts]# > /mybak/public_position --清空文件 [root@node02 scripts]# ./bak_mysql.sh incr 2>/dev/null +------------------+ | Backup_Host | +------------------+ | node02.mysql.com | +------------------+ mysql鏈接正常 建立INCR_BACKUP_DIR目錄 /mybak/incr/incr_20180420172634 /mybak/incr/incr_20180420172634/backup_incr.log 不存在,從新建立. OLD_NUM : PUBLIC_POSITION 沒有獲取到數值,執行全備 --這裏執行全備 0、(1)成功導出 7 個用戶權限 0、(2)成功導出 7 個用戶賬號 一、20180420 17:26:35 開始備份...... 二、備份如下數據庫: cmd ttt 三、20180420 17:28:05 備份成功...... 四、備份用時: 90 秒 五、備份數據量大小: 256M 六、記錄最新的binlog文件名! 開始導出master賬號和權限信息 +-------------------+ | Master_Host | +-------------------+ | mycat01.mysql.com | +-------------------+ 0、(1)master成功導出 8 個用戶權限 0、(2)master成功導出 8 個用戶賬號 全備執行成功 增量備份失敗 增量失敗,刪除備份目錄 --注意:刪除剛纔建立的增量 [root@node02 scripts]# ll /mybak/incr/incr_20180420172634 --增量目錄,顯示已被刪除 ls: cannot access /mybak/incr/incr_20180420172634: No such file or directory
恢復使用全備進行恢復
[root@node02 scripts]# ls -l /mybak/full/full_20180424/ total 258336 -rw-r--r-- 1 root root 526 Apr 24 12:01 backup_full.log -rw-r--r-- 1 root root 176 Apr 24 12:00 dbname -rw-r--r-- 1 root root 264506232 Apr 24 12:01 fullbak.sql.gz -rw-r--r-- 1 root root 1134 Apr 24 12:00 grants.sql -rw-r--r-- 1 root root 1200 Apr 24 12:01 master_grants.sql -rw-r--r-- 1 root root 1630 Apr 24 12:01 master_users.sql -rw-r--r-- 1 root root 187 Apr 24 12:01 position -rw-r--r-- 1 root root 1438 Apr 24 12:00 users.sql [root@node02 scripts]#
GTID模式在原機上重作從庫,須要reset master,清空master,再導入
(root@localhost) 16:21:27 [(none)]> stop slave; Query OK, 0 rows affected, 1 warning (0.00 sec) (root@localhost) 16:21:58 [(none)]> reset master; Query OK, 0 rows affected (0.16 sec)
[root@node02 full_20180424]# cat position -- GTID state at the beginning of the backup SET @@GLOBAL.GTID_PURGED='84865d81-b573-11e7-9668-b8ca3a65693c:1-57436835'; -- CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000079', MASTER_LOG_POS=194; [root@node02 full_20180424]# gunzip <fullbak.sql.gz |mysql -uroot -pxxxxxxx mysql: [Warning] Using a password on the command line interface can be insecure.
步驟3 CHANGE MASTER
(root@localhost) 16:53:09 [(none)]> CHANGE MASTER TO MASTER_HOST='192.168.101.137',MASTER_USER='repl',MASTER_PASSWORD='XXXXX',MASTER_AUTO_POSITION=1; Query OK, 0 rows affected, 2 warnings (0.31 sec) (root@localhost) 16:53:27 [(none)]> start slave ; Query OK, 0 rows affected (0.00 sec) (root@localhost) 16:53:34 [(none)]> show slave status \G; *************************** 1. row *************************** Slave_IO_State: Queueing master event to the relay log Master_Host: 192.168.101.137 Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000081 Read_Master_Log_Pos: 348638 Relay_Log_File: node01-relay-bin.000002 Relay_Log_Pos: 869569 Relay_Master_Log_File: mysql-bin.000079 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 869396 Relay_Log_Space: 29714750 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 49801 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_UUID: 84865d81-b573-11e7-9668-b8ca3a65693c Master_Info_File: /r2/mysqldata/master.info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: update Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: 84865d81-b573-11e7-9668-b8ca3a65693c:57436836-57469553 Executed_Gtid_Set: 84865d81-b573-11e7-9668-b8ca3a65693c:1-57437364 Auto_Position: 1 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: 1 row in set (0.26 sec) ERROR: No query specified
[root@node02 scripts]# sh bak_mysql.sh oemu 2>/dev/null +------------------+ | Backup_Host | +------------------+ | node02.mysql.com | +------------------+ mysql鏈接正常 master 192.168.49.246開始導出賬號和權限信息 +-------------------+ | Master_Host | +-------------------+ | mycat01.mysql.com | +-------------------+ master 192.168.49.246成功導出 8 個用戶權限 master 192.168.49.246成功導出 8 個用戶賬號 master 192.168.49.246導出用戶賬號和權限成功 [root@node02 scripts]# ll /mybak/ total 20 drwxr-xr-x 4 root root 48 Apr 24 13:46 full drwxr-xr-x 27 root root 4096 Apr 24 13:46 incr -rw-r--r-- 1 root root 1200 Apr 25 08:40 master_grants.sql --導出權限 -rw-r--r-- 1 root root 1630 Apr 25 08:40 master_users.sql --導出賬號 -rw-r--r-- 1 root root 232 Apr 25 08:40 public_backup.log -rw-r--r-- 1 root root 17 Apr 24 13:46 public_position [root@node01 scripts]#
從庫作全備在主庫上恢復(步驟同上1-2),建立賬號和權限(master_users.sql,master_grants.sql),刷新權限(flush privilegs;),而後作全備,再作從庫