mysql邏輯備份mydumper

 

mydumper是一個針對MySQL的高性能多線程備份和恢復工具，它提供了併發備份功能，備份效率有很大提升。

安裝mydumper

• yum安裝

  # rpm -ivh https://github.com/maxbube/mydumper/releases/download/v0.9.5/mydumper-0.9.5-2.el7.x86_64.rpm

• 編譯安裝

  # yum install glib2-devel mysql-devel zlib-devel pcre-devel openssl-devel
  # wget https://github.com/maxbube/mydumper/archive/v0.9.5.tar.gz
  # tar xvf v0.9.5.tar.gz
  # cd mydumper-0.9.5
  # cmake .
  # make

備份

mydumper備份時不是像mysqldump同樣指定到某個特定的sql文件，而須要指定一個備份目錄：

mydumper -h ${host} --user ${user} --password ${passwd}  -B ${DB_NAME} -o ${bakdir} -e -t 16

mydumper --help
Usage:
mydumper [OPTION...] multi-threaded MySQL dumping
Help Options:
-?, --help Show help options
Application Options:
-B, --database Database to dump
# 導出指定庫
-T, --tables-list Comma delimited table list to dump (does not exclude regex option)
# 導出指定的表，多個表之間用逗號分隔。db1.t1,db1,t2,db2,t1,db3,t1
-O, --omit-from-file File containing a list of database.table entries to skip, one per line (skips before applying regex option)
-o, --outputdir Directory to output files to
# 指定備份目錄
-s, --statement-size Attempted size of INSERT statement in bytes, default 1000000
# 指定導出SQL文件中每一個insert語句的大小，默認每一個insert 1000000value
-r, --rows Try to split tables into chunks of this many rows. This option turns off --chunk-filesize
-F, --chunk-filesize Split tables into chunks of this output file size. This value is in MB
-c, --compress Compress output files
# 壓縮備份，導出SQL文件以sql.gz結尾
-e, --build-empty-files Build dump files even if no data available from table
# 若是空表也生成對應的文件名
-x, --regex Regular expression for 'db.table' matching
正則匹配庫和表
-i, --ignore-engines Comma delimited list of storage engines to ignore
# 忽略不須要導出指定存儲引擎的表
-N, --insert-ignore Dump rows with INSERT IGNORE
# 導出的SQL文件加上INSERT IGNORE
-m, --no-schemas Do not dump table schemas with the data
# 不導表結構，只導數據
-d, --no-data Do not dump table data
# 不導數據，只導表結構
-G, --triggers Dump triggers
# 備份觸發器
-E, --events Dump events
# 備份事件
-R, --routines Dump stored procedures and functions
# 備份存儲過程和函數
-W, --no-views Do not dump VIEWs
# 不導視圖
-k, --no-locks Do not execute the temporary shared read lock. WARNING: This will cause inconsistent backups
# 備份時不加全局讀鎖，
--no-backup-locks Do not use Percona backup locks
# 不使用 Percona backup locks
--less-locking Minimize locking time on InnoDB tables.
-l, --long-query-guard Set long query timer in seconds, default 60
# 設置執行多長時間SQL將會被KILL
-K, --kill-long-queries Kill long running queries (instead of aborting)
# kill長時間執行的SQL
# -D, --daemon Enable daemon mode
以守護進程進行備份，默認每60分鐘備份一次
-I, --snapshot-interval Interval between each dump snapshot (in minutes), requires --daemon, default 60
# 每一個備份之間的時間間隔（以分鐘爲單位）須要--daemon，默認值爲60
-L, --logfile Log file name to use, by default stdout is used
# 指定備份過程當中的日誌記錄，默認標準輸出
--tz-utc SET TIME_ZONE='+00:00' at top of dump to allow dumping of TIMESTAMP data when a server has data in different time zones or data is being moved between servers with different time zones, defaults to on use --skip-tz-utc to disable.
# 備份時時區設置，默認在備份文件開始添加SET TIME_ZONE='+00:00'語句
--skip-tz-utc
# 禁止在備份文件中輸出SET TIME_ZONE語句
--use-savepoints Use savepoints to reduce metadata locking issues, needs SUPER privilege
# 使用savepoints來減小元數據鎖定問題，須要SUPER權限
--success-on-1146 Not increment error count and Warning instead of Critical in case of table doesn't exist
--lock-all-tables Use LOCK TABLE for all, instead of FTWRL
-U, --updated-since Use Update_time to dump only tables updated in the last U days
--trx-consistency-only Transactional consistency only
--complete-insert Use complete INSERT statements that include column names
# 導出的備份文件帶完整的字段名
-h, --host The host to connect to
-u, --user Username with the necessary privileges
-p, --password User password
-a, --ask-password Prompt For User password
# 交互輸入密碼
-P, --port TCP/IP port to connect to
-S, --socket UNIX domain socket file to use for connection
-t, --threads Number of threads to use, default 4
-C, --compress-protocol Use compression on the MySQL connection
# 備份傳輸使用壓縮
-V, --version Show the program version and exit
-v, --verbose Verbosity of output, 0 = silent, 1 = errors, 2 = warnings, 3 = info, default 2
--defaults-file Use a specific defaults file
# 指定mysql my.cnf配置文件

備份完成後，能夠查看備份目錄下metadata文件，查看備份是否成功。

# cat metadata
Started dump at: 2019-06-19 17:02:45
SHOW MASTER STATUS:
    Log: mysql-bin.002714
    Pos: 73794931
    GTID:xxxxxxxx

SHOW SLAVE STATUS:
    Host: 192.168.1.10
    Log: mysql-bin.002721
    Pos: 133250446
    GTID:xxxxx

Finished dump at: 2019-06-19 17:59:20

View Code

 

恢復

在恢復以前首先把sql_mode設置爲空，不然會報錯

mysql> show global variables like '%sql_mode%';
+---------------+------------------------------------------------------------------------------------------------------------------------+
| Variable_name | Value                                                                                                                  |
+---------------+------------------------------------------------------------------------------------------------------------------------+
| sql_mode      | STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION |
+---------------+------------------------------------------------------------------------------------------------------------------------+
mysql> set global sql_mode='';
執行完myloader以後再恢復設置
mysql> set global sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION';

 

myloader -u ${user} -p ${passwd} -h ${host} -o -t 8 -d  ${bakdir}  

myloader參數介紹：
-d, --directory # 導入備份目錄
-q, --queries-per-transaction #每次執行的查詢數量, 默認1000
-o, --overwrite-tables #若是表存在刪除表
-B, --database  # 須要還原的庫
-s, --source-db Database to restore  # 從備份文件把-s 指定的庫恢復到數據庫中
-e, --enable-binlog # 啓用二進制恢復數據。恢復時是否開啓binlog，默認導入時關閉binlog（有主從環境注意若是-e 那麼從庫也會一塊兒恢復，可是速度較慢）
-h, --host The host to connect to -u, --user Username with the necessary privileges
-p, --password User password
-a, --ask-password Prompt For User password
-P, --port TCP/IP port to connect to
-S, --socket UNIX domain socket file to use for connection  #指定socket file
-t, --threads Number of threads to use, default 4  # 指定線程數，默認爲4
-C, --compress-protocol Use compression on the MySQL connection
-V, --version Show the program version and exit
-v, --verbose Verbosity of output, 0 = silent, 1 = errors, 2 = warnings, 3 = info, default 2
--defaults-file Use a specific defaults file

 

備份恢復示例

master： 172.16.2.11    slave: 172.16.2.22

將master數據庫的數據備份出來，恢復到slave裏面，並配置主從同步。

1.在master建立備份用戶和權限。
mysql> GRANT SELECT, RELOAD, PROCESS, SHOW DATABASES, SUPER, LOCK TABLES, REPLICATION SLAVE, REPLICATION CLIENT, SHOW VIEW, EVENT, TRIGGER ON *.* TO 'backup'@'172.16.2.22' identified by 'xxxxxx';
mysql > flush privileges;

2. 在slave備份遠程庫到本地
# mkdir /data/mydumper
# mydumper -G -E -R -h 172.16.2.11 --user backup --password xxxxx -o /data/mydumper/ -e -t 8

3.恢復。將mydumper備份的數據恢復到數據庫裏。
mysql> show global variables like '%sql_mode%';
+---------------+------------------------------------------------------------------------------------------------------------------------+
| Variable_name | Value |
+---------------+------------------------------------------------------------------------------------------------------------------------+
| sql_mode | STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION |
+---------------+------------------------------------------------------------------------------------------------------------------------+
mysql> set global sql_mode='';
恢復步驟：
myloader -u root -p xxxxxx -o -t 16 -d /data/mydumper/ -S /tmp/mysql.sock

恢復完成後，將sql_mode還原：
mysql> set global sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION';

4.待數據恢復完成後，將slave和master配置成主從，同步數據。
mysql> CHANGE MASTER TO MASTER_HOST='172.16.2.11',MASTER_USER='repl',MASTER_PASSWORD='xxxxxx',MASTER_AUTO_POSITION=0,MASTER_LOG_FILE='mysql-bin.002714',MASTER_LOG_POS=73794931;
# binlog file和position在mydumper目錄的metadata文件裏找。mysql> start slave;