MySQL備份可能遇到的坑
MySQL備份工具，支持各類參數選項，使用不一樣的選項極有可能影響備份處理過程。本文使用咱們常規認爲合理的備份參數，測試/驗證是否存在容易忽視的坑html
# 常規備份參數 # mysqldump shell> mysqldump --single-transaction --master-data=2 -B replcrash >dbname_dump_serverid_`date +%Y%m%d`.sql # mysqlpump shell> mysqlpump --single-transaction -B replcrash >dbname_pump_serverid_`date +%Y%m%d`.sql # mydumper shell> mydumper -B replcrash -o /data/backup/mydumper # XtraBackup # backup shell> innobackupex [--defaults-file=MY.CNF] BACKUP-ROOT-DIR # apply-log shell> innobackupex --apply-log [--defaults-file=MY.CNF] BACKUP-DIR # copy-back shell> innobackupex --copy-back [--defaults-file=MY.CNF] BACKUP-DIR
官方社區版MySQL 5.7.19 基於Row+Position搭建的一主一從異步複製結構：Master->{Slave}python
| ROLE | HOSTNAME | BASEDIR | DATADIR | IP | PORT |
| Master | ZST1 | /usr/local/mysql | /data/mysql/mysql3306/data | 192.168.85.132 | 3306 |
| Slave | ZST1 | /usr/local/mysql | /data/mysql/mysql3308/data | 192.168.85.132 | 3308 |
每次使用備份文件還原數據庫後，從新搭建這個複製結構
備份工具版本：mysqldump、mysqlpump是MySQL 5.7.19中自帶的；mydumper version 0.9.三、innobackupex version 2.4.8mysql
1、mysqldump
1.一、DML操做對備份的影響
建立兩張測試表linux
# 建立兩張測試表(192.168.85.132,3306) use replcrash; create table py_user_innodb( uid int not null auto_increment, name varchar(32), add_time datetime default current_timestamp, server_id varchar(10), primary key(uid), key(name) )engine=innodb; create table py_user_myisam( uid int not null auto_increment, name varchar(32), add_time datetime default current_timestamp, server_id varchar(10), primary key(uid), key(name) )engine=myisam;
運行下面的腳本持續往測試表中寫入數據git
#!/user/bin/python import string import random import MySQLdb import time conn = MySQLdb.connect(host='192.168.85.132', port=3306, user='mydba', passwd='mysql5719', db='replcrash') """ create table py_user( uid int not null auto_increment, name varchar(32), add_time datetime default current_timestamp, server_id varchar(10), primary key(uid), key(name) ); """ while True: r_name = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(random.randint(20,30))) print r_name cursor = conn.cursor() cursor.execute("insert into py_user_myisam(name,add_time,server_id) values('%s',now(),@@server_id);" % str(r_name)) cursor.execute("insert into py_user_innodb(name,add_time,server_id) values('%s',now(),@@server_id);" % str(r_name)) conn.commit() time.sleep(0.001)
開啓general_log，用來查看mysqldump執行過程github
# 開啓general_log mydba@192.168.85.132,3306 [replcrash]> set global general_log_file='/data/mysql/mysql3306/data/mysql-general.log'; mydba@192.168.85.132,3306 [replcrash]> set global general_log=1; # 清空general_log [root@ZST1 logs]# cat /dev/null > /data/mysql/mysql3306/data/mysql-general.log # 備份replcrash數據庫 [root@ZST1 backup]# mysqldump -h127.0.0.1 -P3306 -uroot -p --single-transaction --master-data=2 replcrash >/data/backup/replcrash_dump_1323306_`date +%Y%m%d`.sql
使用備份文件搭建複製sql
# 還原實例清空GTID信息 mydba@192.168.85.132,3308 [replcrash]> reset master; # 還原數據 [root@ZST1 backup]# mysql -h127.0.0.1 -P3308 -uroot -p replcrash </data/backup/replcrash_dump_1323306_`date +%Y%m%d`.sql # 搭建複製 mydba@192.168.85.132,3308 [replcrash]> change master to master_host='192.168.85.132', master_port=3306, master_user='repl', master_password='repl', master_auto_position=1; # 啓動複製，查看複製狀態 mydba@192.168.85.132,3308 [replcrash]> start slave; mydba@192.168.85.132,3308 [replcrash]> show slave status\G *************************** 1. row *************************** Master_Log_File: mysql-bin.000183 Read_Master_Log_Pos: 1541377 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 741 Relay_Master_Log_File: mysql-bin.000183 Slave_IO_Running: Yes Slave_SQL_Running: No Exec_Master_Log_Pos: 1042768 Last_SQL_Errno: 1062 Last_SQL_Error: Could not execute Write_rows event on table replcrash.py_user_myisam; Duplicate entry '332' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event''s master log mysql-bin.000183, end_log_pos 1043062 Retrieved_Gtid_Set: 8ab82362-9c37-11e7-a858-000c29c1025c:251874-253268 Executed_Gtid_Set: 8ab82362-9c37-11e7-a858-000c29c1025c:1-251874 Auto_Position: 1
從上面的結果中能夠看到，主鍵衝突了，在從庫查詢一下這個表中大於等於衝突key的數據shell
# 查詢從庫出錯表大於等於衝突key的數據 mydba@192.168.85.132,3308 [replcrash]> select * from replcrash.py_user_myisam where uid>=332; +-----+--------------------------------+---------------------+-----------+ | uid | name | add_time | server_id | +-----+--------------------------------+---------------------+-----------+ | 332 | X1LME9HO5V7WXNOKBVZE | 2018-01-02 09:05:07 | 1323306 | | 333 | 2PBFQ7KS4BPIJ27G88EYXWEDSX5 | 2018-01-02 09:05:07 | 1323306 | | 334 | E85Y2SS9UD0FZG4YGCNTRSWA8L | 2018-01-02 09:05:07 | 1323306 | | 335 | Y2TQOEVJ58NN7EREL4WRZ | 2018-01-02 09:05:07 | 1323306 | | 336 | O0MEATAXYIAE2V2IZG96YVQ56WEUHF | 2018-01-02 09:05:07 | 1323306 | | 337 | A6QKRWEXHRGUA3V2CH61VXUNBVA3H2 | 2018-01-02 09:05:07 | 1323306 | | 338 | NYCSI1HS61BN6QAVVYTZSC | 2018-01-02 09:05:07 | 1323306 | | 339 | 7CFC1JQPIQGNC97MDTT8ZIMIZL7D | 2018-01-02 09:05:07 | 1323306 | | 340 | GA78AR4Z12WQTEAM41JB | 2018-01-02 09:05:07 | 1323306 | +-----+--------------------------------+---------------------+-----------+ 9 rows in set (0.08 sec)
咱們查看mysqldump備份文件獲取的binlog pos數據庫
[root@ZST1 backup]# more replcrash_dump_1323306_20180102.sql -- GTID state at the beginning of the backup SET @@GLOBAL.GTID_PURGED='8ab82362-9c37-11e7-a858-000c29c1025c:1-251873'; -- Position to start replication or point-in-time recovery from -- CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000183', MASTER_LOG_POS=1042441; [root@ZST1 backup]#
這裏的pos信息是mysqldump經過SHOW MASTER STATUS獲取。查看mysqldump獲得的general-log;vim
[root@ZST1 data]# vim /data/mysql/mysql3306/data/mysql-general.log ... 2018-01-02T01:05:07.693104Z 10 Query FLUSH /*!40101 LOCAL */ TABLES 2018-01-02T01:05:07.694738Z 9 Query insert into py_user_myisam(name,add_time,server_id) values('7ATZSNFNIBW5DZNMNZYBMV',now(),@@server_id) 2018-01-02T01:05:07.701616Z 9 Query insert into py_user_innodb(name,add_time,server_id) values('7ATZSNFNIBW5DZNMNZYBMV',now(),@@server_id) 2018-01-02T01:05:07.702139Z 10 Query FLUSH TABLES WITH READ LOCK 2018-01-02T01:05:07.702344Z 9 Query commit 2018-01-02T01:05:07.702411Z 10 Query SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ 2018-01-02T01:05:07.702597Z 10 Query START TRANSACTION /*!40100 WITH CONSISTENT SNAPSHOT */ 2018-01-02T01:05:07.702721Z 10 Query SHOW VARIABLES LIKE 'gtid\_mode' 2018-01-02T01:05:07.713019Z 10 Query SELECT @@GLOBAL.GTID_EXECUTED 2018-01-02T01:05:07.713179Z 10 Query SHOW MASTER STATUS 2018-01-02T01:05:07.725821Z 10 Query UNLOCK TABLES 2018-01-02T01:05:07.732125Z 9 Query insert into py_user_myisam(name,add_time,server_id) values('X1LME9HO5V7WXNOKBVZE',now(),@@server_id) 2018-01-02T01:05:07.733237Z 9 Query insert into py_user_innodb(name,add_time,server_id) values('X1LME9HO5V7WXNOKBVZE',now(),@@server_id) 2018-01-02T01:05:07.734240Z 9 Query commit 2018-01-02T01:05:07.740508Z 10 Query SELECT LOGFILE_GROUP_NAME, FILE_NAME, TOTAL_EXTENTS, INITIAL_SIZE, ENGINE, EXTRA FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'UNDO LOG' AND FILE_NAME IS NOT NULL AND LOGFILE_GROUP_NAME IS NOT NULL AND LOGFILE_GROUP_NAME IN (SELECT DISTINCT LOGFILE_GROUP_NAME FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'DATAFILE' AND TABLESPACE_NAME IN (SELECT DISTINCT TABLESPACE_NAME FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_SCHEMA IN ('replcrash'))) GROUP BY LOGFILE_GROUP_NAME, FILE_NAME, ENGINE, TOTAL_EXTENTS, INITIAL_SIZE ORDER BY LOGFILE_GROUP_NAME 2018-01-02T01:05:07.741895Z 9 Query insert into py_user_myisam(name,add_time,server_id) values('2PBFQ7KS4BPIJ27G88EYXWEDSX5',now(),@@server_id) 2018-01-02T01:05:07.742720Z 9 Query insert into py_user_innodb(name,add_time,server_id) values('2PBFQ7KS4BPIJ27G88EYXWEDSX5',now(),@@server_id) 2018-01-02T01:05:07.743257Z 9 Query commit 2018-01-02T01:05:07.749840Z 9 Query insert into py_user_myisam(name,add_time,server_id) values('E85Y2SS9UD0FZG4YGCNTRSWA8L',now(),@@server_id) 2018-01-02T01:05:07.750588Z 9 Query insert into py_user_innodb(name,add_time,server_id) values('E85Y2SS9UD0FZG4YGCNTRSWA8L',now(),@@server_id) 2018-01-02T01:05:07.750989Z 9 Query commit 2018-01-02T01:05:07.754180Z 10 Query SELECT DISTINCT TABLESPACE_NAME, FILE_NAME, LOGFILE_GROUP_NAME, EXTENT_SIZE, INITIAL_SIZE, ENGINE FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'DATAFILE' AND TABLESPACE_NAME IN (SELECT DISTINCT TABLESPACE_NAME FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_SCHEMA IN ('replcrash')) ORDER BY TABLESPACE_NAME, LOGFILE_GROUP_NAME 2018-01-02T01:05:07.756229Z 9 Query insert into py_user_myisam(name,add_time,server_id) values('Y2TQOEVJ58NN7EREL4WRZ',now(),@@server_id) 2018-01-02T01:05:07.757030Z 9 Query insert into py_user_innodb(name,add_time,server_id) values('Y2TQOEVJ58NN7EREL4WRZ',now(),@@server_id) 2018-01-02T01:05:07.757598Z 9 Query commit 2018-01-02T01:05:07.763629Z 9 Query insert into py_user_myisam(name,add_time,server_id) values('O0MEATAXYIAE2V2IZG96YVQ56WEUHF',now(),@@server_id) 2018-01-02T01:05:07.764626Z 9 Query insert into py_user_innodb(name,add_time,server_id) values('O0MEATAXYIAE2V2IZG96YVQ56WEUHF',now(),@@server_id) 2018-01-02T01:05:07.765654Z 9 Query commit 2018-01-02T01:05:07.766769Z 10 Query SHOW VARIABLES LIKE 'ndbinfo\_version' 2018-01-02T01:05:07.773997Z 9 Query insert into py_user_myisam(name,add_time,server_id) values('A6QKRWEXHRGUA3V2CH61VXUNBVA3H2',now(),@@server_id) 2018-01-02T01:05:07.774757Z 9 Query insert into py_user_innodb(name,add_time,server_id) values('A6QKRWEXHRGUA3V2CH61VXUNBVA3H2',now(),@@server_id) 2018-01-02T01:05:07.775198Z 9 Query commit 2018-01-02T01:05:07.779582Z 9 Query insert into py_user_myisam(name,add_time,server_id) values('NYCSI1HS61BN6QAVVYTZSC',now(),@@server_id) 2018-01-02T01:05:07.780174Z 10 Init DB replcrash 2018-01-02T01:05:07.780249Z 10 Query SAVEPOINT sp 2018-01-02T01:05:07.780913Z 9 Query insert into py_user_innodb(name,add_time,server_id) values('NYCSI1HS61BN6QAVVYTZSC',now(),@@server_id) 2018-01-02T01:05:07.781387Z 9 Query commit 2018-01-02T01:05:07.781776Z 10 Query show tables 2018-01-02T01:05:07.782078Z 10 Query show table status like 'py\_user' 2018-01-02T01:05:07.782400Z 10 Query SET SQL_QUOTE_SHOW_CREATE=1 2018-01-02T01:05:07.782513Z 10 Query SET SESSION character_set_results = 'binary' 2018-01-02T01:05:07.787051Z 9 Query insert into py_user_myisam(name,add_time,server_id) values('7CFC1JQPIQGNC97MDTT8ZIMIZL7D',now(),@@server_id) 2018-01-02T01:05:07.787810Z 9 Query insert into py_user_innodb(name,add_time,server_id) values('7CFC1JQPIQGNC97MDTT8ZIMIZL7D',now(),@@server_id) 2018-01-02T01:05:07.788502Z 9 Query commit 2018-01-02T01:05:07.788774Z 10 Query show create table `py_user` 2018-01-02T01:05:07.789570Z 10 Query SET SESSION character_set_results = 'utf8' 2018-01-02T01:05:07.789725Z 10 Query show fields from `py_user` 2018-01-02T01:05:07.790423Z 10 Query show fields from `py_user` 2018-01-02T01:05:07.791163Z 10 Query SELECT /*!40001 SQL_NO_CACHE */ * FROM `py_user` 2018-01-02T01:05:07.791447Z 10 Query SET SESSION character_set_results = 'binary' 2018-01-02T01:05:07.791648Z 10 Query use `replcrash` 2018-01-02T01:05:07.791778Z 10 Query select @@collation_database 2018-01-02T01:05:07.791929Z 10 Query SHOW TRIGGERS LIKE 'py\_user' 2018-01-02T01:05:07.792383Z 10 Query SET SESSION character_set_results = 'utf8' 2018-01-02T01:05:07.792492Z 10 Query ROLLBACK TO SAVEPOINT sp 2018-01-02T01:05:07.792651Z 10 Query show table status like 'py\_user\_innodb' 2018-01-02T01:05:07.792874Z 10 Query SET SQL_QUOTE_SHOW_CREATE=1 2018-01-02T01:05:07.792948Z 10 Query SET SESSION character_set_results = 'binary' 2018-01-02T01:05:07.793024Z 10 Query show create table `py_user_innodb` 2018-01-02T01:05:07.793131Z 10 Query SET SESSION character_set_results = 'utf8' 2018-01-02T01:05:07.793220Z 10 Query show fields from `py_user_innodb` 2018-01-02T01:05:07.793607Z 10 Query show fields from `py_user_innodb` 2018-01-02T01:05:07.793985Z 10 Query SELECT /*!40001 SQL_NO_CACHE */ * FROM `py_user_innodb` 2018-01-02T01:05:07.794435Z 9 Query insert into py_user_myisam(name,add_time,server_id) values('GA78AR4Z12WQTEAM41JB',now(),@@server_id) 2018-01-02T01:05:07.795204Z 9 Query insert into py_user_innodb(name,add_time,server_id) values('GA78AR4Z12WQTEAM41JB',now(),@@server_id) 2018-01-02T01:05:07.795688Z 9 Query commit 2018-01-02T01:05:07.798108Z 10 Query SET SESSION character_set_results = 'binary' 2018-01-02T01:05:07.798205Z 10 Query use `replcrash` 2018-01-02T01:05:07.798303Z 10 Query select @@collation_database 2018-01-02T01:05:07.798408Z 10 Query SHOW TRIGGERS LIKE 'py\_user\_innodb' 2018-01-02T01:05:07.798884Z 10 Query SET SESSION character_set_results = 'utf8' 2018-01-02T01:05:07.798965Z 10 Query ROLLBACK TO SAVEPOINT sp 2018-01-02T01:05:07.799049Z 10 Query show table status like 'py\_user\_myisam' 2018-01-02T01:05:07.799271Z 10 Query SET SQL_QUOTE_SHOW_CREATE=1 2018-01-02T01:05:07.799344Z 10 Query SET SESSION character_set_results = 'binary' 2018-01-02T01:05:07.799420Z 10 Query show create table `py_user_myisam` 2018-01-02T01:05:07.799554Z 10 Query SET SESSION character_set_results = 'utf8' 2018-01-02T01:05:07.799661Z 10 Query show fields from `py_user_myisam` 2018-01-02T01:05:07.800098Z 10 Query show fields from `py_user_myisam` 2018-01-02T01:05:07.800418Z 10 Query SELECT /*!40001 SQL_NO_CACHE */ * FROM `py_user_myisam`
mysqldump備份過程(--single-transaction --master-data)：
會話先執行FTWRL(實例只讀)，而後設置RR隔離級別->START TRANSACTION WITH CONSISTENT SNAPSHOT;->SHOW MASTER STATUS;->UNLOCK TABLES;->SELECT /*!40001 SQL_NO_CACHE */ * FROM `tbname`;
在UNLOCK TABLES解鎖後其餘事務就能夠進行寫入操做。general-log中咱們可看到 UNLOCK TABLES 到 SELECT /*!40001 SQL_NO_CACHE */ * FROM `py_user_myisam` 之間往py_user_myisam、py_user_innodb各寫入9條數據
SELECT /*!40001 SQL_NO_CACHE */ * FROM `py_user_innodb`;讀取的是START TRANSACTION WITH CONSISTENT SNAPSHOT創建時的數據
SELECT /*!40001 SQL_NO_CACHE */ * FROM `py_user_myisam`;讀取的是最新的數據
再來查看上述過程期間binary log記錄
[root@ZST1 logs]# mysqlbinlog -vv --base64-output=decode-rows mysql-bin.000183 |more ... COMMIT/*!*/; # at 1042059 #180102 9:05:07 server id 1323306 end_log_pos 1042124 CRC32 0x221cda50 GTID last_committed=2917 sequence_number=2918 rbr_only=yes /*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/; SET @@SESSION.GTID_NEXT= '8ab82362-9c37-11e7-a858-000c29c1025c:251873'/*!*/; # at 1042124 #180102 9:05:07 server id 1323306 end_log_pos 1042209 CRC32 0x5df266e4 Query thread_id=9 exec_time=0 error_code=0 SET TIMESTAMP=1514855107/*!*/; BEGIN /*!*/; # at 1042209 #180102 9:05:07 server id 1323306 end_log_pos 1042279 CRC32 0xc1d41c5f Table_map: `replcrash`.`py_user_myisam` mapped to number 254 # at 1042279 #180102 9:05:07 server id 1323306 end_log_pos 1042355 CRC32 0x27badc02 Write_rows: table id 254 flags: STMT_END_F ### INSERT INTO `replcrash`.`py_user_myisam` ### SET ### @1=331 /* INT meta=0 nullable=0 is_null=0 */ ### @2='7ATZSNFNIBW5DZNMNZYBMV' /* VARSTRING(96) meta=96 nullable=1 is_null=0 */ ### @3='2018-01-02 09:05:07' /* DATETIME(0) meta=0 nullable=1 is_null=0 */ ### @4='1323306' /* VARSTRING(30) meta=30 nullable=1 is_null=0 */ # at 1042355 #180102 9:05:07 server id 1323306 end_log_pos 1042441 CRC32 0x67285443 Query thread_id=9 exec_time=0 error_code=0 SET TIMESTAMP=1514855107/*!*/; COMMIT /*!*/; # at 1042441 ==================== mysqldump備份文件獲取的binlog pos ==================== ==================== 備份文件還原後，從庫GTID_PURGED位置 ==================== #180102 9:05:07 server id 1323306 end_log_pos 1042506 CRC32 0xf77ede80 GTID last_committed=2918 sequence_number=2919 rbr_only=yes /*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/; SET @@SESSION.GTID_NEXT= '8ab82362-9c37-11e7-a858-000c29c1025c:251874'/*!*/; # at 1042506 #180102 9:05:07 server id 1323306 end_log_pos 1042591 CRC32 0x506a2875 Query thread_id=9 exec_time=0 error_code=0 SET TIMESTAMP=1514855107/*!*/; BEGIN /*!*/; # at 1042591 #180102 9:05:07 server id 1323306 end_log_pos 1042661 CRC32 0x90b154e8 Table_map: `replcrash`.`py_user_innodb` mapped to number 255 # at 1042661 #180102 9:05:07 server id 1323306 end_log_pos 1042737 CRC32 0x1d693238 Write_rows: table id 255 flags: STMT_END_F ### INSERT INTO `replcrash`.`py_user_innodb` ### SET ### @1=331 /* INT meta=0 nullable=0 is_null=0 */ ### @2='7ATZSNFNIBW5DZNMNZYBMV' /* VARSTRING(96) meta=96 nullable=1 is_null=0 */ ### @3='2018-01-02 09:05:07' /* DATETIME(0) meta=0 nullable=1 is_null=0 */ ### @4='1323306' /* VARSTRING(30) meta=30 nullable=1 is_null=0 */ # at 1042737 #180102 9:05:07 server id 1323306 end_log_pos 1042768 CRC32 0x87864022 Xid = 4420 COMMIT/*!*/; # at 1042768 ==================== 啓動複製後，py_user_innodb寫入uid=331記錄，成功 ==================== #180102 9:05:07 server id 1323306 end_log_pos 1042833 CRC32 0xe492578a GTID last_committed=2919 sequence_number=2920 rbr_only=yes /*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/; SET @@SESSION.GTID_NEXT= '8ab82362-9c37-11e7-a858-000c29c1025c:251875'/*!*/; # at 1042833 #180102 9:05:07 server id 1323306 end_log_pos 1042918 CRC32 0xf08c4165 Query thread_id=9 exec_time=0 error_code=0 SET TIMESTAMP=1514855107/*!*/; BEGIN /*!*/; # at 1042918 #180102 9:05:07 server id 1323306 end_log_pos 1042988 CRC32 0xf16731d6 Table_map: `replcrash`.`py_user_myisam` mapped to number 257 # at 1042988 #180102 9:05:07 server id 1323306 end_log_pos 1043062 CRC32 0x128aec5e Write_rows: table id 257 flags: STMT_END_F ### INSERT INTO `replcrash`.`py_user_myisam` ### SET ### @1=332 /* INT meta=0 nullable=0 is_null=0 */ ### @2='X1LME9HO5V7WXNOKBVZE' /* VARSTRING(96) meta=96 nullable=1 is_null=0 */ ### @3='2018-01-02 09:05:07' /* DATETIME(0) meta=0 nullable=1 is_null=0 */ ### @4='1323306' /* VARSTRING(30) meta=30 nullable=1 is_null=0 */ # at 1043062 #180102 9:05:07 server id 1323306 end_log_pos 1043148 CRC32 0x5cc8cc30 Query thread_id=9 exec_time=0 error_code=0 SET TIMESTAMP=1514855107/*!*/; COMMIT /*!*/; # at 1043148 ==================== 啓動複製後，py_user_myisam寫入uid=332記錄，失敗 ==================== ==================== 從庫py_user_myisam表已存在332記錄，sql_thread中止 ==================== #180102 9:05:07 server id 1323306 end_log_pos 1043213 CRC32 0xceb1ce4d GTID last_committed=2920 sequence_number=2921 rbr_only=yes /*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/; SET @@SESSION.GTID_NEXT= '8ab82362-9c37-11e7-a858-000c29c1025c:251876'/*!*/; # at 1043213 #180102 9:05:07 server id 1323306 end_log_pos 1043298 CRC32 0x38591b71 Query thread_id=9 exec_time=0 error_code=0 SET TIMESTAMP=1514855107/*!*/; BEGIN /*!*/; # at 1043298 #180102 9:05:07 server id 1323306 end_log_pos 1043368 CRC32 0xf80c2ae9 Table_map: `replcrash`.`py_user_innodb` mapped to number 258 # at 1043368 #180102 9:05:07 server id 1323306 end_log_pos 1043442 CRC32 0x0bf4ae26 Write_rows: table id 258 flags: STMT_END_F ### INSERT INTO `replcrash`.`py_user_innodb` ### SET ### @1=332 /* INT meta=0 nullable=0 is_null=0 */ ### @2='X1LME9HO5V7WXNOKBVZE' /* VARSTRING(96) meta=96 nullable=1 is_null=0 */ ### @3='2018-01-02 09:05:07' /* DATETIME(0) meta=0 nullable=1 is_null=0 */ ### @4='1323306' /* VARSTRING(30) meta=30 nullable=1 is_null=0 */ # at 1043442 #180102 9:05:07 server id 1323306 end_log_pos 1043473 CRC32 0x1b75f9e0 Xid = 4431 COMMIT/*!*/; # at 1043473 ...
二進志日誌顯示，按照py_user_myisam->py_user_innodb->py_user_myisam這樣的順序往表中寫入數據。
使用備份文件搭建的從庫，py_user_myisam表已包含UNLOCK TABLES以後的9條數據，但在備份文件中的@@GLOBAL.GTID_PURGED倒是UNLOCK TABLES時刻的位置。所以在啓動複製後，第一條操做py_user_innodb表成功，第二條操做py_user_myisam表失敗，從庫報主鍵衝突錯誤
那麼應該如何修復這個錯誤呢?
# 刪除從庫py_user_myisam表大於等於衝突key的記錄 mydba@192.168.85.132,3308 [replcrash]> delete from py_user_myisam where uid>=332; # 從新啓動sql_thread mydba@192.168.85.132,3308 [replcrash]> start slave sql_thread;
總得來講就是隻有innodb纔會提供一致性備份！！！
1.二、DDL操做對備份的影響
建議先閱讀後續章節，理解各備份過程後再返回閱讀DDL操做對備份的影響
運行下面的腳本持續DDL操做
#!/user/bin/python import string import random import MySQLdb import time conn = MySQLdb.connect(host='192.168.85.132', port=3306, user='mydba', passwd='mysql5719', db='replcrash') """ create table py_user( uid int not null auto_increment, name varchar(32), add_time datetime default current_timestamp, server_id varchar(10), primary key(uid), key(name) ); """ counter = 1 while counter<=100: addcol = 'col' + str(counter) print addcol cursor = conn.cursor() #DDL cursor.execute("alter table py_user_innodb add %s int;" % addcol) cursor.execute("alter table py_user_innodb drop column %s;" % addcol) conn.commit() counter += 1 time.sleep(0.0001)
邏輯備份數據庫
# 清空general_log [root@ZST1 logs]# cat /dev/null > /data/mysql/mysql3306/data/mysql-general.log # mysqldump [root@ZST1 backup]# mysqldump -h127.0.0.1 -P3306 -uroot -p --single-transaction --master-data=2 replcrash >/data/backup/replcrash_dump_1323306_`date +%Y%m%d`.sql Enter password: Warning: A partial dump from a server that has GTIDs will by default include the GTIDs of all transactions, even those that changed suppressed parts of the database. If you don not want to restore GTIDs, pass --set-gtid-purged=OFF. To make a complete dump, pass --all-databases --triggers --routines --events. mysqldump: Error 1412: Table definition has changed, please retry transaction when dumping table `py_user_innodb` at row: 0 # mysqlpump [root@ZST1 backup]# mysqlpump -h127.0.0.1 -P3306 -uroot -p --single-transaction --add-drop-table --exclude-databases=mysql,sakila,backupdb -A >/data/backup/replcrash_pump_1323306_`date +%Y%m%d`.sql Enter password: mysqlpump: [ERROR] (1412) Table definition has changed, please retry transaction Dump process encountered error and will not continue. Dump progress: 0/3 tables, 250/2431 rows [root@ZST1 backup]# # mydumper [root@ZST1 mydumper]# mydumper -h 127.0.0.1 -P 3306 -u root -p mysql5719 --trx-consistency-only -v 3 -t 2 -o /data/backup/mydumper .. ** Message: Thread 2 dumping data for `replcrash`.`py_user_innodb` ** Message: Thread 1 dumping data for `replcrash`.`py_user_myisam` ** (mydumper:5179): CRITICAL **: Could not read data from replcrash.py_user_innodb: Table definition has changed, please retry transaction ** Message: Empty table replcrash.py_user_innodb ...
mydumper須要使用-v 3 顯示詳細信息，自己是不會報錯的！！！若是想更容易再現錯誤，最好是備份的數據表較大、較多，適當下降並行線程數(-t)，開啓--trx-consistency-only，讓其儘早解鎖
生成的備份文件中，py_user_innodb表只有結構，沒有數據
根據備份邏輯，在UNLOCK TABLES解除FTWRL到SELECT /*!40001 SQL_NO_CACHE */ * FROM `tbname`之間若是有DDL操做，就會形成上述錯誤
使用了with consistent snapshot子句開啓一致性快照事務以後，若是一旦表結構定義發生改變，事務將沒法對該表執行查詢
| 使用WITH CONSISTENT SNAPSHOT子句，會話1顯式開啓一個事務以後先不執行查詢，會話B使用DDL語句添加一個字段 | |
| 會話1 | 會話2 |
| 修改隔離級別爲RR |
修改隔離級別爲RR |
| 顯式開啓一個事務，先不執行查詢 |
|
| 執行DDL語句添加字段，執行成功 |
|
| 執行查詢，報表定義已經改變的錯誤 |
|
物理備份數據庫
[root@ZST1 backup]# innobackupex --defaults-file=/data/mysql/mysql3306/my.cnf -S /tmp/mysql3306.sock -uroot -pmysql5719 /data/backup/full/ 180104 15:29:31 [01] Copying ./sakila/payment.ibd to /data/backup/full/2018-01-04_15-29-21/sakila/payment.ibd 180104 15:29:32 [01] ...done InnoDB: Last flushed lsn: 19337348200 load_index lsn 19337354022 InnoDB: An optimized (without redo logging) DDLoperation has been performed. All modified pages may not have been flushed to the disk yet. PXB will not be able take a consistent backup. Retry the backup operation 180104 15:29:32 [01] Copying ./replcrash/py_user.ibd to /data/backup/full/2018-01-04_15-29-21/replcrash/py_user.ibd 180104 15:29:32 [01] ...done 180104 15:29:32 [01] Copying ./replcrash/py_user_innodb.ibd to /data/backup/full/2018-01-04_15-29-21/replcrash/py_user_innodb.ibd 180104 15:29:32 [01] ...done
只要在備份期間(實際是備份InnoDB表期間，由於備份non-InnoDB表期間會加FTWRL只讀鎖，阻塞DDL、DML操做)執行DDL操做，innobackupex就會報錯退出。緣由就是DDL操做不會記錄到redo log，PXB will not be able take a consistent backup.
所以備份期間要避免執行不記錄事務日誌的操做(ALTER TABLE, CREATE TABLE, DROP TABLE, RENAME TABLE, TRUNCATE TABLE)
2、mysqlpump
2.一、備份過程
藉助前面已開啓general_log，來查看mysqlpump執行過程
MySQL 5.7.11起解決了--single-transaction和--default-parallelism的互斥問題
# 清空general_log [root@ZST1 logs]# cat /dev/null > /data/mysql/mysql3306/data/mysql-general.log # 備份replcrash數據庫，一致性備份(源碼限制只有備份整個實例時才能返回GTID信息，所以這裏使用-A，而後使用--exclude-databases排除不須要備份的db) [root@ZST1 backup]# mysqlpump -h127.0.0.1 -P3306 -uroot -p --single-transaction --add-drop-table --exclude-databases=mysql,sakila,backupdb -A >/data/backup/replcrash_pump_1323306_`date +%Y%m%d`.sql 默認mysqlpump使用一個隊列兩個線程 [root@ZST1 data]# vim /data/mysql/mysql3306/data/mysql-general.log ... 4 Time Id Command Argument 5 2018-01-03T01:24:06.623704Z 14 Connect root@localhost on using TCP/IP 6 2018-01-03T01:24:06.631266Z 14 Query FLUSH TABLES WITH READ LOCK 7 2018-01-03T01:24:06.667093Z 14 Query SHOW WARNINGS 8 2018-01-03T01:24:06.667310Z 14 Query SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ 9 2018-01-03T01:24:06.667520Z 14 Query SHOW WARNINGS 10 2018-01-03T01:24:06.667647Z 14 Query START TRANSACTION WITH CONSISTENT SNAPSHOT 11 2018-01-03T01:24:06.667792Z 14 Query SHOW WARNINGS 12 2018-01-03T01:24:06.679491Z 15 Connect root@localhost on using TCP/IP 13 2018-01-03T01:24:06.683019Z 15 Query SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ 14 2018-01-03T01:24:06.684865Z 15 Query SHOW WARNINGS 15 2018-01-03T01:24:06.685015Z 15 Query START TRANSACTION WITH CONSISTENT SNAPSHOT 16 2018-01-03T01:24:06.685114Z 15 Query SHOW WARNINGS 17 2018-01-03T01:24:06.686057Z 16 Connect root@localhost on using TCP/IP 18 2018-01-03T01:24:06.688856Z 16 Query SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ 19 2018-01-03T01:24:06.716231Z 16 Query SHOW WARNINGS 20 2018-01-03T01:24:06.716447Z 16 Query START TRANSACTION WITH CONSISTENT SNAPSHOT 21 2018-01-03T01:24:06.716558Z 16 Query SHOW WARNINGS 22 2018-01-03T01:24:06.716701Z 14 Query UNLOCK TABLES 23 2018-01-03T01:24:06.716857Z 14 Query SHOW WARNINGS ...各類SHOW... 1176 2018-01-03T01:24:07.344380Z 16 Query SHOW CREATE DATABASE IF NOT EXISTS `replcrash` 1177 2018-01-03T01:24:07.344468Z 16 Query SHOW WARNINGS 1178 2018-01-03T01:24:07.344565Z 16 Query SHOW TABLE STATUS FROM `replcrash` 1179 2018-01-03T01:24:07.380209Z 16 Query SHOW WARNINGS 1180 2018-01-03T01:24:07.380416Z 16 Query SHOW COLUMNS IN `py_user` FROM `replcrash` 1181 2018-01-03T01:24:07.381223Z 16 Query SHOW WARNINGS 1182 2018-01-03T01:24:07.381408Z 16 Query SHOW CREATE TABLE `replcrash`.`py_user` 1183 2018-01-03T01:24:07.381614Z 16 Query SHOW WARNINGS 1184 2018-01-03T01:24:07.381950Z 16 Query SHOW TRIGGERS FROM `replcrash` LIKE 'py_user' 1185 2018-01-03T01:24:07.382575Z 16 Query SHOW WARNINGS 1186 2018-01-03T01:24:07.382764Z 16 Query SHOW COLUMNS IN `py_user_innodb` FROM `replcrash` 1187 2018-01-03T01:24:07.383125Z 14 Query SET SQL_QUOTE_SHOW_CREATE= 1 1188 2018-01-03T01:24:07.383334Z 14 Query SHOW WARNINGS 1189 2018-01-03T01:24:07.383617Z 14 Query SET TIME_ZONE='+00:00' 1190 2018-01-03T01:24:07.384037Z 14 Query SHOW WARNINGS 1191 2018-01-03T01:24:07.385106Z 15 Query SELECT `COLUMN_NAME`, `EXTRA` FROM `INFORMATION_SCHEMA`.`COLUMNS`WHERE TABLE_SCHEMA ='replcrash' AND TABLE_NAME ='py_user' 1192 2018-01-03T01:24:07.386099Z 15 Query SHOW WARNINGS 1193 2018-01-03T01:24:07.386347Z 15 Query SELECT SQL_NO_CACHE `uid`,`name`,`add_time`,`server_id` FROM `replcrash`.`py_user` 1194 2018-01-03T01:24:07.387102Z 15 Query SHOW WARNINGS 1195 2018-01-03T01:24:07.387644Z 16 Query SHOW WARNINGS 1196 2018-01-03T01:24:07.387997Z 16 Query SHOW CREATE TABLE `replcrash`.`py_user_innodb` 1197 2018-01-03T01:24:07.388216Z 16 Query SHOW WARNINGS 1198 2018-01-03T01:24:07.388487Z 16 Query SHOW TRIGGERS FROM `replcrash` LIKE 'py_user_innodb' 1199 2018-01-03T01:24:07.389053Z 14 Query SELECT `COLUMN_NAME`, `EXTRA` FROM `INFORMATION_SCHEMA`.`COLUMNS`WHERE TABLE_SCHEMA ='replcrash' AND TABLE_NAME ='py_user_innodb' 1200 2018-01-03T01:24:07.390054Z 14 Query SHOW WARNINGS 1201 2018-01-03T01:24:07.390293Z 14 Query SELECT SQL_NO_CACHE `uid`,`name`,`add_time`,`server_id` FROM `replcrash`.`py_user_innodb` 1202 2018-01-03T01:24:07.391566Z 16 Query SHOW WARNINGS 1203 2018-01-03T01:24:07.391776Z 16 Query SHOW COLUMNS IN `py_user_myisam` FROM `replcrash` 1204 2018-01-03T01:24:07.392559Z 16 Query SHOW WARNINGS 1205 2018-01-03T01:24:07.392747Z 16 Query SHOW CREATE TABLE `replcrash`.`py_user_myisam` 1206 2018-01-03T01:24:07.393065Z 16 Query SHOW WARNINGS 1207 2018-01-03T01:24:07.393336Z 16 Query SHOW TRIGGERS FROM `replcrash` LIKE 'py_user_myisam' 1208 2018-01-03T01:24:07.394146Z 16 Query SHOW WARNINGS 1209 2018-01-03T01:24:07.394371Z 16 Query SHOW FUNCTION STATUS WHERE db = 'replcrash' 1210 2018-01-03T01:24:07.396083Z 15 Query SELECT `COLUMN_NAME`, `EXTRA` FROM `INFORMATION_SCHEMA`.`COLUMNS`WHERE TABLE_SCHEMA ='replcrash' AND TABLE_NAME ='py_user_myisam' 1211 2018-01-03T01:24:07.399053Z 15 Query SHOW WARNINGS 1212 2018-01-03T01:24:07.399425Z 15 Query SELECT SQL_NO_CACHE `uid`,`name`,`add_time`,`server_id` FROM `replcrash`.`py_user_myisam` 1213 2018-01-03T01:24:07.405719Z 16 Query SHOW WARNINGS 1214 2018-01-03T01:24:07.405915Z 16 Query SHOW PROCEDURE STATUS WHERE db = 'replcrash' 1215 2018-01-03T01:24:07.412340Z 14 Query SHOW WARNINGS 1216 2018-01-03T01:24:07.414496Z 15 Query SHOW WARNINGS 1217 2018-01-03T01:24:07.416755Z 16 Query SHOW WARNINGS 1218 2018-01-03T01:24:07.417261Z 16 Query SHOW EVENTS FROM `replcrash` 1219 2018-01-03T01:24:07.417884Z 16 Query SHOW WARNINGS
即便備份一個db，general_log中也會出現不少其餘庫的內容(⊙_⊙)
mysqlpump備份過程(--single-transaction)：
對於創建的第一個鏈接，執行FLUSH TABLES WITH READ LOCK，加上只讀鎖；對於其餘鏈接(包含第一個鏈接)，設置RR隔離級別，並開啓一致性快照讀START TRANSACTION WITH CONSISTENT SNAPSHOT；當全部鏈接(--default-parallelism設置多少個線程)都創建好後，再執行解鎖UNLOCK TABLES；最後經過SELECT colname FROM `tbname`備份數據
general_log中沒有看到SHOW MASTER STATUS，猜想它相似於mydumper，也是在主線程FLUSH TABLES WITH READ LOCK後，就去獲取GTID信息。
所以mysqlpump也會遇到在mysqldump中的坑
mysqlpump並行備份的部分參數須要在單線程模式才能應用，實際使用中建議先測試，對比輸出結果分辨各參數的做用
2.二、--single-transaction
• --single-transaction
This option sets the transaction isolation mode to REPEATABLE READ and sends a START TRANSACTION SQL statement to the server before dumping data. It is useful only with transactional tables such as InnoDB, because then it dumps the consistent state of the database at the time when START TRANSACTION was issued without blocking any applications.
When using this option, you should keep in mind that only InnoDB tables are dumped in a consistent state. For example, any MyISAM or MEMORY tables dumped while using this option may still change state.
While a --single-transaction dump is in process, to ensure a valid dump file (correct table contents and binary log coordinates), no other connection should use the following statements: ALTER TABLE, CREATE TABLE, DROP TABLE, RENAME TABLE, TRUNCATE TABLE. A consistent read is not isolated from those statements, so use of them on a table to be dumped can cause the SELECT that is performed by mysqldump/mysqlpump to retrieve the table contents to obtain incorrect contents or fail.
• START TRANSACTION
The WITH CONSISTENT SNAPSHOT modifier starts a consistent read for storage engines that are capable of it. This applies only to InnoDB. The effect is the same as issuing a START TRANSACTION followed by a SELECT from any InnoDB table.
The WITH CONSISTENT SNAPSHOT modifier does not change the current transaction isolation level, so it provides a consistent snapshot only if the current isolation level is one that permits a consistent read. The only isolation level that permits a consistent read is REPEATABLE READ.
該參數將事務隔離級別設置成Repeatable Read，並在dump以前發送start transaction語句給服務端。這隻對事務表(好比innodb)頗有用，由於在發出start transaction時，保證了在不阻塞任何應用下的一致性狀態。對myisam和memory等非事務表，仍是會改變狀態的，當使用此參數的時候要確保沒有其餘鏈接在使用ALTER TABLE、CREATE TABLE、DROP TABLE、RENAME TABLE、TRUNCATE TABLE等語句，不然會出現不正確的內容或者失敗。在mysql5.7.11以前，--default-parallelism大於1的時候和此參數互斥，必須使用--default-parallelism=0。5.7.11以後解決了--single-transaction和--default-parallelism的互斥問題
3、mydumper
3.一、備份過程
mydumper備份過程
一、鏈接目標數據庫
二、經過show processlist來判斷是否有長查詢，根據參數long-query-guard和kill-long-queries決定退出或殺掉長查詢
三、主線程flush tables with read lock；start transaction with consistent snapshot
四、主線程讀取當前時間點的二進制日誌文件名和日誌寫入的位置並記錄在metadata文件中，以供即時點恢復使用
五、建立dump線程，缺省爲4個
六、肯定候選表，根據類別分別插入non_innodb_table、innodb_tables以及table_schemas鏈表
七、將候選表經過g_async_queue_push加入任務隊列(隊列最後元素是thread shutdown)，由dump線程從隊列中讀取表信息並執行數據導出
八、備份完non_innodb_table後當即unlock tables解鎖，以減小鎖定時間
九、等待dump InnoDB tables完成
從備份邏輯中能夠看出，mydumper須要在備份完非事務表以後才解鎖，這就保證了事務表和非事務表的一致性備份。默認狀況下，DML操做不會影響mydumper備份的一致性
3.二、--trx-consistency-only
這個參數退化爲僅保證事務表的一致性備份，在備份非事務表前若是有數據寫入就會致使數據與metadata不一致，有點相似前面的mysqldump、mysqlpump的味道
運行前面的PythonDML腳本，一直往py_user_myisam、py_user_innodb表中寫入數據，測試--trx-consistency-only選項的效果
# 清空general_log [root@ZST1 logs]# cat /dev/null > /data/mysql/mysql3306/data/mysql-general.log # 備份replcrash庫(僅事務表一致性備份) [root@ZST1 mydumper]# mydumper -h 127.0.0.1 -P 3306 -u root -p mysql5719 --trx-consistency-only -v 3 -B replcrash -o /data/backup/mydumper ** (mydumper:2808): WARNING **: Using trx_consistency_only, binlog coordinates will not be accurate if you are writing to non transactional tables. ** Message: Connected to a MySQL server ** Message: Started dump at: 2018-01-04 09:34:53 ** Message: Written master status ** Message: Thread 1 connected using MySQL connection ID 50 ** Message: Thread 2 connected using MySQL connection ID 51 ** Message: Thread 3 connected using MySQL connection ID 52 ** Message: Thread 4 connected using MySQL connection ID 53 ** Message: Transactions started, unlocking tables ** Message: Thread 1 dumping data for `replcrash`.`py_user` ** Message: Thread 2 dumping data for `replcrash`.`py_user_innodb` ** Message: Thread 2 dumping data for `replcrash`.`py_user_myisam` ** Message: Thread 1 dumping schema for `replcrash`.`py_user` ** Message: Thread 3 dumping schema for `replcrash`.`py_user_innodb` ** Message: Thread 1 dumping schema for `replcrash`.`py_user_myisam` ** Message: Thread 3 shutting down ** Message: Thread 1 shutting down ** Message: Thread 2 shutting down ** Message: Thread 4 shutting down ** Message: Finished dump at: 2018-01-04 09:34:53 # 查看metadata [root@ZST1 mydumper]# cat metadata Started dump at: 2018-01-04 09:34:53 SHOW MASTER STATUS: Log: mysql-bin.000185 Pos: 2452139 GTID:8ab82362-9c37-11e7-a858-000c29c1025c:1-272303 Finished dump at: 2018-01-04 09:34:53 [root@ZST1 mydumper]# # 還原replcrash庫 [root@ZST1 mydumper]# myloader -h 127.0.0.1 -P 3308 -u root -p mysql5719 -o -B replcrash -d /data/backup/mydumper myloader不會產生binlog，也不會應用metadata中的GTID # 設置GTID_PURGED mydba@192.168.85.132,3308 [replcrash]> reset master; mydba@192.168.85.132,3308 [replcrash]> SET @@GLOBAL.GTID_PURGED='8ab82362-9c37-11e7-a858-000c29c1025c:1-272303'; # 搭建複製 mydba@192.168.85.132,3308 [replcrash]> change master to master_host='192.168.85.132', master_port=3306, master_user='repl', master_password='repl', master_auto_position=1; # 啓動複製，查看複製狀態 mydba@192.168.85.132,3308 [replcrash]> start slave; mydba@192.168.85.132,3308 [replcrash]> show slave status\G *************************** 1. row *************************** Master_Log_File: mysql-bin.000185 Read_Master_Log_Pos: 2758886 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 414 Relay_Master_Log_File: mysql-bin.000185 Slave_IO_Running: Yes Slave_SQL_Running: No Exec_Master_Log_Pos: 2452139 Last_SQL_Errno: 1062 Last_SQL_Error: Could not execute Write_rows event on table replcrash.py_user_myisam; Duplicate entry '783' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event''s master log mysql-bin.000185, end_log_pos 2452443 Retrieved_Gtid_Set: 8ab82362-9c37-11e7-a858-000c29c1025c:272304-273161 Executed_Gtid_Set: 8ab82362-9c37-11e7-a858-000c29c1025c:1-272303 Auto_Position: 1 # 大於等於衝突key的數據 mydba@192.168.85.132,3308 [replcrash]> select * from replcrash.py_user_myisam where uid>=783; +-----+--------------------------------+---------------------+-----------+ | uid | name | add_time | server_id | +-----+--------------------------------+---------------------+-----------+ | 783 | MLGU22VB26RHNNYAY6IPPUJX9A74EM | 2018-01-04 09:34:53 | 1323306 | +-----+--------------------------------+---------------------+-----------+ 1 row in set (0.06 sec) # mydumper產生的general-log [root@ZST1 data]# vim /data/mysql/mysql3306/data/mysql-general.log ... 2018-01-04T01:34:53.769877Z 49 Query SET SESSION net_write_timeout = 2147483 2018-01-04T01:34:53.770302Z 49 Query SHOW PROCESSLIST 2018-01-04T01:34:53.770886Z 49 Query FLUSH TABLES WITH READ LOCK 2018-01-04T01:34:53.771236Z 49 Query START TRANSACTION /*!40108 WITH CONSISTENT SNAPSHOT */ 2018-01-04T01:34:53.771430Z 49 Query /*!40101 SET NAMES binary*/ 2018-01-04T01:34:53.772084Z 49 Query SHOW MASTER STATUS 2018-01-04T01:34:53.772331Z 49 Query SHOW SLAVE STATUS 2018-01-04T01:34:53.773386Z 48 Query insert into py_user_myisam(name,add_time,server_id) values('MLGU22VB26RHNNYAY6IPPUJX9A74EM',now(),@@server_id) 2018-01-04T01:34:53.773981Z 50 Connect root@localhost on using TCP/IP 2018-01-04T01:34:53.788273Z 50 Query SET SESSION wait_timeout = 2147483 2018-01-04T01:34:53.788874Z 50 Query SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ 2018-01-04T01:34:53.788972Z 50 Query START TRANSACTION /*!40108 WITH CONSISTENT SNAPSHOT */ 2018-01-04T01:34:53.789092Z 50 Query /*!40103 SET TIME_ZONE='+00:00' */ 2018-01-04T01:34:53.789201Z 50 Query /*!40101 SET NAMES binary*/ 2018-01-04T01:34:53.790091Z 51 Connect root@localhost on using TCP/IP 2018-01-04T01:34:53.790315Z 51 Query SET SESSION wait_timeout = 2147483 2018-01-04T01:34:53.791912Z 51 Query SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ 2018-01-04T01:34:53.792095Z 51 Query START TRANSACTION /*!40108 WITH CONSISTENT SNAPSHOT */ 2018-01-04T01:34:53.792307Z 51 Query /*!40103 SET TIME_ZONE='+00:00' */ 2018-01-04T01:34:53.792418Z 51 Query /*!40101 SET NAMES binary*/ 2018-01-04T01:34:53.793381Z 52 Connect root@localhost on using TCP/IP 2018-01-04T01:34:53.794631Z 52 Query SET SESSION wait_timeout = 2147483 2018-01-04T01:34:53.796301Z 52 Query SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ 2018-01-04T01:34:53.796404Z 52 Query START TRANSACTION /*!40108 WITH CONSISTENT SNAPSHOT */ 2018-01-04T01:34:53.796538Z 52 Query /*!40103 SET TIME_ZONE='+00:00' */ 2018-01-04T01:34:53.796641Z 52 Query /*!40101 SET NAMES binary*/ 2018-01-04T01:34:53.797414Z 53 Connect root@localhost on using TCP/IP 2018-01-04T01:34:53.798420Z 53 Query SET SESSION wait_timeout = 2147483 2018-01-04T01:34:53.799316Z 53 Query SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ 2018-01-04T01:34:53.799425Z 53 Query START TRANSACTION /*!40108 WITH CONSISTENT SNAPSHOT */ 2018-01-04T01:34:53.800345Z 53 Query /*!40103 SET TIME_ZONE='+00:00' */ 2018-01-04T01:34:53.800449Z 53 Query /*!40101 SET NAMES binary*/ 2018-01-04T01:34:53.800661Z 49 Query UNLOCK TABLES /* trx-only */ 2018-01-04T01:34:53.801868Z 49 Init DB replcrash 2018-01-04T01:34:53.802332Z 49 Query SHOW TABLE STATUS 2018-01-04T01:34:53.804759Z 50 Query SELECT /*!40001 SQL_NO_CACHE */ * FROM `replcrash`.`py_user` 2018-01-04T01:34:53.805137Z 51 Query SELECT /*!40001 SQL_NO_CACHE */ * FROM `replcrash`.`py_user_innodb` 2018-01-04T01:34:53.807164Z 51 Query SELECT /*!40001 SQL_NO_CACHE */ * FROM `replcrash`.`py_user_myisam` 2018-01-04T01:34:53.808786Z 48 Query insert into py_user_innodb(name,add_time,server_id) values('MLGU22VB26RHNNYAY6IPPUJX9A74EM',now(),@@server_id) 2018-01-04T01:34:53.809449Z 49 Query SHOW CREATE DATABASE `replcrash` 2018-01-04T01:34:53.810326Z 52 Query SHOW CREATE TABLE `replcrash`.`py_user_innodb` 2018-01-04T01:34:53.810638Z 50 Query SHOW CREATE TABLE `replcrash`.`py_user` 2018-01-04T01:34:53.811307Z 52 Quit 2018-01-04T01:34:53.811506Z 50 Query SHOW CREATE TABLE `replcrash`.`py_user_myisam` 2018-01-04T01:34:53.812272Z 50 Quit 2018-01-04T01:34:53.812512Z 48 Query commit
48是python持續寫入的線程、49是主線程、50~53是dump線程
全部鏈接創建後，主線程當即解鎖UNLOCK TABLES /* trx-only */。主線程SHOW MASTER STATUS的後面緊接着python寫入nsert into py_user_myisam，實際這個寫入操做應該是在主線程UNLOCK TABLES以後，dump線程備份py_user_myisam數據以前。這就致使備份的數據與metadata不一致，所以不推薦使用--trx-consistency-only選項
3.三、邏輯備份差別
mysqldump：只支持單線程工做，這就使得它沒法迅速的備份數據
mysqlpump：並行的最小粒度是單個數據庫對象，對於每張表的導出只能是單個線程的
mydumper：支持對單表多個線程備份，參數-r
多線程操做提高空間受限於磁盤的IO能力，在使用前作好磁盤IO的評估
4、XtraBackup
4.一、備份過程
innobackupex全備過程
一、start xtrabackup_log
二、copy .ibd、ibdata1
三、FLUSH TABLES WITH READ LOCK
四、copy .frm、.MYD、.MYI、misc files
五、SHOW MASTER STATUS
六、UNLOCK TABLES
七、stop and copy xtrabackup_log
備份開始時首先會開啓一個後臺檢測進程，從當前checkpoint位置開始拷貝redo log，同時持續檢測redo log，一旦發現redo中有新的日誌寫入，馬上將日誌記入後臺日誌文件xtrabackup_log中。以後拷貝innodb的數據文件和系統表空間文件ibdata1，待拷貝結束後，執行flush tables with read lock操做，拷貝.frm，MYI，MYD，等文件，而且在這一時刻得到binlog的位置，最後會發出unlock tables，把表設置爲可讀可寫狀態，最終中止xtrabackup_log。
InnoDB表copy [.ibd、ibdata]+[redo log]藉助InnoDB Crash Recovery機制保證一致性；non-InnoDB表使用FTWRL加只讀鎖後backup non-InnoDB tables and files。DML操做不會影響innobackupex備份的一致性
4.二、總結
| mysqldump | mysqlpump | mydumper | innobackupex | |
| FTWRL被阻塞 | 存在互斥鎖，會被阻塞 | 存在互斥鎖，會被阻塞 | 存在互斥鎖，會被阻塞 | 存在互斥鎖，會被阻塞 |
| DDL操做致使備份異常 | 報錯，表定義變動 | 報錯，表定義變動 | 不報錯，-v 3查看 | 報錯，DDL操做沒寫redo log |
| DML操做影響數據一致性 | non_InnoDB表不一致 | non_InnoDB表不一致 | 默認一致，--trx-consistency-only會致使不一致 | 一致 |
| 其餘 | 沒有指定my.cnf致使備份一個錯誤的實例 |
對生產庫的DDL操做、大事務、或者長時間鎖表的操做，必定要避開備份時間(･ω･)
5、參考文檔
mysqldump與innobackupex備份過程你知多少：http://www.cnblogs.com/xiaoboluo768/p/7560105.html
mysqlpump兩個疑問點：http://www.fordba.com/mysqlpump_some_questions.html
MySQL 5.7 mysqlpump 備份工具說明：http://www.cnblogs.com/zhoujinyi/p/5684903.html
mysqlpump — A Database Backup Program：https://dev.mysql.com/doc/refman/5.7/en/mysqlpump.html
mydumper：https://github.com/maxbube/mydumper
MySQL備份之【mydumper 學習】：http://www.cnblogs.com/zhoujinyi/p/3423641.html
mydumper備份數據庫詳解(已詳細說明)：https://yq.aliyun.com/articles/45741
mydumper備份原理和使用方法：http://www.cnblogs.com/linuxnote/p/3817698.html
相關文章
- 1. MySQL備份可能遇到的坑
- 2. mysql遇到的坑
- 3. 使用Assetbundle時可能遇到的坑
- 4. Go語言可能會遇到的坑
- 5. MHA-Failover可能遇到的坑
- 6. MySQL安裝遇到的坑
- 7. mysql安裝遇到的坑
- 8. 搭建Mysql遇到的坑
- 9. 安裝mysql 遇到的坑
- 10. Hooks 的性能優化及可能會遇到的坑總結
- 更多相關文章...
- • MySQL數據庫備份（INTO OUTFILE） - MySQL教程
- • Redis的兩種備份（持久化）方式：RDB和AOF - Redis教程
- • 漫談MySQL的鎖機制
- • Git可視化極簡易教程 — Git GUI使用方法
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
歡迎關注本站公眾號,獲取更多信息
相關文章
- 1. MySQL備份可能遇到的坑
- 2. mysql遇到的坑
- 3. 使用Assetbundle時可能遇到的坑
- 4. Go語言可能會遇到的坑
- 5. MHA-Failover可能遇到的坑
- 6. MySQL安裝遇到的坑
- 7. mysql安裝遇到的坑
- 8. 搭建Mysql遇到的坑
- 9. 安裝mysql 遇到的坑
- 10. Hooks 的性能優化及可能會遇到的坑總結