爲何還原innobackupex備份後查看到的Executed_Gtid_Set與xtrabackup_binlog_info不一致

基本環境：官方社區版MySQL 5.7.19，innobackupex version 2.4.8

1、什麼不一致

1.一、不一致

首先使用下面腳原本構建Executed_Gtid_Set與xtrabackup_binlog_info不一致，到底指的是什麼不一致

1、準備測試數據
# 切換日誌
mydba@192.168.85.132,3306 [replcrash]> flush binary logs;
Query OK, 0 rows affected (0.07 sec)
# 查看當前位置
mydba@192.168.85.132,3306 [replcrash]> show master status;
+------------------+----------+--------------+------------------+-----------------------------------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                             |
+------------------+----------+--------------+------------------+-----------------------------------------------+
| mysql-bin.000191 |      194 |              |                  | 8ab82362-9c37-11e7-a858-000c29c1025c:1-507450 |
+------------------+----------+--------------+------------------+-----------------------------------------------+
1 row in set (0.00 sec)
# 清空數據
mydba@192.168.85.132,3306 [replcrash]> truncate table py_user;
Query OK, 0 rows affected (0.01 sec)
mydba@192.168.85.132,3306 [replcrash]> show create table py_user;
+---------+---------------------------------------------------------+
| Table   | Create Table                                            |
+---------+---------------------------------------------------------+
| py_user | CREATE TABLE `py_user` (
  `uid` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(32) DEFAULT NULL,
  `add_time` datetime DEFAULT CURRENT_TIMESTAMP,
  `server_id` varchar(10) DEFAULT NULL,
  PRIMARY KEY (`uid`),
  KEY `name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
+---------+---------------------------------------------------------+
1 row in set (0.04 sec)
# 寫入數據
mydba@192.168.85.132,3306 [replcrash]> insert into py_user(name,server_id) select left(uuid(),30),@@server_id;
Query OK, 1 row affected (0.02 sec)
Records: 1  Duplicates: 0  Warnings: 0
# 寫入數據
mydba@192.168.85.132,3306 [replcrash]> insert into py_user(name,server_id) select left(uuid(),30),@@server_id;
Query OK, 1 row affected (0.00 sec)
Records: 1  Duplicates: 0  Warnings: 0
# 刪除數據
mydba@192.168.85.132,3306 [replcrash]> delete from py_user where uid=2;
Query OK, 1 row affected (0.04 sec)
# 查看數據
mydba@192.168.85.132,3306 [replcrash]> select * from py_user;
+-----+--------------------------------+---------------------+-----------+
| uid | name                           | add_time            | server_id |
+-----+--------------------------------+---------------------+-----------+
|   1 | 64410b10-f504-11e7-a71e-000c29 | 2018-01-09 14:14:46 | 1323306   |
+-----+--------------------------------+---------------------+-----------+
1 row in set (0.00 sec)
# 查看當前位置
mydba@192.168.85.132,3306 [replcrash]> show master status;
+------------------+----------+--------------+------------------+-----------------------------------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                             |
+------------------+----------+--------------+------------------+-----------------------------------------------+
| mysql-bin.000191 |     1329 |              |                  | 8ab82362-9c37-11e7-a858-000c29c1025c:1-507454 |
+------------------+----------+--------------+------------------+-----------------------------------------------+
1 row in set (0.00 sec)
# 查看mysql.gtid_executed
mydba@192.168.85.132,3306 [replcrash]> select * from mysql.gtid_executed;
+--------------------------------------+----------------+--------------+
| source_uuid                          | interval_start | interval_end |
+--------------------------------------+----------------+--------------+
| 8ab82362-9c37-11e7-a858-000c29c1025c |              1 |       507450 |
+--------------------------------------+----------------+--------------+
1 row in set (0.00 sec)


2、備份並還原到新實例
# backup
[root@ZST1 ~]# innobackupex --defaults-file=/data/mysql/mysql3306/my.cnf -S /tmp/mysql3306.sock -uroot -pmysql5719 --no-timestamp /data/backup/full/20180109
# 查看備份位置
[root@ZST1 ~]# cat /data/backup/full/20180109/xtrabackup_binlog_info
mysql-bin.000191        1329    8ab82362-9c37-11e7-a858-000c29c1025c:1-507454
[root@ZST1 ~]# 

# apply-log
[root@ZST1 ~]# innobackupex --apply-log /data/backup/full/20180109
# 查看innodb位置(apply-log後生成)
[root@ZST1 ~]# cat /data/backup/full/20180109/xtrabackup_binlog_pos_innodb
mysql-bin.000191        1329
[root@ZST1 ~]# 

# copy-back
• 須要恢復的MySQL實例須要關閉(關閉前reset master方便後續對比)
• 目標datadir爲空
• 手工把apply後的文件copy過去，或者
[root@ZST1 ~]# innobackupex --defaults-file=/data/mysql/mysql3308/my.cnf --copy-back /data/backup/full/20180109
• 更改copy過去的權限
[root@ZST1 ~]# chown -R mysql:mysql /data/mysql/mysql3308/data
• 啓動MySQL
[root@ZST1 ~]# /usr/local/mysql/bin/mysqld --defaults-file=/data/mysql/mysql3308/my.cnf &

# 登陸查詢
mydba@192.168.85.132,3308 [replcrash]> show master status;
+------------------+----------+--------------+------------------+-----------------------------------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                             |
+------------------+----------+--------------+------------------+-----------------------------------------------+
| mysql-bin.000002 |      154 |              |                  | 8ab82362-9c37-11e7-a858-000c29c1025c:1-507450 |
+------------------+----------+--------------+------------------+-----------------------------------------------+
1 row in set (0.00 sec)

mydba@192.168.85.132,3308 [replcrash]> select * from mysql.gtid_executed;
+--------------------------------------+----------------+--------------+
| source_uuid                          | interval_start | interval_end |
+--------------------------------------+----------------+--------------+
| 8ab82362-9c37-11e7-a858-000c29c1025c |              1 |       507450 |
+--------------------------------------+----------------+--------------+
1 row in set (0.00 sec)

View Code

還原後經過show master status獲得的Executed_Gtid_Set爲8ab82362-9c37-11e7-a858-000c29c1025c:1-507450；實際備份的位置是8ab82362-9c37-11e7-a858-000c29c1025c:1-507454(xtrabackup_binlog_info)
上面就是Executed_Gtid_Set與xtrabackup_binlog_info不一致的例子，應該說大部分狀況這兩個值是不一致的。只是大部分的操做中，咱們對Executed_Gtid_Set值基本無感，所以不多會注意到它們不一致

1.二、一致

下面咱們執行flush binary logs，而後備份並進行後續操做

3、flush binary logs
# 續上，3306執行flush logs操做
mydba@192.168.85.132,3306 [replcrash]> flush binary logs;
Query OK, 0 rows affected (0.13 sec)
# 查看mysql.gtid_executed
mydba@192.168.85.132,3306 [replcrash]> select * from mysql.gtid_executed;
+--------------------------------------+----------------+--------------+
| source_uuid                          | interval_start | interval_end |
+--------------------------------------+----------------+--------------+
| 8ab82362-9c37-11e7-a858-000c29c1025c |              1 |       507454 |
+--------------------------------------+----------------+--------------+
1 row in set (0.00 sec)
mysql.gtid_executed表中的信息已經更新


4、再一次備份並還原到新實例
# backup
[root@ZST1 ~]# innobackupex --defaults-file=/data/mysql/mysql3306/my.cnf -S /tmp/mysql3306.sock -uroot -pmysql5719 --no-timestamp /data/backup/full/20180109new
# 查看備份位置
[root@ZST1 ~]# cat /data/backup/full/20180109new/xtrabackup_binlog_info
mysql-bin.000192        194     8ab82362-9c37-11e7-a858-000c29c1025c:1-507454
[root@ZST1 ~]# 

# apply-log
[root@ZST1 ~]# innobackupex --apply-log /data/backup/full/20180109new
# 查看innodb位置(apply-log後生成)
[root@ZST1 ~]# cat /data/backup/full/20180109new/xtrabackup_binlog_pos_innodb 
mysql-bin.000191        1329
[root@ZST1 ~]# 
注意此時xtrabackup_binlog_info與xtrabackup_binlog_pos_innodb也不一致。通常執行flush logs，或者ddl語句(<=5.7不記錄redo)會致使二者不一致

# copy-back
• 須要恢復的MySQL實例須要關閉(關閉前reset master方便後續對比)
• 目標datadir爲空
• 手工把apply後的文件copy過去，或者
[root@ZST1 ~]# innobackupex --defaults-file=/data/mysql/mysql3308/my.cnf --copy-back /data/backup/full/20180109new
• 更改copy過去的權限
[root@ZST1 ~]# chown -R mysql:mysql /data/mysql/mysql3308/data
• 啓動MySQL
[root@ZST1 ~]# /usr/local/mysql/bin/mysqld --defaults-file=/data/mysql/mysql3308/my.cnf &

# 登陸查詢
mydba@192.168.85.132,3308 [replcrash]> show master status;
+------------------+----------+--------------+------------------+-----------------------------------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                             |
+------------------+----------+--------------+------------------+-----------------------------------------------+
| mysql-bin.000002 |      154 |              |                  | 8ab82362-9c37-11e7-a858-000c29c1025c:1-507454 |
+------------------+----------+--------------+------------------+-----------------------------------------------+
1 row in set (0.00 sec)

mydba@192.168.85.132,3308 [replcrash]> select * from mysql.gtid_executed;
+--------------------------------------+----------------+--------------+
| source_uuid                          | interval_start | interval_end |
+--------------------------------------+----------------+--------------+
| 8ab82362-9c37-11e7-a858-000c29c1025c |              1 |       507454 |
+--------------------------------------+----------------+--------------+
1 row in set (0.00 sec)

View Code

還原後經過show master status獲得的Executed_Gtid_Set爲8ab82362-9c37-11e7-a858-000c29c1025c:1-507454；實際備份的位置是8ab82362-9c37-11e7-a858-000c29c1025c:1-507454(xtrabackup_binlog_info)
發現沒有，flush binary logs後獲得的備份，在還原後Executed_Gtid_Set與xtrabackup_binlog_info保持一致

2、爲何不一致

2.一、show master status

數據庫實例剛啓動時，MySQL 5.7從mysql.gtid_executed表獲取執行過的事務的GTID->Executed_Gtid_Set
數據庫實例運行過程當中，show master status返回實時信息。開啓binlog，mysql.gtid_executed需發生切換才寫入新數據，那麼show master status的數據是從哪獲取的？

2.二、mysql.gtid_executed

還原後，咱們看到的Executed_Gtid_Set信息應該來自mysql.gtid_executed(這裏指的是還原後，尚未ddl、dml等操做)
mysql.gtid_executed表按照下面的方式寫入
• 若是沒有開啓log_bin或者沒有開啓log_slave_updates，從庫在應用relay-log中的每一個事務會執行一個insert mysql.gtid_executed操做。這隻針對從庫而言~
• 若是開啓log_bin，在binlog發生rotate(flush binary logs/達到max_binlog_size)或者關閉服務時，會把全部寫入到binlog中的Gtid信息寫入到mysql.gtid_executed表。這適用於主庫和從庫~
原實例自己開啓了binlog，當咱們執行flush binary logs時就將當前的GTID信息寫入到mysql.gtid_executed表。。。
第一次的測試，雖然有不少insert、delete，可是因爲沒有發生binlog切換，相應的GTID信息沒有寫入到mysql.gtid_executed表(innodb)，此時innobackupex備份，將對應的.ibd備份起來，還原後仍是原來的數據
最開始並沒注意到mysql.gtid_executed，總覺得innobackupex會本身執行某個set gtid_purged語句。在配置文件開啓general_log，而且starce 啓動服務，結果毛線都沒看到~~~

# general_log
[mysqld]
general_log=1
general_log_file=/data/mysql/mysql3308/data/mysql-general.log
# strace
shell> strace /usr/local/mysql/bin/mysqld --defaults-file=/data/mysql/mysql3308/my.cnf >/tmp/3308.log 2>&1

View Code

23:09 2018/1/26 補充，上面的解釋不太準確，參考新隨筆：gtid_executed和gtid_purged變量是如何初始化的以及文末總結

2.三、不一致會有什麼影響

使用innobackupex的備份搭建從庫，在還原後都須要經過xtrabackup_binlog_info或者xtrabackup_binlog_pos_innodb中的信息，SET @@GLOBAL.GTID_PURGED='uuid:seq';而後再作change master to操做
其餘幾個備份工具獲得的GTID信息

# mysqldump
[root@ZST1 ~]# mysqldump -h127.0.0.1 -P3306 -uroot -p --single-transaction --master-data=2 replcrash >/data/backup/replcrash_dump_1323306_`date +%Y%m%d`.sql
[root@ZST1 ~]# more /data/backup/replcrash_dump_1323306_`date +%Y%m%d`.sql
SET @@GLOBAL.GTID_PURGED='8ab82362-9c37-11e7-a858-000c29c1025c:1-507455';
# mysqlpump
[root@ZST1 ~]# mysqlpump -h127.0.0.1 -P3306 -uroot -p --single-transaction --add-drop-table --exclude-databases=mysql,sakila,backupdb -A >/data/backup/replcrash_pump_1323306_`date +%Y%m%d`.sql
[root@ZST1 ~]# more /data/backup/replcrash_pump_1323306_`date +%Y%m%d`.sql
SET @@GLOBAL.GTID_PURGED=/*!80000 '+'*/ '8ab82362-9c37-11e7-a858-000c29c1025c:1-507455';
# mydumper
[root@ZST1 ~]# mydumper -h 127.0.0.1 -P 3306 -u root -p mysql5719 -v 3 -B replcrash -o /data/backup/mydumper
[root@ZST1 ~]# more /data/backup/mydumper/metadata 
Started dump at: 2018-01-09 16:59:37
SHOW MASTER STATUS:
        Log: mysql-bin.000192
        Pos: 353
        GTID:8ab82362-9c37-11e7-a858-000c29c1025c:1-507455

Finished dump at: 2018-01-09 16:59:38
# innobackupex
[root@ZST1 ~]# innobackupex --defaults-file=/data/mysql/mysql3306/my.cnf -S /tmp/mysql3306.sock -uroot -pmysql5719 --no-timestamp /data/backup/full/20180109
[root@ZST1 ~]# more /data/backup/full/20180109/xtrabackup_binlog_info 
mysql-bin.000191        1329    8ab82362-9c37-11e7-a858-000c29c1025c:1-507454
[root@ZST1 ~]# 
innobackupex以後，在原實例執行過truncate table py_user，所以上面的gtid不同

View Code

使用mydumper(myloader不會產生binlog，也不會應用metadata中的GTID)和innobackupex的備份文件搭建從庫，須要手動SET @@GLOBAL.GTID_PURGED
咱們建立了很多的數據庫備份，但卻不多去檢驗備份是否有效。即便偶爾使用備份搭建測試環境、搭建從庫、恢復數據
• 搭建測試環境，還原~game over
• 恢復數據，全備+binlog完美~game over
• 搭建從庫，還原+reset master+change master to~game over
咱們腦子裏都有相應的套路，只需按步驟PaPaPa，就能完成大部分工做。經驗之談讓咱們規避了很多坑，同時也"錯過"深挖坑的機會~

---

16:01 2018/3/13 總結補充
一、show master status中的Executed_Gtid_Set信息來自哪裏
@@global.gtid_executed：MySQL數據庫已經執行過的Gtid事務，處於內存中。show master status/show slave status中的Executed_Gtid_Set取自這裏
@@global.gtid_executed是由gtids_in_binlog和mysql.gtid_executed表的並集初始化的
二、gtids_in_binlog是如何計算的
依據mysql-bin.index，迭代對應binary log files中的Previous_gtids_log_event/Gtid_log_event計算，需迭代的文件受binlog_gtid_simple_recovery參數影響
三、mysql.gtid_executed表如何寫入
3.一、mysql.gtid_executed表的寫入由內部機制控制，不會記錄到binlog(解析binlog看不到任何與mysql.gtid_executed相關的信息)
3.二、mysql.gtid_executed表寫入時機
• 若是沒有開啓log_bin或者沒有開啓log_slave_updates，從庫在應用relay-log中的每一個事務會執行一個insert mysql.gtid_executed操做。這隻針對從庫而言~
• 若是開啓log_bin，在binlog發生rotate(flush binary logs/達到max_binlog_size)或者關閉服務時，會把全部寫入到binlog中的Gtid信息寫入到mysql.gtid_executed表。這適用於主庫和從庫~
四、xtrabackup備份流程
start xtrabackup_log拷貝redo log->copy .ibd;ibdata1備份InnoDB表->FTWRL->備份non-InnoDB表等->show master status獲取當前位置->unlock tables->stop and copy xtrabackup_log
mysql.gtid_executed表使用innodb存儲引擎，mysql.gtid_executed表中的數據並非實時更新的，而且對其全部的寫入不會記錄binlog，也就是不會存在於redo log中
所以xtrabackup備份所獲得的mysql.gtid_executed，就是copy gtid_executed.ibd時刻mysql.gtid_executed表中已有的數據。此數據與當時show master status的返回值很可能不一致(沒知足寫入時機)，而且後續拷貝到xtrabackup_logfile文件也不會包含mysql.gtid_executed相關信息，所以apply-log不會影響mysql.gtid_executed表中的數據
五、還原xtrabackup備份根據前面的分析還原後mysql.gtid_executed表中的數據是copy gtid_executed.ibd時刻的數據xtrabackup_binlog_info是備份non-InnoDB表後經過show master status獲得在有寫操做的環境中，只要沒觸發mysql.gtid_executed表的寫入，mysql.gtid_executed表與show master status中的Executed_Gtid_Set不可能一致注意：若是還原實例的../logs/目錄下存在binlog，在啓動數據庫服務時，它們會被計算到gtids_in_binlog，而後結合mysql.gtid_executed表初始化@@global.gtid_executed所以使用xtrabackup備份作從庫的狀況下，必定記得要根據xtrabackup_binlog_info去SET @@GLOBAL.GTID_PURGED