Mysql MGR架構誤操做引起的問題處理

【背景介紹】

故障方描述：一次用戶刷權限的時候不當心把數據庫用戶表記錄刪掉了，執行以後發現不對後重建用戶，殺掉進程後從新MGR啓動報錯。

【報錯信息】

2018-06-13T12:47:41.405593Z 32 [Note] Plugin group_replication reported: 'Group communication SSL configuration: group_replication_ssl_mode: "DISABLED"'
2018-06-13T12:47:41.405820Z 32 [Note] Plugin group_replication reported: '[GCS] Added automatically IP ranges 127.0.0.1/8,172.xx.xxx.xxx/26,192.xxx.xx.xxx/24 to the whitelist'
2018-06-13T12:47:41.406172Z 32 [Note] Plugin group_replication reported: '[GCS] SSL was not enabled'
2018-06-13T12:47:41.406216Z 32 [Note] Plugin group_replication reported: 'Initialized group communication with configuration: group_replication_group_name: "b47a8cea-6cf5-4ea4-933f-a8c20905f900"; group_replication_local_address: "172.xx.xxx.xxx:xxx1"; group_replication_group_seeds: "172.xx.xxx.xxx:xxx1,172.xx.xxx.xxx:24901,172.xx.xxx.xxx:xxxx1"; group_replication_bootstrap_group: true; group_replication_poll_spin_loops: 0; group_replication_compression_threshold: 100; group_replication_ip_whitelist: "AUTOMATIC"'
2018-06-13T12:47:41.406944Z 34 [Note] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
2018-06-13T12:47:41.434136Z 34 [ERROR] Slave SQL for channel 'group_replication_applier': Slave failed to initialize relay log info structure from the repository, Error_code: 1872
2018-06-13T12:47:41.434183Z 34 [ERROR] Plugin group_replication reported: 'Error while starting the group replication applier thread'
2018-06-13T12:47:41.434323Z 34 [Note] Plugin group_replication reported: 'The group replication applier thread was killed'
2018-06-13T12:47:41.434389Z 32 [ERROR] Plugin group_replication reported: 'Unable to initialize the Group Replication applier module.'
2018-06-13T12:47:41.434551Z 32 [Note] Plugin group_replication reported: 'Requesting to leave the group despite of not being a member'
2018-06-13T12:47:41.434588Z 32 [ERROR] Plugin group_replication reported: '[GCS] The member is leaving a group without being on one.'

【問題分析】

從報錯日誌查看，數據庫在識別relay log時出現問題，從Oracle官方文檔能夠確認異常終止MGR服務命中了Bug25534078：

MySQL Innodb Cluster Node Failed To Start With Error: "[ERROR] Slave SQL for channel 'group_replication_applier': Slave failed to initialize relay log info structure from the repository, Error_code: 1872" (文檔 ID 2386403.1)

進行信息查詢符合上面BUG現象：

 

【解決辦法】

清理mysql.slave_relay_log_info時先記錄日誌信息，按照以下方法正常修復，同時建議參數文件要指定relay log參數路徑。

可是因爲再從新建立用戶時，沒有關閉binlog同步到其餘節點，致使其餘節點加入集羣是報錯。

提供第一個方案：進行reset master（此操做很是危險），可是因爲MGR自己存在問題時間比較久致使binlog過時丟失，所以沒法修復。

提供第二個方案：暫時提供單節點服務，制定好方案後，找另外一個時間窗口對MGR架構進行修復。

 

【總結】

在涉及數據庫重要配置時，謹慎操做。

在出現問題的時候更加註意再次誤操做，致使更加難恢復。

對數據庫重要進程進行監控，及時發現問題及時修復。