gh-ost在線主從同步MySQL下DDL生產表結構修改（初步研究）

參考

http://www.javashuo.com/article/p-qpyuesya-ne.html

https://github.com/github/gh-ost

http://www.javashuo.com/article/p-sqemdmud-gb.html

主從同步過程省略。

啓動主從同步

show processlist;
show master status;

記錄Position，執行
change master to master_host='192.168.10.60',
master_port=3306,
master_user='slave',
master_password='123456',
master_log_file='mysql-bin.000002',
master_log_pos=1407;

啓動主從同步
start slave;

開始在線DDL

添加1個字段，奇怪的能夠不填數據庫root用戶的密碼，無需 --password="yourpassword"

./gh-ost --user="root" --host=192.168.10.60 --database="test" --table="tab" --alter="ADD COLUMN oneColumn varchar(10)" --allow-on-master --execute

即便在主次同步時也是能夠執行的。在執行gho複製的時候暫停了主從同步。

[root@hadoop001 home]# ./gh-ost --user="root" --host=192.168.10.60 --database="test" --switch-to-rbr --table="tab" --alter="ADD COLUMN oneColumn varchar(10)" --allow-on-master --execute
[2020/07/21 23:01:35] [info] binlogsyncer.go:133 create BinlogSyncer with config {99999 mysql 192.168.10.60 3306 root false false <nil> false UTC true 0 0s 0s 0 false}
[2020/07/21 23:01:35] [info] binlogsyncer.go:354 begin to sync binlog from position (mysql-bin.000003, 28925)
[2020/07/21 23:01:35] [info] binlogsyncer.go:203 register slave for master server 192.168.10.60:3306
[2020/07/21 23:01:35] [info] binlogsyncer.go:723 rotate to (mysql-bin.000003, 28925)
# Migrating `test`.`tab`; Ghost table is `test`.`_tab_gho`
# Migrating hadoop001:3306; inspecting hadoop001:3306; executing on hadoop001
# Migration started at Tue Jul 21 23:01:35 +0800 2020
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: ; critical-load: ; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/gh-ost.throttle 
# Serving on unix socket: /tmp/gh-ost.test.tab.sock
Copy: 0/22 0.0%; Applied: 0; Backlog: 0/1000; Time: 0s(total), 0s(copy); streamer: mysql-bin.000003:30509; Lag: 0.01s, State: migrating; ETA: N/A
Copy: 0/22 0.0%; Applied: 0; Backlog: 0/1000; Time: 1s(total), 1s(copy); streamer: mysql-bin.000003:34164; Lag: 0.01s, State: migrating; ETA: N/A
Copy: 22/22 100.0%; Applied: 0; Backlog: 0/1000; Time: 1s(total), 1s(copy); streamer: mysql-bin.000003:35132; Lag: 0.01s, State: migrating; ETA: due
Copy: 22/22 100.0%; Applied: 0; Backlog: 1/1000; Time: 2s(total), 1s(copy); streamer: mysql-bin.000003:39597; Lag: 0.01s, State: migrating; ETA: due
# Migrating `test`.`tab`; Ghost table is `test`.`_tab_gho`
# Migrating hadoop001:3306; inspecting hadoop001:3306; executing on hadoop001
# Migration started at Tue Jul 21 23:01:35 +0800 2020
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: ; critical-load: ; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/gh-ost.throttle 
# Serving on unix socket: /tmp/gh-ost.test.tab.sock
Copy: 22/22 100.0%; Applied: 0; Backlog: 0/1000; Time: 2s(total), 1s(copy); streamer: mysql-bin.000003:39996; Lag: 0.01s, State: migrating; ETA: due
[2020/07/21 23:01:37] [info] binlogsyncer.go:164 syncer is closing...
[2020/07/21 23:01:37] [info] binlogsyncer.go:179 syncer is closed
# Done

 

添加多個字段

./gh-ost --user="root" --host=192.168.10.60 --database="test" --table="tab" --alter="ADD COLUMN col02 varchar(10),add column col03 int not null default 0 comment '備註' " --allow-on-master --execute

刪除字段

./gh-ost --user="root" --host=192.168.10.60 --database="test" --table="tab" --initially-drop-old-table --alter="DROP COLUMN col02 " --allow-on-master --execute

若是binlog使用STATEMENT模式，須要修改成ROW模式，看提示須要添加--switch-to-rbr，
也能夠手動改模式

SET SESSION binlog_format = 'ROW';
SET GLOBAL binlog_format = 'ROW';

壓力下測試，同時測試如下場景時同時修改schema

• 一、在大量寫；
• 二、啓動主從同步的狀態下；

大量寫的腳本，灌10萬條數據

create procedure addDatas()
begin
declare pid int;
set pid = 100000;
while pid>0 do
insert into tab(name)values('testuser');
set pid = pid-1; 
end while;
end

先執行
call addDatas()

再執行
./gh-ost --user="root" --host=192.168.10.60 --database="test" --table="tab" --alter="ADD COLUMN oneColumn1 varchar(10)" --allow-on-master --execute

原來的數據爲0條

執行後

輸出結果

[root@hadoop001 home]# ./gh-ost --user="root" --host=192.168.10.60 --database="test" --table="tab" --alter="ADD COLUMN oneColumn1 varchar(10)" --allow-on-master --execute
[2020/07/21 23:13:41] [info] binlogsyncer.go:133 create BinlogSyncer with config {99999 mysql 192.168.10.60 3306 root false false <nil> false UTC true 0 0s 0s 0 false}
[2020/07/21 23:13:41] [info] binlogsyncer.go:354 begin to sync binlog from position (mysql-bin.000003, 48626709)
[2020/07/21 23:13:41] [info] binlogsyncer.go:203 register slave for master server 192.168.10.60:3306
[2020/07/21 23:13:41] [info] binlogsyncer.go:723 rotate to (mysql-bin.000003, 48626709)
# Migrating `test`.`tab`; Ghost table is `test`.`_tab_gho`
# Migrating hadoop001:3306; inspecting hadoop001:3306; executing on hadoop001
# Migration started at Tue Jul 21 23:13:41 +0800 2020
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: ; critical-load: ; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/gh-ost.throttle 
# Serving on unix socket: /tmp/gh-ost.test.tab.sock
Copy: 0/12155 0.0%; Applied: 0; Backlog: 0/1000; Time: 0s(total), 0s(copy); streamer: mysql-bin.000003:48646230; Lag: 0.01s, State: migrating; ETA: N/A
Copy: 0/12155 0.0%; Applied: 4741; Backlog: 3/1000; Time: 1s(total), 1s(copy); streamer: mysql-bin.000003:50271412; Lag: 0.01s, State: migrating; ETA: N/A
Copy: 0/12155 0.0%; Applied: 9301; Backlog: 1/1000; Time: 2s(total), 2s(copy); streamer: mysql-bin.000003:51834667; Lag: 0.01s, State: migrating; ETA: N/A
Copy: 0/12155 0.0%; Applied: 14004; Backlog: 9/1000; Time: 3s(total), 3s(copy); streamer: mysql-bin.000003:53444572; Lag: 0.01s, State: migrating; ETA: N/A
Copy: 0/12155 0.0%; Applied: 18576; Backlog: 1/1000; Time: 4s(total), 4s(copy); streamer: mysql-bin.000003:55008409; Lag: 0.00s, State: migrating; ETA: N/A
Copy: 0/12155 0.0%; Applied: 23194; Backlog: 2/1000; Time: 5s(total), 5s(copy); streamer: mysql-bin.000003:56589121; Lag: 0.01s, State: migrating; ETA: N/A
Copy: 1000/12155 8.2%; Applied: 27909; Backlog: 4/1000; Time: 6s(total), 6s(copy); streamer: mysql-bin.000003:58218868; Lag: 0.01s, State: migrating; ETA: 1m6s
Copy: 1000/12155 8.2%; Applied: 32645; Backlog: 0/1000; Time: 7s(total), 7s(copy); streamer: mysql-bin.000003:59839783; Lag: 0.01s, State: migrating; ETA: 1m18s
Copy: 1000/12155 8.2%; Applied: 37399; Backlog: 5/1000; Time: 8s(total), 8s(copy); streamer: mysql-bin.000003:61468184; Lag: 0.01s, State: migrating; ETA: 1m29s
Copy: 1000/12155 8.2%; Applied: 41908; Backlog: 6/1000; Time: 9s(total), 9s(copy); streamer: mysql-bin.000003:63017054; Lag: 0.01s, State: migrating; ETA: 1m40s
Copy: 1000/12155 8.2%; Applied: 46481; Backlog: 3/1000; Time: 10s(total), 10s(copy); streamer: mysql-bin.000003:64587677; Lag: 0.01s, State: migrating; ETA: 1m51s
Copy: 1000/12155 8.2%; Applied: 51207; Backlog: 3/1000; Time: 11s(total), 11s(copy); streamer: mysql-bin.000003:66206261; Lag: 0.01s, State: migrating; ETA: 2m2s
Copy: 1000/12155 8.2%; Applied: 55899; Backlog: 3/1000; Time: 12s(total), 12s(copy); streamer: mysql-bin.000003:67813910; Lag: 0.01s, State: migrating; ETA: 2m13s
Copy: 1000/12155 8.2%; Applied: 60535; Backlog: 4/1000; Time: 13s(total), 13s(copy); streamer: mysql-bin.000003:69415881; Lag: 0.01s, State: migrating; ETA: 2m25s
Copy: 2000/12155 16.5%; Applied: 65086; Backlog: 2/1000; Time: 14s(total), 14s(copy); streamer: mysql-bin.000003:70998265; Lag: 0.01s, State: migrating; ETA: 1m11s
Copy: 2000/12155 16.5%; Applied: 69623; Backlog: 7/1000; Time: 15s(total), 15s(copy); streamer: mysql-bin.000003:72564672; Lag: 0.01s, State: migrating; ETA: 1m16s
Copy: 3000/12155 24.7%; Applied: 74058; Backlog: 2/1000; Time: 16s(total), 16s(copy); streamer: mysql-bin.000003:74113223; Lag: 0.01s, State: migrating; ETA: 48s
Copy: 3000/12155 24.7%; Applied: 78445; Backlog: 5/1000; Time: 17s(total), 17s(copy); streamer: mysql-bin.000003:75628183; Lag: 0.01s, State: migrating; ETA: 51s
Copy: 3000/12155 24.7%; Applied: 82853; Backlog: 4/1000; Time: 18s(total), 18s(copy); streamer: mysql-bin.000003:77151858; Lag: 0.00s, State: migrating; ETA: 54s
Copy: 3000/12155 24.7%; Applied: 87229; Backlog: 5/1000; Time: 19s(total), 19s(copy); streamer: mysql-bin.000003:78663838; Lag: 0.01s, State: migrating; ETA: 57s
Copy: 12254/12254 100.0%; Applied: 87746; Backlog: 0/1000; Time: 19s(total), 19s(copy); streamer: mysql-bin.000003:79018628; Lag: 0.01s, State: migrating; ETA: due
Copy: 12254/12254 100.0%; Applied: 87746; Backlog: 1/1000; Time: 20s(total), 19s(copy); streamer: mysql-bin.000003:79022790; Lag: 0.00s, State: migrating; ETA: due
# Migrating `test`.`tab`; Ghost table is `test`.`_tab_gho`
# Migrating hadoop001:3306; inspecting hadoop001:3306; executing on hadoop001
# Migration started at Tue Jul 21 23:13:41 +0800 2020
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: ; critical-load: ; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/gh-ost.throttle 
# Serving on unix socket: /tmp/gh-ost.test.tab.sock
Copy: 12254/12254 100.0%; Applied: 87746; Backlog: 0/1000; Time: 20s(total), 19s(copy); streamer: mysql-bin.000003:79023531; Lag: 0.00s, State: migrating; ETA: due
[2020/07/21 23:14:01] [info] binlogsyncer.go:164 syncer is closing...
[2020/07/21 23:14:01] [info] binlogsyncer.go:179 syncer is closed
# Done

若是在執行一次修改後，再執行其餘修改，會被禁止，消息：
2020-07-21 23:23:34 FATAL Table `_tab_del` already exists. Panicking. Use --initially-drop-old-table to force dropping it, though I really prefer that you drop it or rename it away
須要手動刪除`_tab_del`表
或者設置參數：
--ok-to-drop-table
go-ost執行完之後是否刪除老表，加上此參數會自動刪除老表。

一些參數

--max-load
遷移過程當中，gh-ost會時刻關注負載狀況，負載閥值是使用者本身定義，好比數據庫的最大鏈接數，若是超過閥值，gh-ost不會退出，會等待到負載在閥值如下繼續執行。

--critical-load
這個指的是gh-ost退出閥值，當負載超過這個閥值，gh-ost會中止並退出

--max-lag-millis會監控從庫的主從延遲狀況，若是延遲秒數超過這個閥值，遷移不會退出，等待延遲秒數低於這個閥值繼續遷移。