mysql-master-ha 實現mysql master的高可用。

經常使用的mysql 高可用有下面幾種方案: node

名稱  原理   特色  
mysqlmha Perl腳本對mysql master作心跳,master down了之後,選舉new master   ,是要改代理層的路由信息。
mmm    多主複製管理器, 主備經過VIP共享ip , 監控節點作輪詢按期檢查master是否可用,不可用時 ,切換主備,  輪詢有時間間隔, 這段時間內若是master down , 則事務會大量失敗, 不建議
heartbeat+ drbd 主備經過VIP共享ip ,只有一個master提供服務,stand by master經過drbd進行塊的複製,相似於raid1   不適合大集羣    
mysql cluster      ndb 存儲引擎提供存儲集羣,mysqld 集羣提供訪問服務, 用的公司少,沒有大規模部署的經驗,問題多。    
percona galera cluster  galera 爲innodb提供數據同步 ,全部mysql節點都是讀寫節點,原理有點相似於mysql cluster ,都是在存儲集羣上作文章。

理論上節點間的複製隨着節點的增長會變慢。由於是1對 n-1的一個複製關係。mysql

 

 

從上表能夠看到,只剩下mysqlmha + percona galera cluster 做爲咱們的備選方案 ,後者咱們單獨找時間來測試galera cluster。sql

咱們先測試mysqlmha數據庫

 

 

測試環境:centos

OS: centos 6.5架構

mysql: 5.6.27-logapp

mysql-master-ha: 0.56ssh

 

和https://code.google.com/p/mysql-master-ha/wiki/TableOfContents?tm=6提供的文檔同樣,採用了以下mysql測試拓撲結構:測試

 

MySQL master(一臺 )this

mysql slave (2 臺 )

mysql-master-ha-manager(一臺 )

 

在全部機器上安裝 mysql-master-ha node 節點,安裝mysql-master-ha node manager; 安裝方法參考這裏: https://code.google.com/p/mysql-master-ha/wiki/Tutorial

 

 

配置好mysql master 到 MySQL slave 的主從複製關係之後,咱們在manger節點啓動 masterha_manager

masterha_manager --conf=./app1.conf

 

而後咱們在master節點上killall -9 mysqld mysqld_safe把master殺掉, 觀察manager節點上的日誌輸出 , 來理解mysql-master-ha 是如何作mysql master的高可用的:

 

 

 

[root@server9 mysql-master-ha-conf]# masterha_manager --conf=./app1.conf

Tue Nov 10 10:49:40 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Tue Nov 10 10:49:40 2015 - [info] Reading application default configuration from ./app1.conf..

Tue Nov 10 10:49:40 2015 - [info] Reading server configuration from ./app1.conf..

Tue Nov 10 10:49:40 2015 - [info] MHA::MasterMonitor version 0.56.

Tue Nov 10 10:49:40 2015 - [info] GTID failover mode = 0

 

1\發現old master不可用。

 

Tue Nov 10 10:49:40 2015 - [info] Dead Servers:

Tue Nov 10 10:49:40 2015 - [info] server7(server7:3307)

Tue Nov 10 10:49:40 2015 - [info] Alive Servers:

Tue Nov 10 10:49:40 2015 - [info] server6(server6:3307)

Tue Nov 10 10:49:40 2015 - [info] server8(server8:3307)

Tue Nov 10 10:49:40 2015 - [info] Alive Slaves:

Tue Nov 10 10:49:40 2015 - [info] server6(server6:3307) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Tue Nov 10 10:49:40 2015 - [info] Replicating from server7(server7:3307)

Tue Nov 10 10:49:40 2015 - [info] server8(server8:3307) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Tue Nov 10 10:49:40 2015 - [info] Replicating from server7(server7:3307)

Tue Nov 10 10:49:40 2015 - [warning] MySQL master is not currently alive!

 

2\檢查slave配置, 檢查master 的ssh連通性,檢查master binlog備份腳本的有效性。

檢查slave 的ssh聯通性: 進入slave檢查relay-log 的apply_diff_relay_logs腳本:

 

 

 

Tue Nov 10 10:49:40 2015 - [info] Checking slave configurations..

Tue Nov 10 10:49:40 2015 - [info] read_only=1 is not set on slave server6(server6:3307).

Tue Nov 10 10:49:40 2015 - [warning] relay_log_purge=0 is not set on slave server6(server6:3307).

Tue Nov 10 10:49:40 2015 - [info] read_only=1 is not set on slave server8(server8:3307).

Tue Nov 10 10:49:40 2015 - [info] Checking replication filtering settings..

Tue Nov 10 10:49:40 2015 - [info] Replication filtering check ok.

Tue Nov 10 10:49:40 2015 - [info] GTID (with auto-pos) is not supported

Tue Nov 10 10:49:40 2015 - [info] Starting SSH connection tests..

Tue Nov 10 10:49:41 2015 - [info] All SSH connection tests passed successfully.

Tue Nov 10 10:49:41 2015 - [info] Checking MHA Node version..

Tue Nov 10 10:49:42 2015 - [info] Version check ok.

Tue Nov 10 10:49:42 2015 - [info] Getting current master (maybe dead) info ..

Tue Nov 10 10:49:42 2015 - [info] Identified master is server7(server7:3307).

Tue Nov 10 10:49:42 2015 - [info] Checking SSH publickey authentication settings on the current master..

Tue Nov 10 10:49:42 2015 - [info] HealthCheck: SSH to server7 is reachable.

Tue Nov 10 10:49:43 2015 - [info] Master MHA Node version is 0.56.

Tue Nov 10 10:49:43 2015 - [info] Checking recovery script configurations on server7(server7:3307)..

Tue Nov 10 10:49:43 2015 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql,/var/log/mysql --output_file=/var/log/masterha/app1/save_binary_logs_test --manager_version=0.56 --start_file=master-bin.000004

Tue Nov 10 10:49:43 2015 - [info] Connecting to root@server7(server7:22)..

Creating /var/log/masterha/app1 if not exists.. ok.

Checking output directory is accessible or not..

ok.

Binlog found at /var/lib/mysql, up to master-bin.000004

Tue Nov 10 10:49:43 2015 - [info] Binlog setting check done.

Tue Nov 10 10:49:43 2015 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..

 

Tue Nov 10 10:49:43 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=server6 --slave_ip=server6 --slave_port=3307 --workdir=/var/log/masterha/app1 --target_version=5.6.27-log --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx

Tue Nov 10 10:49:43 2015 - [info] Connecting to root@server6(server6:22)..

Checking slave recovery environment settings..

Opening /var/lib/mysql/relay-log.info ... ok.

Relay log found at /var/lib/mysql, up to mysqld-relay-bin.000007

Temporary relay log file is /var/lib/mysql/mysqld-relay-bin.000007

Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.

done.

Testing mysqlbinlog output.. done.

Cleaning up test file(s).. done.

Tue Nov 10 10:49:43 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=server8 --slave_ip=server8 --slave_port=3307 --workdir=/var/log/masterha/app1 --target_version=5.6.27-log --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx

Tue Nov 10 10:49:43 2015 - [info] Connecting to root@server8(server8:22)..

Checking slave recovery environment settings..

Opening /var/lib/mysql/relay-log.info ... ok.

Relay log found at /var/lib/mysql, up to slave-relay-bin.000004

Temporary relay log file is /var/lib/mysql/slave-relay-bin.000004

Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.

done.

Testing mysqlbinlog output.. done.

Cleaning up test file(s).. done.

Tue Nov 10 10:49:44 2015 - [info] Slaves settings check done.

Tue Nov 10 10:49:44 2015 - [info]

server7(server7:3307) (current master)

+--server6(server6:3307)

+--server8(server8:3307)

 

 

三、確認 old master的mysql不可到達 (ping 3307端口不可用, 建議設置二級檢查, 從

多個路由確認old master不可到達。), 連續屢次ping old master的3307端口不可到達之後,

準備啓動master failover;

 

 

----

 

Tue Nov 10 10:49:44 2015 - [warning] master_ip_failover_script is not defined.

Tue Nov 10 10:49:44 2015 - [warning] shutdown_script is not defined.

Tue Nov 10 10:49:44 2015 - [error][/usr/local/share/perl5/MHA/Server.pm, ln457] Checking slave status failed on server6(server6:3307). err=Got error when executing SHOW SLAVE STATUS. MySQL server has gone away

Tue Nov 10 10:49:44 2015 - [info] Set master ping interval 3 seconds.

Tue Nov 10 10:49:44 2015 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.

Tue Nov 10 10:49:44 2015 - [info] Starting ping health check on server7(server7:3307)..

Tue Nov 10 10:49:44 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)

Tue Nov 10 10:49:44 2015 - [warning] Connection failed 1 time(s)..

Tue Nov 10 10:49:44 2015 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql,/var/log/mysql --output_file=/var/log/masterha/app1/save_binary_logs_test --manager_version=0.56 --binlog_prefix=master-bin

Creating /var/log/masterha/app1 if not exists.. ok.

Checking output directory is accessible or not..

ok.

Binlog found at /var/lib/mysql, up to master-bin.000004

Tue Nov 10 10:49:44 2015 - [info] HealthCheck: SSH to server7 is reachable.

Tue Nov 10 10:49:47 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)

Tue Nov 10 10:49:47 2015 - [warning] Connection failed 2 time(s)..

Tue Nov 10 10:49:50 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)

Tue Nov 10 10:49:50 2015 - [warning] Connection failed 3 time(s)..

Tue Nov 10 10:49:53 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)

Tue Nov 10 10:49:53 2015 - [warning] Connection failed 4 time(s)..

Tue Nov 10 10:49:53 2015 - [warning] Master is not reachable from health checker!

Tue Nov 10 10:49:53 2015 - [warning] Master server7(server7:3307) is not reachable!

Tue Nov 10 10:49:53 2015 - [warning] SSH is reachable.

Tue Nov 10 10:49:53 2015 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and ./app1.conf again, and trying to connect to all servers to check server status..

Tue Nov 10 10:49:53 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Tue Nov 10 10:49:53 2015 - [info] Reading application default configuration from ./app1.conf..

Tue Nov 10 10:49:53 2015 - [info] Reading server configuration from ./app1.conf..

Tue Nov 10 10:49:53 2015 - [info] GTID failover mode = 0

Tue Nov 10 10:49:53 2015 - [info] Dead Servers:

Tue Nov 10 10:49:53 2015 - [info] server7(server7:3307)

Tue Nov 10 10:49:53 2015 - [info] Alive Servers:

Tue Nov 10 10:49:53 2015 - [info] server6(server6:3307)

Tue Nov 10 10:49:53 2015 - [info] server8(server8:3307)

Tue Nov 10 10:49:53 2015 - [info] Alive Slaves:

Tue Nov 10 10:49:53 2015 - [info] server6(server6:3307) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Tue Nov 10 10:49:53 2015 - [info] Replicating from server7(server7:3307)

Tue Nov 10 10:49:53 2015 - [info] server8(server8:3307) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Tue Nov 10 10:49:53 2015 - [info] Replicating from server7(server7:3307)

Tue Nov 10 10:49:53 2015 - [info] Checking slave configurations..

Tue Nov 10 10:49:53 2015 - [info] read_only=1 is not set on slave server6(server6:3307).

Tue Nov 10 10:49:53 2015 - [warning] relay_log_purge=0 is not set on slave server6(server6:3307).

Tue Nov 10 10:49:53 2015 - [info] read_only=1 is not set on slave server8(server8:3307).

Tue Nov 10 10:49:53 2015 - [info] Checking replication filtering settings..

Tue Nov 10 10:49:53 2015 - [info] Replication filtering check ok.

Tue Nov 10 10:49:53 2015 - [info] Master is down!

Tue Nov 10 10:49:53 2015 - [info] Terminating monitoring script.

Tue Nov 10 10:49:53 2015 - [info] Got exit code 20 (Master dead).

Tue Nov 10 10:49:53 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Tue Nov 10 10:49:53 2015 - [info] Reading application default configuration from ./app1.conf..

Tue Nov 10 10:49:53 2015 - [info] Reading server configuration from ./app1.conf..

Tue Nov 10 10:49:53 2015 - [info] MHA::MasterFailover version 0.56.

 

 

四、啓動master failover :

1) 階段1 : 配置檢查:

 

Tue Nov 10 10:49:53 2015 - [info] Starting master failover.

Tue Nov 10 10:49:53 2015 - [info]

Tue Nov 10 10:49:53 2015 - [info] * Phase 1: Configuration Check Phase..

Tue Nov 10 10:49:53 2015 - [info]

Tue Nov 10 10:49:53 2015 - [info] GTID failover mode = 0

Tue Nov 10 10:49:53 2015 - [info] Dead Servers:

Tue Nov 10 10:49:53 2015 - [info] server7(server7:3307)

Tue Nov 10 10:49:53 2015 - [info] Checking master reachability via MySQL(double check)...

Tue Nov 10 10:49:53 2015 - [info] ok.

Tue Nov 10 10:49:53 2015 - [info] Alive Servers:

Tue Nov 10 10:49:53 2015 - [info] server6(server6:3307)

Tue Nov 10 10:49:53 2015 - [info] server8(server8:3307)

Tue Nov 10 10:49:53 2015 - [info] Alive Slaves:

Tue Nov 10 10:49:53 2015 - [info] server6(server6:3307) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Tue Nov 10 10:49:53 2015 - [info] Replicating from server7(server7:3307)

Tue Nov 10 10:49:53 2015 - [info] server8(server8:3307) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Tue Nov 10 10:49:53 2015 - [info] Replicating from server7(server7:3307)

Tue Nov 10 10:49:53 2015 - [info] Starting Non-GTID based failover.

Tue Nov 10 10:49:53 2015 - [info]

Tue Nov 10 10:49:53 2015 - [info] ** Phase 1: Configuration Check Phase completed.

Tue Nov 10 10:49:53 2015 - [info]

 

2) 階段2: 關閉dead master(master_ip_failover_script腳本沒有作 , 忽略對deadmaster

的ip 失效,。);

 

Tue Nov 10 10:49:53 2015 - [info] * Phase 2: Dead Master Shutdown Phase..

Tue Nov 10 10:49:53 2015 - [info]

Tue Nov 10 10:49:53 2015 - [info] Forcing shutdown so that applications never connect to the current master..

Tue Nov 10 10:49:53 2015 - [warning] master_ip_failover_script is not set. Skipping invalidating dead master IP address.

Tue Nov 10 10:49:53 2015 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.

Tue Nov 10 10:49:53 2015 - [info] * Phase 2: Dead Master Shutdown Phase completed.

Tue Nov 10 10:49:53 2015 - [info]

 

 

3) 階段3 : master恢復階段:

A:找lastest slave :

show slave status: 獲得的Relay_Master_Log_File + Read_Master_Log_Pos 最大的就能夠 。

由於server6和server8 的讀取master位點同樣,因此都是oldest和lastest ;

 

Tue Nov 10 10:49:53 2015 - [info] * Phase 3: Master Recovery Phase..

Tue Nov 10 10:49:53 2015 - [info]

Tue Nov 10 10:49:53 2015 - [info] * Phase 3.1: Getting Latest Slaves Phase..

Tue Nov 10 10:49:53 2015 - [info]

Tue Nov 10 10:49:53 2015 - [info] The latest binary log file/position on all slaves is master-bin.000004:120

Tue Nov 10 10:49:53 2015 - [info] Latest slaves (Slaves that received relay log files to the latest):

Tue Nov 10 10:49:53 2015 - [info] server6(server6:3307) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Tue Nov 10 10:49:53 2015 - [info] Replicating from server7(server7:3307)

Tue Nov 10 10:49:53 2015 - [info] server8(server8:3307) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Tue Nov 10 10:49:53 2015 - [info] Replicating from server7(server7:3307)

Tue Nov 10 10:49:53 2015 - [info] The oldest binary log file/position on all slaves is master-bin.000004:120

Tue Nov 10 10:49:53 2015 - [info] Oldest slaves:

Tue Nov 10 10:49:53 2015 - [info] server6(server6:3307) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Tue Nov 10 10:49:53 2015 - [info] Replicating from server7(server7:3307)

Tue Nov 10 10:49:53 2015 - [info] server8(server8:3307) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Tue Nov 10 10:49:53 2015 - [info] Replicating from server7(server7:3307)

Tue Nov 10 10:49:53 2015 - [info]

 

B:保存dead master binlog:

從lastest slave讀到的master binlog 的位點開始截取dead master binlog ,加上binlog的文件描述信息。

 

Tue Nov 10 10:49:53 2015 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..

Tue Nov 10 10:49:53 2015 - [info]

Tue Nov 10 10:49:53 2015 - [info] Fetching dead master's binary logs..

Tue Nov 10 10:49:53 2015 - [info] Executing command on the dead master server7(server7:3307): save_binary_logs --command=save --start_file=master-bin.000004 --start_pos=120 --binlog_dir=/var/lib/mysql,/var/log/mysql --output_file=/var/log/masterha/app1/saved_master_binlog_from_server7_3307_20151110104953.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56

Creating /var/log/masterha/app1 if not exists.. ok.

Concat binary/relay logs from master-bin.000004 pos 120 to master-bin.000004 EOF into /var/log/masterha/app1/saved_master_binlog_from_server7_3307_20151110104953.binlog ..

Binlog Checksum enabled

Dumping binlog format description event, from position 0 to 120.. ok.

No need to dump effective binlog data from /var/lib/mysql/master-bin.000004 (pos starts 120, filesize 120). Skipping.

Binlog Checksum enabled

/var/log/masterha/app1/saved_master_binlog_from_server7_3307_20151110104953.binlog has no effective data events.

Event not exists.

Tue Nov 10 10:49:54 2015 - [info] Additional events were not found from the orig master. No need to save.

 

C: 選舉新的master, 由於全部slave接收到的master位點信息是同樣的,因此他們不用再作同步了。

隨機找了一臺server6 promote成爲new master.

 

Tue Nov 10 10:49:54 2015 - [info]

Tue Nov 10 10:49:54 2015 - [info] * Phase 3.3: Determining New Master Phase..

Tue Nov 10 10:49:54 2015 - [info]

Tue Nov 10 10:49:54 2015 - [info] Finding the latest slave that has all relay logs for recovering other slaves..

Tue Nov 10 10:49:54 2015 - [info] All slaves received relay logs to the same position. No need to resync each other.

Tue Nov 10 10:49:54 2015 - [info] Searching new master from slaves..

Tue Nov 10 10:49:54 2015 - [info] Candidate masters from the configuration file:

Tue Nov 10 10:49:54 2015 - [info] Non-candidate masters:

Tue Nov 10 10:49:54 2015 - [info] New master is server6(server6:3307)

Tue Nov 10 10:49:54 2015 - [info] Starting master failover..

Tue Nov 10 10:49:54 2015 - [info]

From:

server7(server7:3307) (current master)

+--server6(server6:3307)

+--server8(server8:3307)

 

To:

server6(server6:3307) (new master)

+--server8(server8:3307)

 

D: 新的master的差別日誌生成階段

new master(server6)已經包含了全部 relay log (就是說dead master 的binlog和new master的 relay log沒有差別了--即所謂的deadmaster binlog 和new master relaylog 的差別)

 

Tue Nov 10 10:49:54 2015 - [info]

Tue Nov 10 10:49:54 2015 - [info] * Phase 3.3: New Master Diff Log Generation Phase..

Tue Nov 10 10:49:54 2015 - [info]

Tue Nov 10 10:49:54 2015 - [info] This server has all relay logs. No need to generate diff files from the latest slave.

Tue Nov 10 10:49:54 2015 - [info]

 

 

E: 新的master log應用階段,即把 new master的relay log和dead master的binlog差別在new master作一次重放。

而後獲取到new master的binlog 文件和位點信息(master-bin.000003, pos=120)至此 ,dead master和new master已經同步了。

 

 

 

Tue Nov 10 10:49:54 2015 - [info] * Phase 3.4: Master Log Apply Phase..

Tue Nov 10 10:49:54 2015 - [info]

Tue Nov 10 10:49:54 2015 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.

Tue Nov 10 10:49:54 2015 - [info] Starting recovery on server6(server6:3307)..

Tue Nov 10 10:49:54 2015 - [info] This server has all relay logs. Waiting all logs to be applied..

Tue Nov 10 10:49:54 2015 - [info] done.

Tue Nov 10 10:49:54 2015 - [info] All relay logs were successfully applied.

Tue Nov 10 10:49:54 2015 - [info] Getting new master's binlog name and position..

Tue Nov 10 10:49:54 2015 - [info] master-bin.000003:120

Tue Nov 10 10:49:54 2015 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='server6 or server6', MASTER_PORT=3307, MASTER_LOG_FILE='master-bin.000003', MASTER_LOG_POS=120, MASTER_USER='repl_user', MASTER_PASSWORD='xxx';

Tue Nov 10 10:49:54 2015 - [warning] master_ip_failover_script is not set. Skipping taking over new master IP address.

Tue Nov 10 10:49:54 2015 - [info] ** Finished master recovery successfully.

Tue Nov 10 10:49:54 2015 - [info] * Phase 3: Master Recovery Phase completed.

Tue Nov 10 10:49:54 2015 - [info]

 

階段4: 並行操做,對每個slave .比較它和new master的relay log的位點差別,

而後把這個差別在slave 補全, 最後作change master to new master;

啓動slave;

 

 

 

Tue Nov 10 10:49:54 2015 - [info] * Phase 4: Slaves Recovery Phase..

Tue Nov 10 10:49:54 2015 - [info]

Tue Nov 10 10:49:54 2015 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..

Tue Nov 10 10:49:54 2015 - [info]

Tue Nov 10 10:49:54 2015 - [info] -- Slave diff file generation on host server8(server8:3307) started, pid: 34574. Check tmp log /var/log/masterha/app1/server8_3307_20151110104953.log if it takes time..

Tue Nov 10 10:49:54 2015 - [info]

Tue Nov 10 10:49:54 2015 - [info] Log messages from server8 ...

Tue Nov 10 10:49:54 2015 - [info]

Tue Nov 10 10:49:54 2015 - [info] This server has all relay logs. No need to generate diff files from the latest slave.

Tue Nov 10 10:49:54 2015 - [info] End of log messages from server8.

Tue Nov 10 10:49:54 2015 - [info] -- server8(server8:3307) has the latest relay log events.

Tue Nov 10 10:49:54 2015 - [info] Generating relay diff files from the latest slave succeeded.

Tue Nov 10 10:49:54 2015 - [info]

Tue Nov 10 10:49:54 2015 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..

Tue Nov 10 10:49:54 2015 - [info]

Tue Nov 10 10:49:54 2015 - [info] -- Slave recovery on host server8(server8:3307) started, pid: 34576. Check tmp log /var/log/masterha/app1/server8_3307_20151110104953.log if it takes time..

Tue Nov 10 10:49:54 2015 - [info]

Tue Nov 10 10:49:54 2015 - [info] Log messages from server8 ...

Tue Nov 10 10:49:54 2015 - [info]

Tue Nov 10 10:49:54 2015 - [info] Starting recovery on server8(server8:3307)..

Tue Nov 10 10:49:54 2015 - [info] This server has all relay logs. Waiting all logs to be applied..

Tue Nov 10 10:49:54 2015 - [info] done.

Tue Nov 10 10:49:54 2015 - [info] All relay logs were successfully applied.

Tue Nov 10 10:49:54 2015 - [info] Resetting slave server8(server8:3307) and starting replication from the new master server6(server6:3307)..

Tue Nov 10 10:49:54 2015 - [info] Executed CHANGE MASTER.

Tue Nov 10 10:49:54 2015 - [info] Slave started.

Tue Nov 10 10:49:54 2015 - [info] End of log messages from server8.

Tue Nov 10 10:49:54 2015 - [info] -- Slave recovery on host server8(server8:3307) succeeded.

Tue Nov 10 10:49:54 2015 - [info] All new slave servers recovered successfully.

Tue Nov 10 10:49:54 2015 - [info]

 

階段5: : 重置new master上的slave info ., 啓動new master;

 

 

 

 

Tue Nov 10 10:49:54 2015 - [info] * Phase 5: New master cleanup phase..

Tue Nov 10 10:49:54 2015 - [info]

Tue Nov 10 10:49:54 2015 - [info] Resetting slave info on the new master..

Tue Nov 10 10:49:54 2015 - [info] server6: Resetting slave info succeeded.

Tue Nov 10 10:49:54 2015 - [info] Master failover to server6(server6:3307) completed successfully.

Tue Nov 10 10:49:54 2015 - [info]

 

階段6: 故障轉移報告:

 

----- Failover Report -----

 

app1: MySQL Master failover server7(server7:3307) to server6(server6:3307) succeeded

 

Master server7(server7:3307) is down!

 

Check MHA Manager logs at server9 for details.

 

Started automated(non-interactive) failover.

The latest slave server6(server6:3307) has all relay logs for recovery.

Selected server6(server6:3307) as a new master.

server6(server6:3307): OK: Applying all logs succeeded.

server8(server8:3307): This host has the latest relay log events.

Generating relay diff files from the latest slave succeeded.

server8(server8:3307): OK: Applying all logs succeeded. Slave started, replicating from server6(server6:3307)

server6(server6:3307): Resetting slave info succeeded.

Master failover to server6(server6:3307) completed successfully.

 

 

簡而言之 ,mysql-master-ha的作法是: 

一、檢測master 不可用。(ping 端口+ 用戶本身擴展腳本確認的方式)

二、從多個slaves中選擇relay log同步到最大master binlog 的作爲new master , 即找到latest slave;

三、從dead master 找latest slave和它的master binlog 日誌差別,而後在new master relay log 重放。 ----至此,  new master和dead master一致。 

四、對其餘slave  ,找它們和new master relay log的差別, 並重放 ,而後change master to new master -至此,new master和slaves一致。

至此, 完成mysql master的故障轉移。

可是mysql-master-ha的前提是old master物理機器的ssh連通性是可用的, 即物理節點自己可用。

 

而阿里的mysql ha 採用的作法以下: 

一、 zk 節點探測mysql 可用性 。

二、若是dead master能夠啓動, 則從新啓動, 不作master切換。-- 最理想的方式 。

三、若是dead master不可啓動了,則slaves利用zk的搶鎖機制(每個mysql slave的zk agent 在/lock目錄下建立臨時順序zk 節點,並判斷本身是不是/lock下的最小節點,若是是, 則搶鎖成功,若是否 ,則監控/lock子節點,等待搶鎖)  成爲new master ,而後dead master利用binlog回滾(對dead master和new master的日誌差別作逆向重放) 和new master保持一致 。

 

四、若是在slave 的relay log 的exec 和read 位點有差別時, 處理方案: reset slave & change master to new master .

可是ppt有幾點疑問 ,第3步能夠保證搶到zk鎖的是 latest slave麼 ???????第4步slave直接作reset slave  把relay log清理掉,從新和 new master作同步,效率是否會過低了?????  另外,從ppt來看,這種搶zk鎖的操做 , 從邏輯上來看彷佛也沒有問題,經過dead master的binlog回滾,保持old 和new master的一致 ,而後其餘slaves和new master保持一致便可, 可是理解起來有點疑惑。

 

騰訊的作法: 

一、利用半同步保證master的binlog 同步到slave並寫入relay log之後, master 才返回事務commit給client .

二、也是zk agent監控master 可用性,當master不可用時 ,數據庫proxy中的路由信息改成master 不可用。

三、 slave上報relay log中讀到的最新master binlog ,scheduler從中選擇一個latest slave(讀到old master binlog 最大位點的)做爲new master, 並要求new master

重放完relay log (表示存儲引擎中已經有relay log所表示的數據庫變動了 )

四、修改其餘slave的master變成new master; 

值得注意的是, 騰訊的TDSQL 文檔中沒有提到new master 和old master的日誌差別,以及new master和其餘slave的relay log 日誌差別問題, 

由於前者經過半同步獲得瞭解決,後者經過其餘slave的change master to new master之後,進行位點追趕。好比 new master已經到

master-bin.00002
135

而slave纔到master-bin.00001的120 ,則直接change to new master   master-bin.00001 ,120 進行位點追趕。 

五、 修改new master的slave info .

六、修改代理的路由信息,master完成failover . 。

這裏, 騰訊TDSQL 的作法理解起來更直觀一些。

 

整體比較來看,在old master 和new master的binlog同步上,都是一致的(即要保證old master和new master 的徹底一致 ,mysql-master-ha用了perl腳本批量作new master的位點追趕, 騰訊用了半同步,阿里用了old master的日誌回滾) 。而在new master和其餘slave的slave log同步上, mysql-master-ha要求slave 和new master徹底一致,騰訊和阿里則沒有要求(mysql-master-ha強調數據的一致性,阿里和騰訊強調的是系統的可用性, 由於自己這種master-slave的架構在使用上就要注意主從同步的延遲性問題)。 

整體我的以爲,騰訊的TDSQL設計上更簡明些, 而mysql-master-ha在作完master failover之後,須要再通知數據庫代理層修改主備路由信息。這個是後話 (要本身實現TDSQL的功能 ) 。

相關文章
相關標籤/搜索