MySQL高可用之MHA的搭建
MySQL MHA架構介紹：html
MHA（Master High Availability）目前在MySQL高可用方面是一個相對成熟解決方案，它由日本DeNA公司youshimaton（現就任於Facebook公司）開發，是一套優秀的做爲MySQL高可用性環境下故障切換和主從提高的高可用軟件。在MySQL故障切換過程當中，MHA能作到在0~30秒以內自動完成數據庫故障切換操做，而且在進行故障切換的過程當中，MHA能在最大程度上保證數據的一致性，以達到真正的高可用。node
該軟件由兩部分組成：MHA Manager（管理節點)和MHA Node（數據節點）。MHA Manager能夠單獨部署在一臺獨立的機器上管理多個master-slave集羣，也能夠部署在一臺slave節點上。MHA Node運行在每臺MySQL服務器上，MHA Manager會定時探測集羣中的master節點，當master出現故障時，它能夠自動將最新數據的slave提高爲新的master，而後將全部其餘的slave從新指向新的master。整個故障轉移過程對應用程序徹底透明。mysql
在MHA自動故障切換過程當中，MHA試圖從宕機的主服務器上保存二進制日誌，最大程度的保證數據的不丟失，但這並不老是可行的。例如，若是主服務器硬件故障或沒法經過ssh訪問，MHA無法保存二進制日誌，只進行故障轉移而丟失了最新的數據。使用MySQL 5.5的半同步複製，能夠大大下降數據丟失的風險。MHA能夠與半同步複製結合起來。若是隻有一個slave已經收到了最新的二進制日誌，MHA能夠將最新的二進制日誌應用於其餘全部的slave服務器上，所以能夠保證全部節點的數據一致性。linux
目前MHA主要支持一主多從的架構，要搭建MHA,要求一個複製集羣中必須最少有三臺數據庫服務器，一主二從，即一臺充當master，一臺充當備用master，另一臺充當從庫，由於至少須要三臺服務器，出於機器成本的考慮，淘寶也在該基礎上進行了改造，目前淘寶TMHA已經支持一主一從。（出自：《深刻淺出MySQL(第二版)》）sql
MySQL MHA架構圖：shell
MySQL MHA工做原理數據庫
（1）從宕機崩潰的master保存二進制日誌事件（binlog events）;bash
（2）識別含有最新更新的slave;服務器
（3）應用差別的中繼日誌(relay log) 到其餘slave;session
（4）應用從master保存的二進制日誌事件(binlog events);
（5）提高一個slave爲新master;
（6）使用其餘的slave鏈接新的master進行復制。
官網介紹：https://code.google.com/p/mysql-master-ha/
實驗環境：
Centos6.25-X64
MySQL版本5.5
server03和server04是server02的slave,其中master對外提供寫服務，備選master（實際的slave，主機名server03）提供讀服務，slave也提供相關的讀服務，一旦master宕機，將會把備選master提高爲新的master，slave指向新的master.
MySQL MHA部署過程
方法一
(1) 在全部節點都要安裝MHA node所需的perl模塊（DBD:mysql），能夠經過yum安裝，若是沒epel源,先安裝epel源，（系統時間必定要是最新的，不然安裝時會出各類問題）
在server02(192.168.2.128)操做：
$ rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
$ yum install perl-DBD-MySQL -y
在server03(192.168.2.129)操做：
$ rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
$ yum install perl-DBD-MySQL -y
在server04(192.168.2.130)操做：
$ rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
$ yum install perl-DBD-MySQL -y
（2）在全部的節點安裝MHA node：
（下面以server02爲例，記得server03和server04也同樣的操做），
MHA node和MHA Manager都在要官網下載，
下載地址：https://code.google.com/p/mysql-master-ha/wiki/Downloads?tm=2
$ tar xf mha4mysql-node-0.56.tar.gz $ cd mha4mysql-node-0.56 $ perl Makefile.PL Can't locate ExtUtils/MakeMaker.pm in @INC (@INC contains: inc /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at inc/Module/Install/Makefile.pm line 4. $ yum install -y perl-devel $ perl Makefile.PL *** Module::AutoInstall version 1.03 *** Checking for Perl dependencies... Can't locate CPAN.pm in @INC (@INC contains: inc /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at inc/Module/AutoInstall.pm line 279. 192.168.2.128 [root mha4mysql-node-0.56]$ yum install -y perl-CPAN 192.168.2.128 [root mha4mysql-node-0.56]$ perl Makefile.PL *** Module::AutoInstall version 1.03 *** Checking for Perl dependencies... [Core Features] - DBI ...loaded. (1.609) - DBD::mysql ...loaded. (4.013) *** Module::AutoInstall configuration finished. Checking if your kit is complete... Looks good Writing Makefile for mha4mysql::node $ make && make install
方法2、
安裝MHA node所需的perl模塊（DBD:mysql）也能夠經過腳本安裝，安裝腳本以下：（我的不建議用這樣的方法安裝)
$ cat install.sh #!/bin/bash wget mv cpanm /usr/bin chmod 755 /usr/bin/cpanm cat > /root/list << EOF install DBD::mysql EOF for package in `cat /root/list` do cpanm $package done $ bash install.sh
再安裝MHA node節點：
$ tar xf mha4mysql-node-0.56.tar.gz $ cd mha4mysql-node-0.56 $ perl Makefile.PL $ make && make install $ cd /usr/local/bin/ $ pwd /usr/local/bin $ ll -r-xr-xr-x 1 root root 15498 1月 18 11:02 apply_diff_relay_logs -r-xr-xr-x 1 root root 4807 1月 18 11:02 filter_mysqlbinlog -r-xr-xr-x 1 root root 7401 1月 18 11:02 purge_relay_logs -r-xr-xr-x 1 root root 7263 1月 18 11:02 save_binary_logs
Node腳本說明：(這些工具一般由MHA Manager的腳本觸發，無需人爲操做)
save_binary_logs //保存和複製master的二進制日誌
apply_diff_relay_logs //識別差別的中繼日誌事件並將其差別的事件應用於其餘的slave
filter_mysqlbinlog //去除沒必要要的ROLLBACK事件（MHA已再也不使用這個工具）
purge_relay_logs //清除中繼日誌（不會阻塞SQL線程）
2.安裝MHA Manager,在MHA Manager的主機也是須要安裝MHA Node,MHA Manger也依賴於perl模塊
（1）在MHA Manager的主機須要安裝MHA Node,因此步驟和上面操做同樣，以下（在server01 192.168.2.131操做）：
$ rpm -ivh $ yum install perl-DBD-MySQL -y $ yum install -y perl-devel perl-CPAN $ tar xf mha4mysql-node-0.56.tar.gz $ cd mha4mysql-node-0.56 $ perl Makefile.PL $ make && make install
（2）安裝MHA Manager。首先安裝MHA Manger依賴的perl模塊（我這裏使用yum安裝）：
$ yum install perl-DBD-MySQL perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager perl-Time-HiRes -y
安裝MHA Manager軟件包：
$ tar xf mha4mysql-manager-0.56.tar.gz $ cd mha4mysql-manager-0.56 $ perl Makefile.PL *** Module::AutoInstall version 1.03 *** Checking for Perl dependencies... [Core Features] - DBI ...loaded. (1.609) - DBD::mysql ...loaded. (4.013) - Time::HiRes ...loaded. (1.9721) - Config::Tiny ...loaded. (2.12) - Log::Dispatch ...loaded. (2.26) - Parallel::ForkManager ...loaded. (0.7.9) - MHA::NodeConst ...loaded. (0.56) *** Module::AutoInstall configuration finished. Writing Makefile for mha4mysql::manager $ make && make install
安裝完成後，在/usr/local/bin會產生相關的腳本：
$ pwd /usr/local/bin 192.168.2.131 [root bin]$ ll -r-xr-xr-x. 1 root root 15498 2月 4 2014 apply_diff_relay_logs -r-xr-xr-x. 1 root root 4807 2月 4 2014 filter_mysqlbinlog -r-xr-xr-x. 1 root root 1995 1月 11 22:55 masterha_check_repl -r-xr-xr-x. 1 root root 1779 1月 11 22:55 masterha_check_ssh -r-xr-xr-x. 1 root root 1865 1月 11 22:55 masterha_check_status -r-xr-xr-x. 1 root root 3201 1月 11 22:55 masterha_conf_host -r-xr-xr-x. 1 root root 2517 1月 11 22:55 masterha_manager -r-xr-xr-x. 1 root root 2165 1月 11 22:55 masterha_master_monitor -r-xr-xr-x. 1 root root 2373 1月 11 22:55 masterha_master_switch -r-xr-xr-x. 1 root root 3749 1月 11 22:55 masterha_secondary_check -r-xr-xr-x. 1 root root 1739 1月 11 22:55 masterha_stop -rwxr-xr-x. 1 root root 2133 1月 11 23:40 master_ip_failover -rwxr-xr-x. 1 root root 11867 1月 11 22:55 power_manager -r-xr-xr-x. 1 root root 7401 2月 4 2014 purge_relay_logs -r-xr-xr-x. 1 root root 7263 2月 4 2014 save_binary_logs -rwxr-xr-x. 1 root root 1360 1月 11 22:55 send_report
複製相關腳本到/usr/local/bin目錄(軟件包解壓縮後就有了，不是必須，由於這些腳本不完整，須要本身修改，這是軟件開發着留給咱們本身發揮的,若是開啓下面的任何一個腳本對應的參數，而對應這裏的腳本又沒有修改，則會報錯）。
$ pwd /root/mha4mysql-manager-0.56/samples/scripts $ ll -rwxr-xr-x. 1 root root 3443 1月 8 2012 master_ip_failover //自動切換時vip管理的腳本，不是必須，若是咱們使用keepalived的，咱們能夠本身編寫腳本完成對vip的管理，好比監控mysql，若是mysql異常， 咱們中止keepalived就行，這樣vip就會自動漂移 -rwxr-xr-x. 1 root root 9186 1月 8 2012 master_ip_online_change //在線切換時vip的管理，不是必須，一樣能夠能夠自行編寫簡單的shell完成 -rwxr-xr-x. 1 root root 11867 1月 8 2012 power_manager //故障發生後關閉主機的腳本，不是必須 -rwxr-xr-x. 1 root root 1360 1月 8 2012 send_report //因故障切換後發送報警的腳本，不是必須，可自行編寫簡單的shell完成 $ cp * /usr/local/bin/
3.配置SSH登陸無密碼驗證
在server02 192.168.2.131操做（Monitor）：
$ ssh-keygen -t rsa
$ ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.2.128
$ ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.2.129
$ ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.2.130
在server02 192.168.2.128操做（Master）：
$ ssh-keygen -t rsa
$ ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.2.129
$ ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.2.130
在server03 192.168.2.129操做(slave)：
$ ssh-keygen -t rsa
$ ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.2.128
$ ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.2.130
在server04 192.168.2.130操做(slave)：
$ ssh-keygen -t rsa
$ ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.2.128
$ ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.2.129
4.搭建主從複製環境
注意：binlog-do-db 和 replicate-ignore-db 設置必須相同。 MHA 在啓動時候會檢測過濾規則，若是過濾規則不一樣，MHA 不啓動監控和故障轉移。
（1）在Master 192.168.2.128(server02)上備份一份完整的數據：
$ mysqldump -uroot -p123456 --master-data=2 --single-transaction -R --triggers -A > all.sql
其中--master-data=2表明備份時刻記錄master的Binlog位置和Position，--single-transaction意思是獲取一致性快照，-R意思是備份存儲過程和函數，--triggres的意思是備份觸發器，-A表明備份全部的庫。
（2）在Master 192.168.2.128(server02)上建立複製用戶:
mysql> grant replication slave on *.* to 'repl'@'192.168.2.%' identified by '123456';
mysql> flush privileges;
（3）查看主庫備份時的binlog名稱和位置，MASTER_LOG_FILE和MASTER_LOG_POS：
$ head -n 30 all.sql | grep 'CHANGE MASTER TO'
-- CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000004', MASTER_LOG_POS=245;
（4）把備份複製到192.168.2.129和192.168.2.130
$ scp all.sql 192.168.2.129:/root/
$ scp all.sql 192.168.2.130:/root/
（5）分別在兩臺服務器上導入備份，執行復制相關命令
在slave主機server03 192.168.2.129上操做：
$ mysql -uroot -p123456 < ./all.sql
mysql> stop slave;
mysql> CHANGE MASTER TO MASTER_HOST='192.168.2.128',MASTER_USER='repl', MASTER_PASSWORD='123456',MASTER_LOG_FILE='mysql-bin.000004',MASTER_LOG_POS=245;
mysql> start slave;
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.2.128
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000004
Read_Master_Log_Pos: 472
Relay_Log_File: localhost-relay-bin.000002
Relay_Log_Pos: 480
Relay_Master_Log_File: mysql-bin.000004
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
在slave master04 192.168.2.130上操做，導入備份，執行同步操做，以下：
$ mysql -uroot -p123456 < ./all.sql
mysql> slave stop;
mysql> CHANGE MASTER TO MASTER_HOST='192.168.2.128',MASTER_USER='repl', MASTER_PASSWORD='123456',MASTER_LOG_FILE='mysql-bin.000004',MASTER_LOG_POS=245;
mysql> slave start;
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.2.128
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000004
Read_Master_Log_Pos: 472
Relay_Log_File: localhost-relay-bin.000002
Relay_Log_Pos: 480
Relay_Master_Log_File: mysql-bin.000004
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
（6）兩臺slave服務器設置read_only（從庫對外提供讀服務，之因此沒有寫進配置文件，是由於隨時slave會提高爲master）
$ mysql -uroot -p123456 -e "set global read_only=1"
$ mysql -uroot -p123456 -e "set global read_only=1"
（7）建立監控用戶（在master上執行，也就是server02 192.168.2.128）：
mysql> grant all privileges on *.* to 'root'@'192.168.2.%' identified by '123456';
mysql> flush privileges;
到這裏整個集羣環境已經搭建完畢，剩下的就是配置MHA軟件了。
5.配置MHA
（1）在192.168.2.131也就是server01上面建立MHA的工做目錄，而且建立相關配置文件（在軟件包解壓後的目錄裏面有樣例配置文件）。
$ mkdir -p /etc/masterha
$ cp mha4mysql-manager-0.56/samples/conf/app1.cnf /etc/masterha/
修改app1.cnf配置文件，修改後的文件內容以下（注意，配置文件中的註釋須要去掉，我這裏是爲了解釋清楚）：
[root@192.168.2.131 ~]# cat /etc/masterha/app1.cnf [server default] manager_workdir=/var/log/masterha/app1.log //設置manager的工做目錄 manager_log=/var/log/masterha/app1/manager.log //設置manager的日誌 master_binlog_dir=/data/mysql //設置master 保存binlog的位置，以便MHA能夠找到master的日誌，我這裏的也就是mysql的數據目錄 master_ip_failover_script= /usr/local/bin/master_ip_failover //設置自動failover時候的切換腳本 master_ip_online_change_script= /usr/local/bin/master_ip_online_change //設置手動切換時候的切換腳本 password=123456 //設置mysql中root用戶的密碼，這個密碼是前文中建立監控用戶的那個密碼 user=root //設置監控用戶root ping_interval=1 //設置監控主庫，發送ping包的時間間隔，默認是3秒，嘗試三次沒有迴應的時候自動進行railover remote_workdir=/tmp //設置遠端mysql在發生切換時binlog的保存位置 repl_password=123456 //設置複製用戶的密碼 repl_user=repl //設置複製環境中的複製用戶名 report_script=/usr/local/bin/send_report //設置發生切換後發送的報警的腳本 secondary_check_script= /usr/local/bin/masterha_secondary_check -s server03 -s server02 --user=root --master_host=server02 --master_ip=192.168.2.128 --master_port=3306 //一旦MHA到server02的監控之間出現問題，MHA Manager將會嘗試從server03登陸到server02 shutdown_script="" //設置故障發生後關閉故障主機腳本（該腳本的主要做用是關閉主機放在發生腦裂,這裏沒有使用） ssh_user=root //設置ssh的登陸用戶名 [server1] hostname=192.168.2.128 port=3306 [server2] hostname=192.168.2.129 port=3306 candidate_master=1 //設置爲候選master，若是設置該參數之後，發生主從切換之後將會將此從庫提高爲主庫，即便這個主庫不是集羣中事件最新的slave check_repl_delay=0 //默認狀況下若是一個slave落後master 100M的relaylogs的話，MHA將不會選擇該slave做爲一個新的master，由於對於這個slave的恢復須要花費很長時間，經過設置check_repl_delay=0,MHA觸發切換在選擇一個新的master的時候將會忽略複製延時，這個參數對於設置了candidate_master=1的主機很是有用，由於這個候選主在切換的過程當中必定是新的master [server3] hostname=192.168.2.130 port=3306
(2)設置relay log的清除方式（在每一個slave節點上）：
在slave master03 192.168.2.129操做：
$ mysql -uroot -p123456 -e "set global relay_log_purge=0"
在slave master04 192.168.2.130操做：
$ mysql -uroot -p123456 -e "set global relay_log_purge=0"
注意：
MHA在發生切換的過程當中，從庫的恢復過程當中依賴於relay log的相關信息，因此這裏要將relay log的自動清除設置爲OFF，採用手動清除relay log的方式。在默認狀況下，從服務器上的中繼日誌會在SQL線程執行完畢後被自動刪除。可是在MHA環境中，這些中繼日誌在恢復其餘從服務器時可能會被用到，所以須要禁用中繼日誌的自動刪除功能。按期清除中繼日誌須要考慮到複製延時的問題。在ext3的文件系統下，刪除大的文件須要必定的時間，會致使嚴重的複製延時。爲了不復制延時，須要暫時爲中繼日誌建立硬連接，由於在linux系統中經過硬連接刪除大文件速度會很快。（在mysql數據庫中，刪除大表時，一般也採用創建硬連接的方式）
設置按期清理relay腳本（兩臺slave服務器）：
在slave 192.168.2.129操做：
[root ~]$ cat purge_relay_log.sh #!/bin/bash user=root passwd=123456 port=3306 log_dir='/data/masterha/log' work_dir='/data' purge='/usr/local/bin/purge_relay_logs' if [ ! -d $log_dir ] then mkdir $log_dir -p fi $purge --user=$user --password=$passwd --disable_relay_log_purge --port=$port --workdir=$work_dir >> $log_dir/purge_relay_logs.log 2>&1 [root ~]$ crontab -l 4 * * * /bin/bash /root/purge_relay_log.sh
在slave 192.168.2.130操做跟上面是同樣的
[root ~]$ cat purge_relay_log.sh #!/bin/bash user=root passwd=123456 port=3306 log_dir='/data/masterha/log' work_dir='/data' purge='/usr/local/bin/purge_relay_logs' if [ ! -d $log_dir ] then mkdir $log_dir -p fi $purge --user=$user --password=$passwd --disable_relay_log_purge --port=$port --workdir=$work_dir >> $log_dir/purge_relay_logs.log 2>&1 [root ~]$ crontab -l 4 * * * /bin/bash /root/purge_relay_log.sh
參數說明
--user mysql //用戶名 --password mysql //密碼 --port //端口號 --workdir //指定建立relay log的硬連接的位置，默認是/var/tmp，因爲系統不一樣分區建立硬連接文件會失敗，故須要執行硬連接具體位置，成功執行腳本後，硬連接的中繼日誌文件被刪除 --disable_relay_log_purge //默認狀況下，若是relay_log_purge=1，腳本會什麼都不清理，自動退出，經過設定這個參數，當relay_log_purge=1的狀況下會將relay_log_purge設置爲0。清理relay log以後，最後將參數設置爲OFF。
purge_relay_logs腳本刪除中繼日誌不會阻塞SQL線程。下面咱們手動執行看看什麼狀況:
[root ~]$ purge_relay_logs --user=root --password=123456 --port=3306 -disable_relay_log_purge --workdir=/data/ 2015-01-18 12:30:51: purge_relay_logs script started. Found relay_log.info: /data/mysql/relay-log.info Removing hard linked relay log files localhost-relay-bin* under /data/.. done. Current relay log file: /data/mysql/localhost-relay-bin.000002 Archiving unused relay log files (up to /data/mysql/localhost-relay-bin.000001) ... Creating hard link for /data/mysql/localhost-relay-bin.000001 under /data//localhost-relay-bin.000001 .. ok. Creating hard links for unused relay log files completed. Executing SET GLOBAL relay_log_purge=1; FLUSH LOGS; sleeping a few seconds so that SQL thread can delete older relay log files (if it keeps up); SET GLOBAL relay_log_purge=0; .. ok. Removing hard linked relay log files localhost-relay-bin* under /data/.. done. 2015-01-18 12:30:54: All relay log purging operations succeeded.
6.檢查SSH配置(server01 192.168.2.131 Monitor 監控節點上操做)，以下：
[root ~]$ masterha_check_ssh --conf=/etc/masterha/app1.cnf Sun Jan 18 12:31:48 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Jan 18 12:31:48 2015 - [info] Reading application default configurations from /etc/masterha/app1.cnf.. Sun Jan 18 12:31:48 2015 - [info] Reading server configurations from /etc/masterha/app1.cnf.. Sun Jan 18 12:31:48 2015 - [info] Starting SSH connection tests.. Sun Jan 18 12:31:49 2015 - [debug] Sun Jan 18 12:31:48 2015 - [debug] Connecting via SSH from root@192.168.2.128(192.168.2.128:22 ) to root@192.168.2.129(192.168.2.129:22 ).. Sun Jan 18 12:31:49 2015 - [debug] ok. Sun Jan 18 12:31:49 2015 - [debug] Connecting via SSH from root@192.168.2.128(192.168.2.128:22 ) to root@192.168.2.130(192.168.2.130:22 ).. Sun Jan 18 12:31:49 2015 - [debug] ok. Sun Jan 18 12:31:50 2015 - [debug] Sun Jan 18 12:31:49 2015 - [debug] Connecting via SSH from root@192.168.2.129(192.168.2.129:22 ) to root@192.168.2.128(192.168.2.128:22 ).. Sun Jan 18 12:31:49 2015 - [debug] ok. Sun Jan 18 12:31:49 2015 - [debug] Connecting via SSH from root@192.168.2.129(192.168.2.129:22 ) to root@192.168.2.130(192.168.2.130:22 ).. Sun Jan 18 12:31:50 2015 - [debug] ok. Sun Jan 18 12:31:50 2015 - [debug] Sun Jan 18 12:31:49 2015 - [debug] Connecting via SSH from root@192.168.2.130(192.168.2.130:22 ) to root@192.168.2.128(192.168.2.128:22 ).. Sun Jan 18 12:31:50 2015 - [debug] ok. Sun Jan 18 12:31:50 2015 - [debug] Connecting via SSH from root@192.168.2.130(192.168.2.130:22 ) to root@192.168.2.129(192.168.2.129:22 ).. Sun Jan 18 12:31:50 2015 - [debug] ok. Sun Jan 18 12:31:50 2015 - [info] All SSH connection tests passed successfully.
能夠看見各個節點ssh驗證都是ok的。
7.檢查整個複製環境情況(server01 192.168.2.131 Monitor 監控節點上操做)，以下：
[root ~]$ masterha_check_repl --conf=/etc/masterha/app1.cnf Sun Jan 18 13:08:11 2015 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql --output_file=/tmp/save_binary_logs_test --manager_version=0.56 --start_file=mysql-bin.000004 Sun Jan 18 13:08:11 2015 - [info] Connecting to root@192.168.2.128(192.168.2.128).. Creating /tmp if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /data/mysql, up to mysql-bin.000004 Sun Jan 18 13:08:11 2015 - [info] Master setting check done. Sun Jan 18 13:08:11 2015 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers.. Sun Jan 18 13:08:11 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=root --slave_host=192.168.2.129 --slave_ip=192.168.2.129 --slave_port=3306 --workdir=/tmp --target_version=5.5.60-log --manager_version=0.56 --relay_log_info=/data/mysql/relay-log.info --relay_dir=/data/mysql/ --slave_pass=xxx Sun Jan 18 13:08:11 2015 - [info] Connecting to root@192.168.2.129(192.168.2.129:22).. Can't exec "mysqlbinlog": 沒有那個文件或目錄 at /usr/local/share/perl5/MHA/BinlogManager.pm line 99. mysqlbinlog version not found! at /usr/local/bin/apply_diff_relay_logs line 463 Sun Jan 18 13:08:12 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln193] Slaves settings check failed! Sun Jan 18 13:08:12 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln372] Slave configuration failed. Sun Jan 18 13:08:12 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln383] Error happend on checking configurations. at /usr/local/bin/masterha_check_repl line 48 Sun Jan 18 13:08:12 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln478] Error happened on monitoring servers. Sun Jan 18 13:08:12 2015 - [info] Got exit code 1 (Not master dead). MySQL Replication Health is NOT OK! 192.168.2.131 [root ~]$
若是發現以下錯誤：
能夠經過如下方法解決（在全部節點上執行192.168.2.128 192.168.2.129 192.168.2.130）：
[root ~]$ ln -s /usr/local/mysql/bin/mysqlbinlog /usr/local/bin/mysqlbinlog [root ~]$ ln -s /usr/local/mysql/bin/mysql /usr/local/bin/mysql [root ~]$ ln -s /usr/local/mysql/bin/mysqlbinlog /usr/local/bin/mysqlbinlog [root ~]$ ln -s /usr/local/mysql/bin/mysql /usr/local/bin/mysql [root ~]$ ln -s /usr/local/mysql/bin/mysqlbinlog /usr/local/bin/mysqlbinlog [root ~]$ ln -s /usr/local/mysql/bin/mysql /usr/local/bin/mysql
再進行檢查(server01 192.168.2.131 Monitor 監控節點上操做)，
[root ~]$ masterha_check_repl --conf=/etc/masterha/app1.cnf Sun Jan 18 13:19:41 2015 - [info] Checking replication health on 192.168.2.129.. Sun Jan 18 13:19:41 2015 - [info] ok. Sun Jan 18 13:19:41 2015 - [info] Checking replication health on 192.168.2.130.. Sun Jan 18 13:19:41 2015 - [info] ok. Sun Jan 18 13:19:41 2015 - [info] Checking master_ip_failover_script status: Sun Jan 18 13:19:41 2015 - [info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.2.128 --orig_master_ip=192.168.2.128 --orig_master_port=3306 Bareword "FIXME_xxx" not allowed while "strict subs" in use at /usr/local/bin/master_ip_failover line 88. Execution of /usr/local/bin/master_ip_failover aborted due to compilation errors. Sun Jan 18 13:19:41 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln214] Failed to get master_ip_failover_script status with return code 255:0. Sun Jan 18 13:19:41 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln383] Error happend on checking configurations. at /usr/local/bin/masterha_check_repl line 48 Sun Jan 18 13:19:41 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln478] Error happened on monitoring servers. Sun Jan 18 13:19:41 2015 - [info] Got exit code 1 (Not master dead). MySQL Replication Health is NOT OK! 192.168.2.131 [root ~]$
仍是報錯，發現緣由是：原來Failover兩種方式：一種是虛擬IP地址，一種是全局配置文件。MHA並無限定使用哪種方式，而是讓用戶本身選擇，虛擬IP地址的方式會牽扯到其它的軟件,好比keepalive軟件，並且還要修改腳本master_ip_failover。
因此先暫時註釋master_ip_failover_script= /usr/local/bin/master_ip_failover這個選項。後面引入keepalived後和修改該腳本之後再開啓該選項。
[root ~]$ grep master_ip_failover /etc/masterha/app1.cnf #master_ip_failover_script= /usr/local/bin/master_ip_failover
再次進行狀態查看，(server01 192.168.2.131 Monitor 監控節點上操做)，
[root ~]$ masterha_check_repl --conf=/etc/masterha/app1.cnf Sun Jan 18 13:23:57 2015 - [info] Slaves settings check done. Sun Jan 18 13:23:57 2015 - [info] 192.168.2.128 (current master) +--192.168.2.129 +--192.168.2.130 Sun Jan 18 13:23:57 2015 - [info] Checking replication health on 192.168.2.129.. Sun Jan 18 13:23:57 2015 - [info] ok. Sun Jan 18 13:23:57 2015 - [info] Checking replication health on 192.168.2.130.. Sun Jan 18 13:23:57 2015 - [info] ok. Sun Jan 18 13:23:57 2015 - [warning] master_ip_failover_script is not defined. Sun Jan 18 13:23:57 2015 - [warning] shutdown_script is not defined. Sun Jan 18 13:23:57 2015 - [info] Got exit code 0 (Not master dead). MySQL Replication Health is OK.
已經沒有明顯報錯，只有兩個警告而已，複製也顯示正常了。
8.檢查MHA Manager的狀態
經過master_check_status腳本查看Manager的狀態：
[root ~]$ masterha_check_status --conf=/etc/masterha/app1.cnf app1 is stopped(2:NOT_RUNNING).
注意：若是正常，會顯示"PING_OK"，不然會顯示"NOT_RUNNING"，這表明MHA監控沒有開啓。
9.開啓MHA Manager監控(server01 192.168.2.131操做)以下：
[root ~]$ mkdir -p /var/log/masterha/app1/ [root ~]$ nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 &
啓動參數說明
--remove_dead_master_conf //該參數表明當發生主從切換後，老的主庫的ip將會從配置文件中移除。 --manger_log //日誌存放位置 --ignore_last_failover //在缺省狀況下，若是MHA檢測到連續發生宕機，且兩次宕機間隔不足8小時的話，則不會進行Failover，之因此這樣限制是爲了不ping-pong效應。該參數表明忽略上次MHA觸發切換產生的文件，默認狀況下，MHA發生切換後會在日誌目錄，也就是上面我設置的/data產生app1.failover.complete文件，下次再次切換的時候若是發現該目錄下存在該文件將不容許觸發切換，除非在第一次切換後收到刪除該文件，爲了方便，這裏設置爲--ignore_last_failover。
查看MHA Manager監控是否正常：（server01 192.168.2.131操做）
[root ~]$ masterha_check_status --conf=/etc/masterha/app1.cnf app1 (pid:13014) is running(0:PING_OK), master:192.168.2.128
能夠看見已經在監控了，並且master的主機爲192.168.2.128
10.查看啓動日誌(server01 192.168.2.131操做)以下：
[root ~]$ tail -n 20 /var/log/masterha/app1/manager.log Sun Jan 18 13:27:22 2015 - [info] Connecting to root@192.168.2.130(192.168.2.130:22).. Checking slave recovery environment settings.. Opening /data/mysql/relay-log.info ... ok. Relay log found at /data/mysql, up to localhost-relay-bin.000002 Temporary relay log file is /data/mysql/localhost-relay-bin.000002 Testing mysql connection and privileges.. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sun Jan 18 13:27:22 2015 - [info] Slaves settings check done. Sun Jan 18 13:27:22 2015 - [info] 192.168.2.128 (current master) +--192.168.2.129 +--192.168.2.130 Sun Jan 18 13:27:22 2015 - [warning] master_ip_failover_script is not defined. Sun Jan 18 13:27:22 2015 - [warning] shutdown_script is not defined. Sun Jan 18 13:27:22 2015 - [info] Set master ping interval 1 seconds. Sun Jan 18 13:27:22 2015 - [info] Set secondary check script: /usr/local/bin/masterha_secondary_check -s server03 -s server02 --user=root --master_host=server02 --master_ip=192.168.2.128 --master_port=3306 Sun Jan 18 13:27:22 2015 - [info] Starting ping health check on 192.168.2.128(192.168.2.128:3306).. Sun Jan 18 13:27:22 2015 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
其中"Ping(SELECT) succeeded, waiting until MySQL doesn't respond.."說明整個系統開始監控了。
11.關閉MHA Manage監控(server01 192.168.2.131操做)以下：
關閉很簡單，使用masterha_stop命令完成。（只是演示關閉，在測試中，必須是開啓的狀態，若是關了，在測試的時候務必記得開啓）
$ masterha_stop --conf=/etc/masterha/app1.cnf Stopped app1 successfully. [1]+ Exit nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1
12.配置VIP
vip配置能夠採用兩種方式，
一種經過keepalived的方式管理虛擬ip的浮動；
另一種經過腳本方式啓動虛擬ip的方式（即不須要keepalived或者heartbeat相似的軟件）。
下面先介紹經過安裝keepalived來管理虛擬IP的浮動：
（1）下載軟件進行並進行安裝（兩臺master，準確的說一臺是master，另一臺是備選master，在沒有切換之前是slave）server02 192.168.2.128操做：
[root ~]$ wget [root ~]$ tar xf keepalived-1.2.12.tar.gz [root ~]$ cd keepalived-1.2.12 [root keepalived-1.2.12]$ ./configure --prefix=/usr/local/keepalived [root keepalived-1.2.12]$ make && make install [root keepalived-1.2.12]$ cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/ [root keepalived-1.2.12]$ cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/ [root keepalived-1.2.12]$ mkdir /etc/keepalived [root keepalived-1.2.12]$ cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/ [root keepalived-1.2.12]$ cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
server03 192.168.2.129也要執行上面的操做，安裝是同樣的，配置文件不同
[root ~]$ wget [root ~]$ tar xf keepalived-1.2.12.tar.gz [root ~]$ cd keepalived-1.2.12 [root keepalived-1.2.12]$ ./configure --prefix=/usr/local/keepalived [root keepalived-1.2.12]$ make && make install [root keepalived-1.2.12]$ cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/ [root keepalived-1.2.12]$ cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/ [root keepalived-1.2.12]$ mkdir /etc/keepalived [root keepalived-1.2.12]$ cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/ [root keepalived-1.2.12]$ cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
（2）配置keepalived的配置文件，在master上配置（server02 192.168.2.128）操做以下:
[root keepalived-1.2.12]$ cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
saltstack@163.com
}
notification_email_from dba@dbserver.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id MySQL-HA
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 150
advert_int 1
nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.2.88
}
}
其中router_id MySQL HA表示設定keepalived組的名稱，將192.168.2.88這個虛擬ip綁定到該主機的eth0網卡上，而且設置了狀態爲backup模式，將keepalived的模式設置爲非搶佔模式（nopreempt），priority 150表示設置的優先級爲150。（要看清楚本身的網卡是eth0作模擬VIP,仍是eth1）
在候選master上配置（server03 192.168.2.129）操做以下：
[root keepalived-1.2.12]$ cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
saltstack@163.com
}
notification_email_from dba@dbserver.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id MySQL-HA
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 120
advert_int 1
nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.2.88
}
}
（3）啓動keepalived服務，在master上啓動並查看日誌（server02 192.168.2.128）操做以下:
[root keepalived-1.2.12]$ /etc/init.d/keepalived start [root keepalived-1.2.12]$ tail -f /var/log/messages Jan 18 13:47:20 localhost Keepalived_healthcheckers[4638]: Registering Kernel netlink reflector Jan 18 13:47:20 localhost Keepalived_healthcheckers[4638]: Registering Kernel netlink command channel Jan 18 13:47:20 localhost Keepalived_healthcheckers[4638]: Opening file '/etc/keepalived/keepalived.conf'. Jan 18 13:47:20 localhost Keepalived_healthcheckers[4638]: Configuration is using : 7105 Bytes Jan 18 13:47:20 localhost Keepalived_healthcheckers[4638]: Using LinkWatch kernel netlink reflector... Jan 18 13:47:23 localhost Keepalived_vrrp[4639]: VRRP_Instance(VI_1) Transition to MASTER STATE Jan 18 13:47:24 localhost Keepalived_vrrp[4639]: VRRP_Instance(VI_1) Entering MASTER STATE Jan 18 13:47:24 localhost Keepalived_vrrp[4639]: VRRP_Instance(VI_1) setting protocol VIPs. Jan 18 13:47:24 localhost Keepalived_vrrp[4639]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.88 Jan 18 13:47:24 localhost Keepalived_healthcheckers[4638]: Netlink reflector reports IP 192.168.2.88 added Jan 18 13:47:29 localhost Keepalived_vrrp[4639]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.88
啓動候選master的keepalived並查看日誌（server03 192.168.2.129）操做以下：
[root keepalived-1.2.12]$ /etc/init.d/keepalived start [root keepalived-1.2.12]$ tail -f /var/log/messages Jan 18 13:52:31 localhost Keepalived_vrrp[4990]: Registering gratuitous ARP shared channel Jan 18 13:52:31 localhost Keepalived_healthcheckers[4989]: Registering Kernel netlink command channel Jan 18 13:52:31 localhost Keepalived_vrrp[4990]: Opening file '/etc/keepalived/keepalived.conf'. Jan 18 13:52:31 localhost Keepalived_healthcheckers[4989]: Opening file '/etc/keepalived/keepalived.conf'. Jan 18 13:52:31 localhost Keepalived_healthcheckers[4989]: Configuration is using : 7105 Bytes Jan 18 13:52:31 localhost Keepalived_healthcheckers[4989]: Using LinkWatch kernel netlink reflector... Jan 18 13:52:31 localhost Keepalived_vrrp[4990]: Configuration is using : 62850 Bytes Jan 18 13:52:31 localhost Keepalived_vrrp[4990]: Using LinkWatch kernel netlink reflector... Jan 18 13:52:31 localhost Keepalived_vrrp[4990]: VRRP_Instance(VI_1) Entering BACKUP STATE Jan 18 13:52:31 localhost Keepalived_vrrp[4990]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)] Jan 18 13:52:34 localhost Keepalived_vrrp[4990]: VRRP_Instance(VI_1) Transition to MASTER STATE Jan 18 13:52:35 localhost Keepalived_vrrp[4990]: VRRP_Instance(VI_1) Entering MASTER STATE Jan 18 13:52:35 localhost Keepalived_vrrp[4990]: VRRP_Instance(VI_1) setting protocol VIPs. Jan 18 13:52:35 localhost Keepalived_vrrp[4990]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.88 Jan 18 13:52:35 localhost Keepalived_healthcheckers[4989]: Netlink reflector reports IP 192.168.2.88 added Jan 18 13:52:40 localhost Keepalived_vrrp[4990]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.88
（4）查看綁定狀況（server02 192.168.2.128）操做以下
[root keepalived-1.2.12]$ ip address eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:86:dc:2a brd ff:ff:ff:ff:ff:ff inet 192.168.2.128/24 brd 192.168.2.255 scope global eth0 inet 192.168.2.88/32 scope global eth0 inet6 fe80::20c:29ff:fe86:dc2a/64 scope link valid_lft forever preferred_lft forever 192.168.2.128 [root keepalived-1.2.12]$
發現已經將虛擬IP 192.168.2.88綁定了master02 192.168.2.128的網卡eth0上了
從上面的信息能夠看到keepalived已經配置成功。
注意：
上面兩臺服務器的keepalived都設置爲了BACKUP模式，
在keepalived中2種模式，分別master->backup模式和backup->backup模式。這兩種模式有很大區別。在master->backup模式下，一旦主庫宕機，虛擬ip會自動漂移到從庫，當主庫修復後，keepalived啓動後，還會把虛擬ip搶佔過來，即便設置了非搶佔模式（nopreempt）搶佔ip的動做也會發生。
在backup->backup模式下，當主庫宕機後虛擬ip會自動漂移到從庫上，當原主庫恢復和keepalived服務啓動後，並不會搶佔新主的虛擬ip，即便是優先級高於從庫的優先級別，也不會發生搶佔。爲了減小ip漂移次數，一般是把修復好的主庫當作新的備庫。
（5）MHA引入keepalived（MySQL服務進程掛掉時經過MHA 中止keepalived）:
要想把keepalived服務引入MHA，咱們只須要修改切換是觸發的腳本文件master_ip_failover便可，在該腳本中添加在master發生宕機時對keepalived的處理。
一、編輯腳本/usr/local/bin/master_ip_failover，修改後以下（server01 192.168.2.131）操做：
[root ~]$ cat /usr/local/bin/master_ip_failover
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port
);
my $vip = '192.168.2.88';
my $ssh_start_vip = "/etc/init.d/keepalived start";
my $ssh_stop_vip = "/etc/init.d/keepalived stop";
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);
exit &main();
sub main {
print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";
if ( $command eq "stop" || $command eq "stopssh" ) {
my $exit_code = 1;
eval {
print "Disabling the VIP on old master: $orig_master_host \n";
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
exit 0;
}
else {
&usage();
exit 1;
}
}
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}
把#master_ip_failover_script= /usr/local/bin/master_ip_failover打開
[root ~]$ grep 'master_ip_failover_script' /etc/masterha/app1.cnf master_ip_failover_script= /usr/local/bin/master_ip_failover
執行檢測（(server01 192.168.2.131 Monitor 監控節點上操做）：
[root ~]$ masterha_check_repl --conf=/etc/masterha/app1.cnf Sun Jan 18 14:02:21 2015 - [info] Slaves settings check done. Sun Jan 18 14:02:21 2015 - [info] 192.168.2.128 (current master) +--192.168.2.129 +--192.168.2.130 Sun Jan 18 14:02:21 2015 - [info] Checking replication health on 192.168.2.129.. Sun Jan 18 14:02:21 2015 - [info] ok. Sun Jan 18 14:02:21 2015 - [info] Checking replication health on 192.168.2.130.. Sun Jan 18 14:02:21 2015 - [info] ok. Sun Jan 18 14:02:21 2015 - [info] Checking master_ip_failover_script status: Sun Jan 18 14:02:21 2015 - [info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.2.128 --orig_master_ip=192.168.2.128 --orig_master_port=3306 IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start=== Checking the Status of the script.. OK Sun Jan 18 14:02:21 2015 - [info] OK. Sun Jan 18 14:02:21 2015 - [warning] shutdown_script is not defined. Sun Jan 18 14:02:21 2015 - [info] Got exit code 0 (Not master dead). MySQL Replication Health is OK.
能夠看見沒有報錯。
/usr/local/bin/master_ip_failover添加或者修改的內容意思是當主庫數據庫發生故障時，會觸發MHA切換，MHA Manager會停掉主庫上的keepalived服務，觸發虛擬ip漂移到備選從庫，從而完成切換。固然能夠在keepalived裏面引入腳本，這個腳本監控mysql是否正常運行，若是不正常，則調用該腳本殺掉keepalived進程。
二、如下進行模擬主Master（192.168.2.128）down了：
[root keepalived-1.2.12]$ /etc/init.d/mysqld stop Shutting down MySQL... SUCCESS!
在管理節點(server01 192.168.2.131)查看日誌：（報錯）
[root ~]$ tail -f /var/log/masterha/app1/manager.log 192.168.2.128 (current master) +--192.168.2.129 +--192.168.2.130 Sun Jan 18 13:32:37 2015 - [warning] master_ip_failover_script is not defined. Sun Jan 18 13:32:37 2015 - [warning] shutdown_script is not defined. Sun Jan 18 13:32:37 2015 - [info] Set master ping interval 1 seconds. Sun Jan 18 13:32:37 2015 - [info] Set secondary check script: /usr/local/bin/masterha_secondary_check -s server03 -s server02 --user=root --master_host=server02 --master_ip=192.168.2.128 --master_port=3306 Sun Jan 18 13:32:37 2015 - [info] Starting ping health check on 192.168.2.128(192.168.2.128:3306).. Sun Jan 18 13:32:37 2015 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond.. Sun Jan 18 14:32:03 2015 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away) Sun Jan 18 14:32:03 2015 - [info] Executing seconary network check script: /usr/local/bin/masterha_secondary_check -s server03 -s server02 --user=root --master_host=server02 --master_ip=192.168.2.128 --master_port=3306 --user=root --master_host=192.168.2.128 --master_ip=192.168.2.128 --master_port=3306 Sun Jan 18 14:32:03 2015 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql --output_file=/tmp/save_binary_logs_test --manager_version=0.56 --binlog_prefix=mysql-bin Sun Jan 18 14:32:03 2015 - [info] HealthCheck: SSH to 192.168.2.128 is reachable. Sun Jan 18 14:32:04 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111) Sun Jan 18 14:32:04 2015 - [warning] Connection failed 1 time(s).. Sun Jan 18 14:32:05 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111) Sun Jan 18 14:32:05 2015 - [warning] Connection failed 2 time(s).. Sun Jan 18 14:32:06 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111) Sun Jan 18 14:32:06 2015 - [warning] Connection failed 3 time(s).. ssh: Could not resolve hostname server03: Name or service not known Monitoring server server03 is NOT reachable!
在管理節服務器192.168.2.131上添加hosts:
[root ~]$ cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.2.128 server01 192.168.2.129 server02 192.168.2.130 server03
再查看日誌（server01 192.168.2.131）：
[root ~]$ tail -f /var/log/masterha/app1/manager.log IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start=== Checking the Status of the script.. OK Sun Jan 18 17:11:19 2015 - [info] OK. Sun Jan 18 17:11:19 2015 - [warning] shutdown_script is not defined. Sun Jan 18 17:11:19 2015 - [info] Set master ping interval 1 seconds. Sun Jan 18 17:11:19 2015 - [info] Set secondary check script: /usr/local/bin/masterha_secondary_check -s server03 -s server02 --user=root --master_host=server02 --master_ip=192.168.2.128 --master_port=3306 Sun Jan 18 17:11:19 2015 - [info] Starting ping health check on 192.168.2.128(192.168.2.128:3306).. Sun Jan 18 17:11:19 2015 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond.. Sun Jan 18 17:11:48 2015 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away) Sun Jan 18 17:11:48 2015 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql --output_file=/tmp/save_binary_logs_test --manager_version=0.53 --binlog_prefix=mysql-bin Sun Jan 18 17:11:48 2015 - [info] Executing seconary network check script: /usr/local/bin/masterha_secondary_check -s server03 -s server02 --user=root --master_host=server02 --master_ip=192.168.2.128 --master_port=3306 --user=root --master_host=192.168.2.128 --master_ip=192.168.2.128 --master_port=3306 Sun Jan 18 17:11:48 2015 - [info] HealthCheck: SSH to 192.168.2.128 is reachable. Monitoring server server03 is reachable, Master is not reachable from server03. OK. Monitoring server server02 is reachable, Master is not reachable from server02. OK. Sun Jan 18 17:11:48 2015 - [info] Master is not reachable from all other monitoring servers. Failover should start. Sun Jan 18 17:11:49 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111) Sun Jan 18 17:11:49 2015 - [warning] Connection failed 1 time(s).. Sun Jan 18 17:11:50 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111) Sun Jan 18 17:11:50 2015 - [warning] Connection failed 2 time(s).. Sun Jan 18 17:11:51 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111) Sun Jan 18 17:11:51 2015 - [warning] Connection failed 3 time(s).. Sun Jan 18 17:11:51 2015 - [warning] Master is not reachable from health checker! Sun Jan 18 17:11:51 2015 - [warning] Master 192.168.2.128(192.168.2.128:3306) is not reachable! Sun Jan 18 17:11:51 2015 - [warning] SSH is reachable. Sun Jan 18 17:11:51 2015 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/masterha/app1.cnf again, and trying to connect to all servers to check server status.. Sun Jan 18 17:11:51 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Jan 18 17:11:51 2015 - [info] Reading application default configurations from /etc/masterha/app1.cnf.. Sun Jan 18 17:11:51 2015 - [info] Reading server configurations from /etc/masterha/app1.cnf.. Sun Jan 18 17:11:51 2015 - [info] Dead Servers: Sun Jan 18 17:11:51 2015 - [info] 192.168.2.128(192.168.2.128:3306) Sun Jan 18 17:11:51 2015 - [info] Alive Servers: Sun Jan 18 17:11:51 2015 - [info] 192.168.2.129(192.168.2.129:3306) Sun Jan 18 17:11:51 2015 - [info] 192.168.2.130(192.168.2.130:3306) Sun Jan 18 17:11:51 2015 - [info] Alive Slaves: Sun Jan 18 17:11:51 2015 - [info] 192.168.2.129(192.168.2.129:3306) Version=5.5.30-log (oldest major version between slaves) log-bin:enabled Sun Jan 18 17:11:51 2015 - [info] Replicating from 192.168.2.128(192.168.2.128:3306) Sun Jan 18 17:11:51 2015 - [info] Primary candidate for the new Master (candidate_master is set) Sun Jan 18 17:11:51 2015 - [info] 192.168.2.130(192.168.2.130:3306) Version=5.5.25-log (oldest major version between slaves) log-bin:enabled Sun Jan 18 17:11:51 2015 - [info] Replicating from 192.168.2.128(192.168.2.128:3306) Sun Jan 18 17:11:51 2015 - [info] Checking slave configurations.. Sun Jan 18 17:11:51 2015 - [info] read_only=1 is not set on slave 192.168.2.129(192.168.2.129:3306). Sun Jan 18 17:11:51 2015 - [warning] relay_log_purge=0 is not set on slave 192.168.2.129(192.168.2.129:3306). Sun Jan 18 17:11:51 2015 - [warning] relay_log_purge=0 is not set on slave 192.168.2.130(192.168.2.130:3306). Sun Jan 18 17:11:51 2015 - [info] Checking replication filtering settings.. Sun Jan 18 17:11:51 2015 - [info] Replication filtering check ok. Sun Jan 18 17:11:51 2015 - [info] Master is down! Sun Jan 18 17:11:51 2015 - [info] Terminating monitoring script. Sun Jan 18 17:11:51 2015 - [info] Got exit code 20 (Master dead). Sun Jan 18 17:11:51 2015 - [info] MHA::MasterFailover version 0.53. Sun Jan 18 17:11:51 2015 - [info] Starting master failover. Sun Jan 18 17:11:51 2015 - [info] Sun Jan 18 17:11:51 2015 - [info] * Phase 1: Configuration Check Phase.. Sun Jan 18 17:11:51 2015 - [info] Sun Jan 18 17:11:51 2015 - [info] Dead Servers: Sun Jan 18 17:11:51 2015 - [info] 192.168.2.128(192.168.2.128:3306) Sun Jan 18 17:11:51 2015 - [info] Checking master reachability via mysql(double check).. Sun Jan 18 17:11:51 2015 - [info] ok. Sun Jan 18 17:11:51 2015 - [info] Alive Servers: Sun Jan 18 17:11:51 2015 - [info] 192.168.2.129(192.168.2.129:3306) Sun Jan 18 17:11:51 2015 - [info] 192.168.2.130(192.168.2.130:3306) Sun Jan 18 17:11:51 2015 - [info] Alive Slaves: Sun Jan 18 17:11:51 2015 - [info] 192.168.2.129(192.168.2.129:3306) Version=5.5.30-log (oldest major version between slaves) log-bin:enabled Sun Jan 18 17:11:51 2015 - [info] Replicating from 192.168.2.128(192.168.2.128:3306) Sun Jan 18 17:11:51 2015 - [info] Primary candidate for the new Master (candidate_master is set) Sun Jan 18 17:11:51 2015 - [info] 192.168.2.130(192.168.2.130:3306) Version=5.5.25-log (oldest major version between slaves) log-bin:enabled Sun Jan 18 17:11:51 2015 - [info] Replicating from 192.168.2.128(192.168.2.128:3306) Sun Jan 18 17:11:51 2015 - [info] ** Phase 1: Configuration Check Phase completed. Sun Jan 18 17:11:51 2015 - [info] Sun Jan 18 17:11:51 2015 - [info] * Phase 2: Dead Master Shutdown Phase.. Sun Jan 18 17:11:51 2015 - [info] Sun Jan 18 17:11:51 2015 - [info] Forcing shutdown so that applications never connect to the current master.. Sun Jan 18 17:11:51 2015 - [info] Executing master IP deactivatation script: Sun Jan 18 17:11:51 2015 - [info] /usr/local/bin/master_ip_failover --orig_master_host=192.168.2.128 --orig_master_ip=192.168.2.128 --orig_master_port=3306 --command=stopssh --ssh_user=root Sun Jan 18 17:11:51 2015 - [info] Fetching dead master's binary logs.. Sun Jan 18 17:11:51 2015 - [info] Executing command on the dead master 192.168.2.128(192.168.2.128:3306): save_binary_logs --command=save --start_file=mysql-bin.000014 --start_pos=107 --binlog_dir=/data/mysql --output_file=/tmp/saved_master_binlog_from_192.168.2.128_3306_20150118171151.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.53 Creating /tmp if not exists.. ok. Concat binary/relay logs from mysql-bin.000014 pos 107 to mysql-bin.000014 EOF into /tmp/saved_master_binlog_from_192.168.2.128_3306_20150118171151.binlog .. Dumping binlog format description event, from position 0 to 107.. ok. Dumping effective binlog data from /data/mysql/mysql-bin.000014 position 107 to tail(126).. ok. Concat succeeded. Sun Jan 18 17:11:52 2015 - [info] scp from root@192.168.2.128:/tmp/saved_master_binlog_from_192.168.2.128_3306_20150118171151.binlog to local:/var/log/masterha/app1.log/saved_master_binlog_from_192.168.2.128_3306_20150118171151.binlog succeeded. Sun Jan 18 17:11:52 2015 - [info] HealthCheck: SSH to 192.168.2.129 is reachable. Sun Jan 18 17:11:52 2015 - [info] HealthCheck: SSH to 192.168.2.130 is reachable. Sun Jan 18 17:11:53 2015 - [info] Sun Jan 18 17:11:53 2015 - [info] * Phase 3.3: Determining New Master Phase.. Sun Jan 18 17:11:53 2015 - [info] Sun Jan 18 17:11:53 2015 - [info] Finding the latest slave that has all relay logs for recovering other slaves.. Sun Jan 18 17:11:53 2015 - [info] All slaves received relay logs to the same position. No need to resync each other. Sun Jan 18 17:11:53 2015 - [info] Searching new master from slaves.. Sun Jan 18 17:11:53 2015 - [info] Candidate masters from the configuration file: Sun Jan 18 17:11:53 2015 - [info] 192.168.2.129(192.168.2.129:3306) Version=5.5.30-log (oldest major version between slaves) log-bin:enabled Sun Jan 18 17:11:53 2015 - [info] Replicating from 192.168.2.128(192.168.2.128:3306) Sun Jan 18 17:11:53 2015 - [info] Primary candidate for the new Master (candidate_master is set) Sun Jan 18 17:11:53 2015 - [info] Non-candidate masters: Sun Jan 18 17:11:53 2015 - [info] Searching from candidate_master slaves which have received the latest relay log events.. Sun Jan 18 17:11:53 2015 - [info] New master is 192.168.2.129(192.168.2.129:3306) Sun Jan 18 17:11:53 2015 - [info] Starting master failover.. Sun Jan 18 17:11:53 2015 - [info] From: 192.168.2.128 (current master) +--192.168.2.129 +--192.168.2.130 To: 192.168.2.129 (new master) +--192.168.2.130 Sun Jan 18 17:11:53 2015 - [info] Sun Jan 18 17:11:53 2015 - [info] * Phase 3.3: New Master Diff Log Generation Phase.. Sun Jan 18 17:11:53 2015 - [info] Sun Jan 18 17:11:53 2015 - [info] This server has all relay logs. No need to generate diff files from the latest slave. Sun Jan 18 17:11:53 2015 - [info] Sending binlog.. Sun Jan 18 17:11:53 2015 - [info] scp from local:/var/log/masterha/app1.log/saved_master_binlog_from_192.168.2.128_3306_20150118171151.binlog to root@192.168.2.129:/tmp/saved_master_binlog_from_192.168.2.128_3306_20150118171151.binlog succeeded. Sun Jan 18 17:11:53 2015 - [info] Sun Jan 18 17:11:53 2015 - [info] * Phase 3.4: Master Log Apply Phase.. Sun Jan 18 17:11:53 2015 - [info] Sun Jan 18 17:11:53 2015 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed. Sun Jan 18 17:11:53 2015 - [info] Starting recovery on 192.168.2.129(192.168.2.129:3306).. Sun Jan 18 17:11:53 2015 - [info] Generating diffs succeeded. Sun Jan 18 17:11:53 2015 - [info] Waiting until all relay logs are applied. Sun Jan 18 17:11:53 2015 - [info] done. Sun Jan 18 17:11:53 2015 - [info] Getting slave status.. Sun Jan 18 17:11:53 2015 - [info] This slave(192.168.2.129)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000014:107). No need to recover from Exec_Master_Log_Pos. Sun Jan 18 17:11:53 2015 - [info] Connecting to the target slave host 192.168.2.129, running recover script.. Sun Jan 18 17:11:53 2015 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user=root --slave_host=192.168.2.129 --slave_ip=192.168.2.129 --slave_port=3306 --apply_files=/tmp/saved_master_binlog_from_192.168.2.128_3306_20150118171151.binlog --workdir=/tmp --target_version=5.5.30-log --timestamp=20150118171151 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.53 --slave_pass=xxx Sun Jan 18 17:11:53 2015 - [info] Applying differential binary/relay log files /tmp/saved_master_binlog_from_192.168.2.128_3306_20150118171151.binlog on 192.168.2.129:3306. This may take long time... Applying log files succeeded. Sun Jan 18 17:11:53 2015 - [info] All relay logs were successfully applied. Sun Jan 18 17:11:53 2015 - [info] Getting new master's binlog name and position.. Sun Jan 18 17:11:53 2015 - [info] mysql-bin.000005:61791 Sun Jan 18 17:11:53 2015 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.2.129', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000005', MASTER_LOG_POS=61791, MASTER_USER='repl', MASTER_PASSWORD='xxx'; Sun Jan 18 17:11:53 2015 - [info] Executing master IP activate script: Sun Jan 18 17:11:53 2015 - [info] /usr/local/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.2.128 --orig_master_ip=192.168.2.128 --orig_master_port=3306 --new_master_host=192.168.2.129 --new_master_ip=192.168.2.129 --new_master_port=3306 IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start=== Enabling the VIP - 192.168.2.88 on the new master - 192.168.2.129 Sun Jan 18 17:11:53 2015 - [info] OK. Sun Jan 18 17:11:53 2015 - [info] ** Finished master recovery successfully. Sun Jan 18 17:11:53 2015 - [info] * Phase 3: Master Recovery Phase completed. Sun Jan 18 17:11:53 2015 - [info] Sun Jan 18 17:11:53 2015 - [info] * Phase 4: Slaves Recovery Phase.. Sun Jan 18 17:11:53 2015 - [info] Sun Jan 18 17:11:53 2015 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase.. Sun Jan 18 17:11:53 2015 - [info] Sun Jan 18 17:11:53 2015 - [info] -- Slave diff file generation on host 192.168.2.130(192.168.2.130:3306) started, pid: 19762. Check tmp log /var/log/masterha/app1.log/192.168.2.130_3306_20150118171151.log if it takes time.. Sun Jan 18 17:11:53 2015 - [info] Sun Jan 18 17:11:53 2015 - [info] Log messages from 192.168.2.130 ... Sun Jan 18 17:11:53 2015 - [info] Sun Jan 18 17:11:53 2015 - [info] This server has all relay logs. No need to generate diff files from the latest slave. Sun Jan 18 17:11:53 2015 - [info] End of log messages from 192.168.2.130. Sun Jan 18 17:11:53 2015 - [info] -- 192.168.2.130(192.168.2.130:3306) has the latest relay log events. Sun Jan 18 17:11:53 2015 - [info] Generating relay diff files from the latest slave succeeded. Sun Jan 18 17:11:53 2015 - [info] Sun Jan 18 17:11:53 2015 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase.. Sun Jan 18 17:11:53 2015 - [info] Sun Jan 18 17:11:53 2015 - [info] -- Slave recovery on host 192.168.2.130(192.168.2.130:3306) started, pid: 19764. Check tmp log /var/log/masterha/app1.log/192.168.2.130_3306_20150118171151.log if it takes time.. Sun Jan 18 17:11:55 2015 - [info] Sun Jan 18 17:11:55 2015 - [info] Log messages from 192.168.2.130 ... Sun Jan 18 17:11:55 2015 - [info] Sun Jan 18 17:11:53 2015 - [info] Sending binlog.. Sun Jan 18 17:11:54 2015 - [info] scp from local:/var/log/masterha/app1.log/saved_master_binlog_from_192.168.2.128_3306_20150118171151.binlog to root@192.168.2.130:/tmp/saved_master_binlog_from_192.168.2.128_3306_20150118171151.binlog succeeded. Sun Jan 18 17:11:54 2015 - [info] Starting recovery on 192.168.2.130(192.168.2.130:3306).. Sun Jan 18 17:11:54 2015 - [info] Generating diffs succeeded. Sun Jan 18 17:11:54 2015 - [info] Waiting until all relay logs are applied. Sun Jan 18 17:11:54 2015 - [info] done. Sun Jan 18 17:11:54 2015 - [info] Getting slave status.. Sun Jan 18 17:11:54 2015 - [info] This slave(192.168.2.130)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000014:107). No need to recover from Exec_Master_Log_Pos. Sun Jan 18 17:11:54 2015 - [info] Connecting to the target slave host 192.168.2.130, running recover script.. Sun Jan 18 17:11:54 2015 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user=root --slave_host=192.168.2.130 --slave_ip=192.168.2.130 --slave_port=3306 --apply_files=/tmp/saved_master_binlog_from_192.168.2.128_3306_20150118171151.binlog --workdir=/tmp --target_version=5.5.25-log --timestamp=20150118171151 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.53 --slave_pass=xxx Sun Jan 18 17:11:54 2015 - [info] Applying differential binary/relay log files /tmp/saved_master_binlog_from_192.168.2.128_3306_20150118171151.binlog on 192.168.2.130:3306. This may take long time... Applying log files succeeded. Sun Jan 18 17:11:54 2015 - [info] All relay logs were successfully applied. Sun Jan 18 17:11:54 2015 - [info] Resetting slave 192.168.2.130(192.168.2.130:3306) and starting replication from the new master 192.168.2.129(192.168.2.129:3306).. Sun Jan 18 17:11:55 2015 - [info] Executed CHANGE MASTER. Sun Jan 18 17:11:55 2015 - [info] Slave started. Sun Jan 18 17:11:55 2015 - [info] End of log messages from 192.168.2.130. Sun Jan 18 17:11:55 2015 - [info] -- Slave recovery on host 192.168.2.130(192.168.2.130:3306) succeeded. Sun Jan 18 17:11:55 2015 - [info] All new slave servers recovered successfully. Sun Jan 18 17:11:55 2015 - [info] Sun Jan 18 17:11:55 2015 - [info] * Phase 5: New master cleanup phease.. Sun Jan 18 17:11:55 2015 - [info] Sun Jan 18 17:11:55 2015 - [info] Resetting slave info on the new master.. Sun Jan 18 17:11:55 2015 - [info] 192.168.2.129: Resetting slave info succeeded. Sun Jan 18 17:11:55 2015 - [info] Master failover to 192.168.2.129(192.168.2.129:3306) completed successfully. Sun Jan 18 17:11:55 2015 - [info] Deleted server1 entry from /etc/masterha/app1.cnf . Sun Jan 18 17:11:55 2015 - [info] ----- Failover Report ----- app1: MySQL Master failover 192.168.2.128 to 192.168.2.129 succeeded Master 192.168.2.128 is down! Check MHA Manager logs at localhost.localdomain:/var/log/masterha/app1/manager.log for details. Started automated(non-interactive) failover. Invalidated master IP address on 192.168.2.128. The latest slave 192.168.2.129(192.168.2.129:3306) has all relay logs for recovery. Selected 192.168.2.129 as a new master. 192.168.2.129: OK: Applying all logs succeeded. 192.168.2.129: OK: Activated master IP address. 192.168.2.130: This host has the latest relay log events. Generating relay diff files from the latest slave succeeded. 192.168.2.130: OK: Applying all logs succeeded. Slave started, replicating from 192.168.2.129. 192.168.2.129: Resetting slave info succeeded. Master failover to 192.168.2.129(192.168.2.129:3306) completed successfully. Sun Jan 18 17:11:55 2015 - [info] Sending mail.. Unknown option: conf
三、在以前的Master（192.168.2.128）上查看一下vip：
[root keepalived-1.2.12]$ ip address eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:86:dc:2a brd ff:ff:ff:ff:ff:ff inet 192.168.2.128/24 brd 192.168.2.255 scope global eth0 inet6 fe80::20c:29ff:fe86:dc2a/64 scope link valid_lft forever preferred_lft forever
能夠看到vip已經不在down的機器上了
去候選的master（server03 192.168.2.129）也就是如今的新master查看是否有vip漂過：
[root keepalived-1.2.12]$ ip a eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:66:95:64 brd ff:ff:ff:ff:ff:ff inet 192.168.2.129/24 brd 192.168.2.255 scope global eth0 inet 192.168.2.88/32 scope global eth0 inet6 fe80::20c:29ff:fe66:9564/64 scope link valid_lft forever preferred_lft forever
看到vip已經成功漂移過來了。
從tail -f /var/log/masterha/app1/manager.log的信息能夠發現最後有這樣的字眼：
The latest slave 192.168.2.129(192.168.2.129:3306) has all relay logs for recovery. Selected 192.168.2.129 as a new master. 192.168.2.129: OK: Applying all logs succeeded. 192.168.2.129: OK: Activated master IP address. 192.168.2.130: This host has the latest relay log events. Generating relay diff files from the latest slave succeeded. 192.168.2.130: OK: Applying all logs succeeded. Slave started, replicating from 192.168.2.129. 192.168.2.129: Resetting slave info succeeded. Master failover to 192.168.2.129(192.168.2.129:3306) completed successfully. Sun Jan 18 17:11:55 2015 - [info] Sending mail.. Unknown option: conf
看到上面的Sending mail了吧，已經正常發郵件了，看圖：
發郵件的設置要在監控節點192.168.2.131上操做：
[root bin]$ cat /etc/masterha/app1.cnf |grep "report_script" report_script=/usr/local/bin/send_report
send_report這個腳本在安裝好軟件後就會有，但這些腳本有不少地方不夠完善，包括send_report的發郵件腳本，下面說明發送郵件設置，並把代碼share出來：
腳本代碼：（該腳本是37wan DBA-鄧亞運分享，博客地址在博文後面貼出）
#!/usr/bin/perl
## Note: This is a sample script and is not complete. Modify the script based on your environment.
use strict;
use warnings FATAL => 'all';
use Mail::Sender;
use Getopt::Long;
#new_master_host and new_slave_hosts are set only when recovering master succeeded
my ( $dead_master_host, $new_master_host, $new_slave_hosts, $subject, $body );
my $smtp='smtp.163.com';
my $mail_from='xxxxxxx@163.com';
my $mail_user='xxxxxxx@163.com';
my $mail_pass='Password';
my $mail_to=['1509773564@qq.com','15521xxxx@139.com'];
GetOptions(
'orig_master_host=s' => \$dead_master_host,
'new_master_host=s' => \$new_master_host,
'new_slave_hosts=s' => \$new_slave_hosts,
'subject=s' => \$subject,
'body=s' => \$body,
);
mailToContacts($smtp,$mail_from,$mail_user,$mail_pass,$mail_to,$subject,$body);
sub mailToContacts {
my ( $smtp, $mail_from, $user, $passwd, $mail_to, $subject, $msg ) = @_;
open my $DEBUG, "> /tmp/monitormail.log"
or die "Can't open the debug file:$!\n";
my $sender = new Mail::Sender {
ctype => 'text/plain; charset=utf-8',
encoding => 'utf-8',
smtp => $smtp,
from => $mail_from,
auth => 'LOGIN',
TLS_allowed => '0',
authid => $user,
authpwd => $passwd,
to => $mail_to,
subject => $subject,
debug => $DEBUG
};
$sender->MailMsg(
{ msg => $msg,
debug => $DEBUG
}
) or print $Mail::Sender::Error;
return 1;
}
# Do whatever you want here
exit 0;
四、在管理節點查看一下配置文件/etc/masterha/app1.cnf能夠發現[server1]的內容已經被自動去掉了（server01 192.168.2.131操做）：
[root ~]$ cat /etc/masterha/app1.cnf [server default] manager_log=/var/log/masterha/app1/manager.log manager_workdir=/var/log/masterha/app1.log master_binlog_dir=/data/mysql master_ip_failover_script=/usr/local/bin/master_ip_failover master_ip_online_change_script=/usr/local/bin/master_ip_online_change password=123456 ping_interval=1 remote_workdir=/tmp repl_password=123456 repl_user=repl report_script=/usr/local/bin/send_report secondary_check_script=/usr/local/bin/masterha_secondary_check -s server03 -s server02 --user=root --master_host=server02 --master_ip=192.168.2.128 --master_port=3306 shutdown_script="" ssh_user=root user=root [server2] candidate_master=1 check_repl_delay=0 hostname=192.168.2.129 port=3306 [server3] hostname=192.168.2.130 port=3306
（2）經過腳本的方式管理VIP。這裏是修改/usr/local/bin/master_ip_failover，修改完成後內容以下，並且若是使用腳本管理vip的話，須要手動在master服務器上綁定一個vip
（server02 192.168.2.128操做查看）
[root ~]$ /sbin/ifconfig eth0:1 192.168.2.88/24 [root ~]$ ifconfig eth0 Link encap:Ethernet HWaddr 00:0C:29:86:DC:2A inet addr:192.168.2.128 Bcast:192.168.2.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fe86:dc2a/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:41643 errors:0 dropped:0 overruns:0 frame:0 TX packets:24696 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:31624443 (30.1 MiB) TX bytes:3388815 (3.2 MiB) eth0:1 Link encap:Ethernet HWaddr 00:0C:29:86:DC:2A inet addr:192.168.2.88 Bcast:192.168.2.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
在管理節點（server01 192.168.2.131操做）修改下/usr/local/bin/master_ip_failover腳本，以下：
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port
);
my $vip = '192.168.2.88';
my $key = '1';
my $ssh_start_vip = "/sbin/ifconfig eth1:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig eth1:$key down";
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);
exit &main();
sub main {
print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";
if ( $command eq "stop" || $command eq "stopssh" ) {
my $exit_code = 1;
eval {
print "Disabling the VIP on old master: $orig_master_host \n";
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
exit 0;
}
else {
&usage();
exit 1;
}
}
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}
（1）在slave庫（192.168.2.129上面操做）上停掉slave IO線程，模擬主從延時
mysql> stop slave io_thread;
（2）在master庫（192.168.2.128上面操做）安裝sysbench，進行sysbench數據生成，在sbtest庫下生成sbtest表，共10W記錄。
[root ~]$ yum install sysbench -y [root ~]$ mysql -uroot -p123456 -e "create database sbtest;" [root ~]$ sysbench --test=oltp --oltp-table-size=100000 --oltp-read-only=off --init-rng=on --num-threads=1 --max-requests=0 --oltp-dist-type=uniform --max-time=1800 --mysql-user=root --mysql-socket=/tmp/mysql.sock --mysql-password=123456 --db-driver=mysql --mysql-table-engine=innodb --oltp-test-mode=complex prepare
另一臺slave咱們沒有中止io線程，因此還在繼續接收日誌。
（3）在slave庫（192.168.2.129上面操做）開啓slave IO線程：
mysql> start slave io_thread;
（4）停掉master庫（192.168.2.128上面操做）操做以下：
[root ~]$ /etc/init.d/mysqld stop Shutting down MySQL.... SUCCESS!
（5）在管理節點（192.168.2.131上面操做）查看日誌：
[root ~]$ tail -f /var/log/masterha/app1/manager.log IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start=== Checking the Status of the script.. OK Sun Jan 18 17:48:32 2015 - [info] OK. Sun Jan 18 17:48:32 2015 - [warning] shutdown_script is not defined. Sun Jan 18 17:48:32 2015 - [info] Set master ping interval 1 seconds. Sun Jan 18 17:48:32 2015 - [info] Set secondary check script: /usr/local/bin/masterha_secondary_check -s server03 -s server02 --user=root --master_host=server02 --master_ip=192.168.2.128 --master_port=3306 Sun Jan 18 17:48:32 2015 - [info] Starting ping health check on 192.168.2.128(192.168.2.128:3306).. Sun Jan 18 17:48:32 2015 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond.. Sun Jan 18 17:52:38 2015 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away) Sun Jan 18 17:52:38 2015 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql --output_file=/tmp/save_binary_logs_test --manager_version=0.53 --binlog_prefix=mysql-bin Sun Jan 18 17:52:38 2015 - [info] Executing seconary network check script: /usr/local/bin/masterha_secondary_check -s server03 -s server02 --user=root --master_host=server02 --master_ip=192.168.2.128 --master_port=3306 --user=root --master_host=192.168.2.128 --master_ip=192.168.2.128 --master_port=3306 Sun Jan 18 17:52:38 2015 - [info] HealthCheck: SSH to 192.168.2.128 is reachable. Monitoring server server03 is reachable, Master is not reachable from server03. OK. Monitoring server server02 is reachable, Master is not reachable from server02. OK. Sun Jan 18 17:52:38 2015 - [info] Master is not reachable from all other monitoring servers. Failover should start. Sun Jan 18 17:52:39 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111) Sun Jan 18 17:52:43 2015 - [info] Sun Jan 18 17:52:43 2015 - [info] * Phase 3.3: Determining New Master Phase.. Sun Jan 18 17:52:43 2015 - [info] Sun Jan 18 17:52:43 2015 - [info] Finding the latest slave that has all relay logs for recovering other slaves.. Sun Jan 18 17:52:43 2015 - [info] All slaves received relay logs to the same position. No need to resync each other. Sun Jan 18 17:52:43 2015 - [info] Searching new master from slaves.. Sun Jan 18 17:52:43 2015 - [info] Candidate masters from the configuration file: Sun Jan 18 17:52:43 2015 - [info] 192.168.2.129(192.168.2.129:3306) Version=5.5.30-log (oldest major version between slaves) log-bin:enabled Sun Jan 18 17:52:43 2015 - [info] Replicating from 192.168.2.128(192.168.2.128:3306) Sun Jan 18 17:52:43 2015 - [info] Primary candidate for the new Master (candidate_master is set) Sun Jan 18 17:52:43 2015 - [info] Non-candidate masters: Sun Jan 18 17:52:43 2015 - [info] Searching from candidate_master slaves which have received the latest relay log events.. Sun Jan 18 17:52:43 2015 - [info] New master is 192.168.2.129(192.168.2.129:3306) Sun Jan 18 17:52:43 2015 - [info] Starting master failover.. Sun Jan 18 17:52:43 2015 - [info] From: 192.168.2.128 (current master) +--192.168.2.129 +--192.168.2.130 IN SCRIPT TEST====/sbin/ifconfig eth0:1 down==/sbin/ifconfig eth0:1 192.168.2.88/24=== Enabling the VIP - 192.168.2.88/24 on the new master - 192.168.2.129 Sun Jan 18 17:52:44 2015 - [info] OK. Sun Jan 18 17:52:44 2015 - [info] ** Finished master recovery successfully. Sun Jan 18 17:52:44 2015 - [info] * Phase 3: Master Recovery Phase completed. Sun Jan 18 17:52:44 2015 - [info] Sun Jan 18 17:52:44 2015 - [info] * Phase 4: Slaves Recovery Phase.. Sun Jan 18 17:52:44 2015 - [info] Sun Jan 18 17:52:44 2015 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase.. Sun Jan 18 17:52:44 2015 - [info] ----- Failover Report ----- app1: MySQL Master failover 192.168.2.128 to 192.168.2.129 succeeded Master 192.168.2.128 is down! Started automated(non-interactive) failover. Invalidated master IP address on 192.168.2.128. The latest slave 192.168.2.129(192.168.2.129:3306) has all relay logs for recovery. Selected 192.168.2.129 as a new master. 192.168.2.129: OK: Applying all logs succeeded. 192.168.2.129: OK: Activated master IP address. 192.168.2.130: This host has the latest relay log events. Generating relay diff files from the latest slave succeeded. 192.168.2.130: OK: Applying all logs succeeded. Slave started, replicating from 192.168.2.129. 192.168.2.129: Resetting slave info succeeded. Master failover to 192.168.2.129(192.168.2.129:3306) completed successfully. Sun Jan 18 17:52:45 2015 - [info] Sending mail.. Unknown option: conf
（6）在新的Master192.168.2.129上查看數據有沒有同步過來，由於在還沒建立tbtest庫的時候，就停了slave sql線程:
mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sbtest | | test | +--------------------+ rows in set (0.00 sec) mysql> use sbtest mysql> select count(*) from sbtest; +----------+ | count(*) | +----------+ | 100000 | +----------+ row in set (0.03 sec)
能夠看到落後的數據也同步過來了
（7）查看來vip的漂移狀況（192.168.2.129上面操做）：
[root keepalived-1.2.12]$ ifconfig eth0 Link encap:Ethernet HWaddr 00:0C:29:66:95:64 inet addr:192.168.2.129 Bcast:192.168.2.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fe66:9564/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:48779 errors:0 dropped:0 overruns:0 frame:0 TX packets:31696 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:46635239 (44.4 MiB) TX bytes:3067487 (2.9 MiB) eth0:1 Link encap:Ethernet HWaddr 00:0C:29:66:95:64 inet addr:192.168.2.88 Bcast:192.168.2.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
虛擬IP已經成功漂移到候選的master 192.168.2.129上了。
在作上面經過使用腳本管理vip的實驗時，發現很奇怪的事情，就是我查看切換成功後，我在去monitor(192.168.2.131)查看再那臺是master庫時：
[root bin]$ masterha_check_status --conf=/etc/masterha/app1.cnf app1 is stopped(2:NOT_RUNNING).
發現MHA Manager掛了，這下就呆了，該不會那裏配置錯了吧，後來看到官網上對這種狀況有解釋
意思是安裝一個進程工具，經過該工具結合腳原本管理進程。
能夠參考官方資料：https://code.google.com/p/mysql-master-ha/wiki/Runnning_Background
二.手動Failover（MHA Manager必須沒有運行）
當主服務器故障時，人工手動調用MHA來進行故障切換操做，具體命令以下：
先停MHA Manager(server01 192.168.2.131上面操做):
[root ~]$ masterha_stop --conf=/etc/masterha/app1.cnf Stopped app1 successfully. [1]+ Exit 1 nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 (wd: /usr/local/bin)
在Manager主機上操做（192.168.2.131上面）以下：
[root bin]$ masterha_master_switch --master_state=dead --conf=/etc/masterha/app1.cnf --dead_master_host=192.168.2.128 --dead_master_port=3306 --new_master_host=192.168.2.129 --new_master_port=3306 --ignore_last_failover --dead_master_ip=<dead_master_ip> is not set. Using 192.168.2.128. Mon Jan 19 00:42:18 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Mon Jan 19 00:42:18 2015 - [info] Reading application default configurations from /etc/masterha/app1.cnf.. Mon Jan 19 00:42:18 2015 - [info] Reading server configurations from /etc/masterha/app1.cnf.. Mon Jan 19 00:42:18 2015 - [info] MHA::MasterFailover version 0.56. Mon Jan 19 00:42:18 2015 - [info] Starting master failover. Mon Jan 19 00:42:18 2015 - [info] Mon Jan 19 00:42:18 2015 - [info] * Phase 1: Configuration Check Phase.. Mon Jan 19 00:42:18 2015 - [info] Mon Jan 19 00:42:19 2015 - [info] Dead Servers: Mon Jan 19 00:42:19 2015 - [error][/usr/local/share/perl5/MHA/MasterFailover.pm, ln181] None of server is dead. Stop failover. Mon Jan 19 00:42:19 2015 - [error][/usr/local/share/perl5/MHA/ManagerUtil.pm, ln178] Got ERROR: at /usr/local/bin/masterha_master_switch line 53
看到報錯了，報錯的緣由：MHA manager檢測到沒有dead的server，進行報錯，並結束failover，也就說，咱們須要先手動關了主庫，才能正常切換：
在master（server02 192.168.2.128上面操做）先關掉mysql服務進程
[root ~]$ /etc/init.d/mysqld stop Shutting down MySQL... SUCCESS!
再執行手動failover命令（192.168.2.131上面操做）：
[root bin]$ masterha_master_switch --master_state=dead --conf=/etc/masterha/app1.cnf --dead_master_host=192.168.2.128 --dead_master_port=3306 --new_master_host=192.168.2.129 --new_master_port=3306 --ignore_last_failover --dead_master_ip=<dead_master_ip> is not set. Using 192.168.2.128. Sun Jan 18 19:49:20 2015 - [info] Starting master failover. Sun Jan 18 19:49:20 2015 - [info] Sun Jan 18 19:49:20 2015 - [info] * Phase 1: Configuration Check Phase.. Sun Jan 18 19:49:20 2015 - [info] Sun Jan 18 19:49:20 2015 - [info] Dead Servers: Sun Jan 18 19:49:20 2015 - [info] 192.168.2.128(192.168.2.128:3306) Sun Jan 18 19:49:20 2015 - [info] Checking master reachability via mysql(double check).. Sun Jan 18 19:49:20 2015 - [info] ok. Sun Jan 18 19:49:20 2015 - [info] Alive Servers: Sun Jan 18 19:49:20 2015 - [info] 192.168.2.129(192.168.2.129:3306) Sun Jan 18 19:49:20 2015 - [info] 192.168.2.130(192.168.2.130:3306) Sun Jan 18 19:49:20 2015 - [info] Alive Slaves: Sun Jan 18 19:49:20 2015 - [info] 192.168.2.129(192.168.2.129:3306) Version=5.5.30-log (oldest major version between slaves) log-bin:enabled Sun Jan 18 19:49:20 2015 - [info] Replicating from 192.168.2.128(192.168.2.128:3306) Sun Jan 18 19:49:20 2015 - [info] Primary candidate for the new Master (candidate_master is set) Sun Jan 18 19:49:20 2015 - [info] 192.168.2.130(192.168.2.130:3306) Version=5.5.25-log (oldest major version between slaves) log-bin:enabled Sun Jan 18 19:49:20 2015 - [info] Replicating from 192.168.2.128(192.168.2.128:3306) Master 192.168.2.128 is dead. Proceed? (yes/NO): yes Sun Jan 18 19:49:24 2015 - [info] ** Phase 1: Configuration Check Phase completed. Sun Jan 18 19:49:24 2015 - [info] Sun Jan 18 19:49:24 2015 - [info] * Phase 2: Dead Master Shutdown Phase.. Sun Jan 18 19:49:24 2015 - [info] Sun Jan 18 19:49:24 2015 - [info] HealthCheck: SSH to 192.168.2.128 is reachable. Sun Jan 18 19:49:24 2015 - [info] Forcing shutdown so that applications never connect to the current master.. Sun Jan 18 19:49:24 2015 - [info] Executing master IP deactivatation script: Sun Jan 18 19:49:24 2015 - [info] /usr/local/bin/master_ip_failover --orig_master_host=192.168.2.128 --orig_master_ip=192.168.2.128 --orig_master_port=3306 --command=stopssh --ssh_user=root IN SCRIPT TEST====/sbin/ifconfig eth0:1 down==/sbin/ifconfig eth0:1 192.168.2.88/24=== Concat binary/relay logs from mysql-bin.000016 pos 107 to mysql-bin.000016 EOF into /tmp/saved_master_binlog_from_192.168.2.128_3306_20150118194920.binlog .. Dumping binlog format description event, from position 0 to 107.. ok. Dumping effective binlog data from /data/mysql/mysql-bin.000016 position 107 to tail(126).. ok. Concat succeeded. saved_master_binlog_from_192.168.2.128_3306_20150118194920.binlog 100% 126 0.1KB/s 00:00 Sun Jan 18 19:49:25 2015 - [info] scp from root@192.168.2.128:/tmp/saved_master_binlog_from_192.168.2.128_3306_20150118194920.binlog to local:/var/log/masterha/app1.log/saved_master_binlog_from_192.168.2.128_3306_20150118194920.binlog succeeded. Sun Jan 18 19:49:25 2015 - [info] HealthCheck: SSH to 192.168.2.129 is reachable. Sun Jan 18 19:49:26 2015 - [info] HealthCheck: SSH to 192.168.2.130 is reachable. Sun Jan 18 19:49:26 2015 - [info] Sun Jan 18 19:49:26 2015 - [info] * Phase 3.3: Determining New Master Phase.. Sun Jan 18 19:49:26 2015 - [info] Sun Jan 18 19:49:26 2015 - [info] Finding the latest slave that has all relay logs for recovering other slaves.. Sun Jan 18 19:49:26 2015 - [info] All slaves received relay logs to the same position. No need to resync each other. Sun Jan 18 19:49:26 2015 - [info] 192.168.2.129 can be new master. Sun Jan 18 19:49:26 2015 - [info] New master is 192.168.2.129(192.168.2.129:3306) Sun Jan 18 19:49:26 2015 - [info] Starting master failover.. Sun Jan 18 19:49:26 2015 - [info] From: 192.168.2.128 (current master) +--192.168.2.129 +--192.168.2.130 To: 192.168.2.129 (new master) +--192.168.2.130 Starting master switch from 192.168.2.128(192.168.2.128:3306) to 192.168.2.129(192.168.2.129:3306)? (yes/NO): yes Sun Jan 18 19:49:31 2015 - [info] New master decided manually is 192.168.2.129(192.168.2.129:3306) Sun Jan 18 19:49:31 2015 - [info] Sun Jan 18 19:49:31 2015 - [info] * Phase 3.3: New Master Diff Log Generation Phase.. Sun Jan 18 19:49:31 2015 - [info] Sun Jan 18 19:49:31 2015 - [info] This server has all relay logs. No need to generate diff files from the latest slave. Sun Jan 18 19:49:31 2015 - [info] Sending binlog.. saved_master_binlog_from_192.168.2.128_3306_20150118194920.binlog 100% 126 0.1KB/s 00:00 Sun Jan 18 19:49:31 2015 - [info] scp from local:/var/log/masterha/app1.log/saved_master_binlog_from_192.168.2.128_3306_20150118194920.binlog to root@192.168.2.129:/tmp/saved_master_binlog_from_192.168.2.128_3306_20150118194920.binlog succeeded. Sun Jan 18 19:49:32 2015 - [info] Executing master IP activate script: Sun Jan 18 19:49:32 2015 - [info] /usr/local/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.2.128 --orig_master_ip=192.168.2.128 --orig_master_port=3306 --new_master_host=192.168.2.129 --new_master_ip=192.168.2.129 --new_master_port=3306 IN SCRIPT TEST====/sbin/ifconfig eth0:1 down==/sbin/ifconfig eth0:1 192.168.2.88/24=== Enabling the VIP - 192.168.2.88/24 on the new master - 192.168.2.129 Sun Jan 18 19:49:32 2015 - [info] OK. Applying differential binary/relay log files /tmp/saved_master_binlog_from_192.168.2.128_3306_20150118194920.binlog on 192.168.2.130:3306. This may take long time... Applying log files succeeded. Sun Jan 18 19:49:33 2015 - [info] All relay logs were successfully applied. Sun Jan 18 19:49:33 2015 - [info] Resetting slave 192.168.2.130(192.168.2.130:3306) and starting replication from the new master 192.168.2.129(192.168.2.129:3306).. Sun Jan 18 19:49:33 2015 - [info] Executed CHANGE MASTER. Sun Jan 18 19:49:33 2015 - [info] Slave started. Sun Jan 18 19:49:33 2015 - [info] End of log messages from 192.168.2.130. Sun Jan 18 19:49:33 2015 - [info] -- Slave recovery on host 192.168.2.130(192.168.2.130:3306) succeeded. Sun Jan 18 19:49:33 2015 - [info] All new slave servers recovered successfully. Sun Jan 18 19:49:33 2015 - [info] Sun Jan 18 19:49:33 2015 - [info] * Phase 5: New master cleanup phease.. Sun Jan 18 19:49:33 2015 - [info] Sun Jan 18 19:49:33 2015 - [info] Resetting slave info on the new master.. Sun Jan 18 19:49:33 2015 - [info] 192.168.2.129: Resetting slave info succeeded. Sun Jan 18 19:49:33 2015 - [info] Master failover to 192.168.2.129(192.168.2.129:3306) completed successfully. Sun Jan 18 19:49:33 2015 - [info] ----- Failover Report ----- app1: MySQL Master failover 192.168.2.128 to 192.168.2.129 succeeded Master 192.168.2.128 is down!
3、MHA的在線切換
在許多狀況下， 須要將現有的主服務器遷移到另一臺服務器上。 好比主服務器硬件故障，RAID 控制卡須要重建，將主服務器移到性能更好的服務器上等。當維護主服務器會引發性能降低， 致使停機時間至少沒法寫入數據。 另外， 阻塞或殺掉當前運行的會話會致使主主之間數據不一致的問題發生。 MHA 提供快速切換和優雅的阻塞寫入，這個切換過程只須要 0.5-2s 的時間，這段時間內數據是沒法寫入的。在不少狀況下，0.5-2s 的阻塞寫入是能夠接受的。所以切換主服務器不須要計劃分配維護時間窗口。
MHA在線切換的大概過程：
（1）檢測複製設置和肯定當前主服務器
（2）肯定新的主服務器
（3）阻塞寫入到當前主服務器
（4）等待全部從服務器遇上覆制
（5）授予寫入到新的主服務器
（6）從新設置從服務器
注意，在線切換的時候應用架構須要考慮如下兩個問題：
1.自動識別master和slave的問題（master的機器可能會切換），若是採用了vip的方式，基本能夠解決這個問題。
2.負載均衡的問題（能夠定義大概的讀寫比例，每臺機器可承擔的負載比例，當有機器離開集羣時，須要考慮這個問題）
爲了保證數據徹底一致性，在最快的時間內完成切換，MHA的在線切換必須知足如下條件纔會切換成功，不然會切換失敗。
(1)全部slave的IO線程都在運行
(2)全部slave的SQL線程都在運行
(3)全部的show slave status的輸出中Seconds_Behind_Master參數小於或者等於running_updates_limit秒，若是在切換過程當中不指定running_updates_limit,那麼默認狀況下running_updates_limit爲1秒。
(4)在master端，經過show processlist輸出，沒有一個更新花費的時間大於running_updates_limit秒。
在線切換步驟以下：
在MHA Manager服務器192.168.2.131上操做，首先，停掉MHA監控：
[root ~]$ masterha_stop --conf=/etc/masterha/app1.cnf Stopped app1 successfully. [1]+ Exit 1 nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 (wd: /usr/local/bin)
執行在線切換命令：（如下是0.53版本的manager和node包報的錯）
[root bin]$ masterha_master_switch --conf=/etc/masterha/app1.cnf --master_state=alive --new_master_host=192.168.2.129 --new_master_port=3306 --orig_master_is_new_slave --running_updates_limit=10000
Starting master switch from 192.168.2.128(192.168.2.128:3306) to 192.168.2.129(192.168.2.129:3306)? (yes/NO): yes
Sun Jan 18 20:06:17 2015 - [info] Checking whether 192.168.2.129(192.168.2.129:3306) is ok for the new master..
Sun Jan 18 20:06:17 2015 - [info] ok.
Sun Jan 18 20:06:17 2015 - [info] 192.168.2.128(192.168.2.128:3306): SHOW SLAVE STATUS returned empty result. To check replication filtering rules, temporarily executing CHANGE MASTER to a dummy host.
Sun Jan 18 20:06:17 2015 - [info] 192.168.2.128(192.168.2.128:3306): Resetting slave pointing to the dummy host.
Sun Jan 18 20:06:17 2015 - [info] ** Phase 1: Configuration Check Phase completed.
Sun Jan 18 20:06:17 2015 - [info]
Sun Jan 18 20:06:17 2015 - [info] * Phase 2: Rejecting updates Phase..
Sun Jan 18 20:06:17 2015 - [info]
Sun Jan 18 20:06:17 2015 - [info] Executing master ip online change script to disable write on the current master:
Sun Jan 18 20:06:17 2015 - [info] /usr/local/bin/master_ip_online_change --command=stop --orig_master_host=192.168.2.128 --orig_master_ip=192.168.2.128 --orig_master_port=3306 --new_master_host=192.168.2.129 --new_master_ip=192.168.2.129 --new_master_port=3306
Got Error: DBI connect(';host=192.168.2.129;port=3306;mysql_connect_timeout=4','',...) failed: Access denied for user 'root'@'192.168.2.131' (using password: NO) at /usr/local/share/perl5/MHA/DBHelper.pm line 181
at /usr/local/bin/master_ip_online_change line 138
Sun Jan 18 20:06:17 2015 - [error][/usr/local/share/perl5/MHA/ManagerUtil.pm, ln178] Got ERROR: at /usr/local/bin/masterha_master_switch line 53
下面來看來0.56版本的執行狀況：
[root bin]$ masterha_master_switch --conf=/etc/masterha/app1.cnf --master_state=alive --new_master_host=192.168.2.129 --new_master_port=3306 --orig_master_is_new_slave --running_updates_limit=10000 192.168.2.131 [root bin]$ masterha_master_switch --conf=/etc/masterha/app1.cnf --master_state=alive --new_master_host=192.168.2.129 --new_master_port=3306 --orig_master_is_new_slave --running_updates_limit=10000 Mon Jan 19 01:51:39 2015 - [info] MHA::MasterRotate version 0.56. Mon Jan 19 01:51:39 2015 - [info] Starting online master switch.. It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 192.168.2.128(192.168.2.128:3306)? (YES/no): yes Mon Jan 19 01:51:46 2015 - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time.. Mon Jan 19 01:51:46 2015 - [info] ok. Mon Jan 19 01:51:46 2015 - [info] Checking MHA is not monitoring or doing failover.. Mon Jan 19 01:51:46 2015 - [info] Checking replication health on 192.168.2.129.. Mon Jan 19 01:51:46 2015 - [info] ok. Mon Jan 19 01:51:46 2015 - [info] Checking replication health on 192.168.2.130.. Mon Jan 19 01:51:46 2015 - [info] ok. Mon Jan 19 01:51:46 2015 - [info] 192.168.2.129 can be new master. Mon Jan 19 01:51:46 2015 - [info] From: 192.168.2.128(192.168.2.128:3306) (current master) +--192.168.2.129(192.168.2.129:3306) +--192.168.2.130(192.168.2.130:3306) To: 192.168.2.129(192.168.2.129:3306) (new master) +--192.168.2.130(192.168.2.130:3306) +--192.168.2.128(192.168.2.128:3306) Starting master switch from 192.168.2.128(192.168.2.128:3306) to 192.168.2.129(192.168.2.129:3306)? (yes/NO): yes Mon Jan 19 01:51:50 2015 - [info] Checking whether 192.168.2.129(192.168.2.129:3306) is ok for the new master.. Mon Jan 19 01:51:50 2015 - [info] ok. Mon Jan 19 01:51:50 2015 - [info] Executing master ip online change script to disable write on the current master: Mon Jan 19 01:51:50 2015 - [info] /usr/local/bin/master_ip_online_change --command=stop --orig_master_host=192.168.2.128 --orig_master_ip=192.168.2.128 --orig_master_port=3306 --orig_master_user='root' --orig_master_password='123456' --new_master_host=192.168.2.129 --new_master_ip=192.168.2.129 --new_master_port=3306 --new_master_user='root' --new_master_password='123456' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_is_new_slave Mon Jan 19 01:51:50 2015 173112 Set read_only on the new master.. ok. Mon Jan 19 01:51:50 2015 178943 Drpping app user on the orig master.. Mon Jan 19 01:51:50 2015 180438 Set read_only=1 on the orig master.. ok. Mon Jan 19 01:51:50 2015 183258 Killing all application threads.. Mon Jan 19 01:51:50 2015 183387 done. Mon Jan 19 01:51:50 2015 - [info] ok. Mon Jan 19 01:51:50 2015 - [info] Locking all tables on the orig master to reject updates from everybody (including root): Mon Jan 19 01:51:50 2015 - [info] Executing FLUSH TABLES WITH READ LOCK.. Mon Jan 19 01:51:50 2015 - [info] ok. Mon Jan 19 01:51:50 2015 - [info] Orig master binlog:pos is mysql-bin.000017:107. Mon Jan 19 01:51:50 2015 - [info] Waiting to execute all relay logs on 192.168.2.129(192.168.2.129:3306).. Mon Jan 19 01:51:50 2015 - [info] master_pos_wait(mysql-bin.000017:107) completed on 192.168.2.129(192.168.2.129:3306). Executed 0 events. Mon Jan 19 01:51:50 2015 - [info] done. Mon Jan 19 01:51:50 2015 - [info] Getting new master's binlog name and position.. Mon Jan 19 01:51:50 2015 - [info] mysql-bin.000005:61791 Mon Jan 19 01:51:50 2015 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.2.129', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000005', MASTER_LOG_POS=61791, MASTER_USER='repl', MASTER_PASSWORD='xxx'; Mon Jan 19 01:51:50 2015 - [info] Executing master ip online change script to allow write on the new master: Mon Jan 19 01:51:50 2015 - [info] /usr/local/bin/master_ip_online_change --command=start --orig_master_host=192.168.2.128 --orig_master_ip=192.168.2.128 --orig_master_port=3306 --orig_master_user='root' --orig_master_password='123456' --new_master_host=192.168.2.129 --new_master_ip=192.168.2.129 --new_master_port=3306 --new_master_user='root' --new_master_password='123456' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_is_new_slave Mon Jan 19 01:51:50 2015 443208 Set read_only=0 on the new master. Mon Jan 19 01:51:50 2015 444741 Creating app user on the new master.. Mon Jan 19 01:51:50 2015 - [info] ok. Mon Jan 19 01:51:50 2015 - [info] Mon Jan 19 01:51:50 2015 - [info] * Switching slaves in parallel.. Mon Jan 19 01:51:50 2015 - [info] Mon Jan 19 01:51:50 2015 - [info] -- Slave switch on host 192.168.2.130(192.168.2.130:3306) started, pid: 23040 Mon Jan 19 01:51:50 2015 - [info] Mon Jan 19 01:51:50 2015 - [info] Log messages from 192.168.2.130 ... Mon Jan 19 01:51:50 2015 - [info] Mon Jan 19 01:51:50 2015 - [info] Waiting to execute all relay logs on 192.168.2.130(192.168.2.130:3306).. Mon Jan 19 01:51:50 2015 - [info] master_pos_wait(mysql-bin.000017:107) completed on 192.168.2.130(192.168.2.130:3306). Executed 0 events. Mon Jan 19 01:51:50 2015 - [info] done. Mon Jan 19 01:51:50 2015 - [info] Resetting slave 192.168.2.130(192.168.2.130:3306) and starting replication from the new master 192.168.2.129(192.168.2.129:3306).. Mon Jan 19 01:51:50 2015 - [info] Executed CHANGE MASTER. Mon Jan 19 01:51:50 2015 - [info] Slave started. Mon Jan 19 01:51:50 2015 - [info] Starting orig master as a new slave.. Mon Jan 19 01:51:50 2015 - [info] Resetting slave 192.168.2.128(192.168.2.128:3306) and starting replication from the new master 192.168.2.129(192.168.2.129:3306).. Mon Jan 19 01:51:50 2015 - [info] 192.168.2.129: Resetting slave info succeeded. Mon Jan 19 01:51:50 2015 - [info] Switching master to 192.168.2.129(192.168.2.129:3306) completed successfully.
參數說明：
--orig_master_is_new_slave 切換時加上此參數是將原 master變爲slave 節點，若是不加此參數，原來的master 將不啓動
--running_updates_limit=10000,故障切換時,候選master若是有延遲的話， mha 切換不能成功，加上此參數表示延遲在此時間範圍內均可切換（單位爲s），可是切換的時間長短是由recover時relay日誌的大小決定
master_ip_online_change腳本代碼以下：
#!/usr/bin/env perl
## Note: This is a sample script and is not complete. Modify the script based on your environment.
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
use MHA::DBHelper;
use MHA::NodeUtil;
use Time::HiRes qw( sleep gettimeofday tv_interval );
use Data::Dumper;
my $_tstart;
my $_running_interval = 0.1;
my (
$command, $orig_master_is_new_slave, $orig_master_host,
$orig_master_ip, $orig_master_port, $orig_master_user,
$orig_master_password, $orig_master_ssh_user, $new_master_host,
$new_master_ip, $new_master_port, $new_master_user,
$new_master_password, $new_master_ssh_user
);
my $vip = '192.168.2.88/24';
my $key = '1';
my $ssh_start_vip = "/sbin/ifconfig eth0:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig eth0:$key down";
my $orig_master_ssh_port = 22;
my $new_master_ssh_port = 22;
GetOptions(
'command=s' => \$command,
'orig_master_is_new_slave' => \$orig_master_is_new_slave,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'orig_master_user=s' => \$orig_master_user,
'orig_master_password=s' => \$orig_master_password,
'orig_master_ssh_user=s' => \$orig_master_ssh_user,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
'new_master_user=s' => \$new_master_user,
'new_master_password=s' => \$new_master_password,
'new_master_ssh_user=s' => \$new_master_ssh_user,
'orig_master_ssh_port=i' => \$orig_master_ssh_port,
'new_master_ssh_port=i' => \$new_master_ssh_port,
);
exit &main();
sub current_time_us {
my ( $sec, $microsec ) = gettimeofday();
my $curdate = localtime($sec);
return $curdate . " " . sprintf( "%06d", $microsec );
}
sub sleep_until {
my $elapsed = tv_interval($_tstart);
if ( $_running_interval > $elapsed ) {
sleep( $_running_interval - $elapsed );
}
}
sub get_threads_util {
my $dbh = shift;
my $my_connection_id = shift;
my $running_time_threshold = shift;
my $type = shift;
$running_time_threshold = 0 unless ($running_time_threshold);
$type = 0 unless ($type);
my @threads;
my $sth = $dbh->prepare("SHOW PROCESSLIST");
$sth->execute();
while ( my $ref = $sth->fetchrow_hashref() ) {
my $id = $ref->{Id};
my $user = $ref->{User};
my $host = $ref->{Host};
my $command = $ref->{Command};
my $state = $ref->{State};
my $query_time = $ref->{Time};
my $info = $ref->{Info};
$info =~ s/^\s*(.*?)\s*$/$1/ if defined($info);
next if ( $my_connection_id == $id );
next if ( defined($query_time) && $query_time < $running_time_threshold );
next if ( defined($command) && $command eq "Binlog Dump" );
next if ( defined($user) && $user eq "system user" );
next
if ( defined($command)
&& $command eq "Sleep"
&& defined($query_time)
&& $query_time >= 1 );
if ( $type >= 1 ) {
next if ( defined($command) && $command eq "Sleep" );
next if ( defined($command) && $command eq "Connect" );
}
if ( $type >= 2 ) {
next if ( defined($info) && $info =~ m/^select/i );
next if ( defined($info) && $info =~ m/^show/i );
}
push @threads, $ref;
}
return @threads;
}
sub main {
if ( $command eq "stop" ) {
## Gracefully killing connections on the current master
# 1. Set read_only= 1 on the new master
# 2. DROP USER so that no app user can establish new connections
# 3. Set read_only= 1 on the current master
# 4. Kill current queries
# * Any database access failure will result in script die.
my $exit_code = 1;
eval {
## Setting read_only=1 on the new master (to avoid accident)
my $new_master_handler = new MHA::DBHelper();
# args: hostname, port, user, password, raise_error(die_on_error)_or_not
$new_master_handler->connect( $new_master_ip, $new_master_port,
$new_master_user, $new_master_password, 1 );
print current_time_us() . " Set read_only on the new master.. ";
$new_master_handler->enable_read_only();
if ( $new_master_handler->is_read_only() ) {
print "ok.\n";
}
else {
die "Failed!\n";
}
$new_master_handler->disconnect();
# Connecting to the orig master, die if any database error happens
my $orig_master_handler = new MHA::DBHelper();
$orig_master_handler->connect( $orig_master_ip, $orig_master_port,
$orig_master_user, $orig_master_password, 1 );
## Drop application user so that nobody can connect. Disabling per-session binlog beforehand
$orig_master_handler->disable_log_bin_local();
print current_time_us() . " Drpping app user on the orig master..\n";
#FIXME_xxx_drop_app_user($orig_master_handler);
## Waiting for N * 100 milliseconds so that current connections can exit
my $time_until_read_only = 15;
$_tstart = [gettimeofday];
my @threads = get_threads_util( $orig_master_handler->{dbh},
$orig_master_handler->{connection_id} );
while ( $time_until_read_only > 0 && $#threads >= 0 ) {
if ( $time_until_read_only % 5 == 0 ) {
printf
"%s Waiting all running %d threads are disconnected.. (max %d milliseconds)\n",
current_time_us(), $#threads + 1, $time_until_read_only * 100;
if ( $#threads < 5 ) {
print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . "\n"
foreach (@threads);
}
}
sleep_until();
$_tstart = [gettimeofday];
$time_until_read_only--;
@threads = get_threads_util( $orig_master_handler->{dbh},
$orig_master_handler->{connection_id} );
}
## Setting read_only=1 on the current master so that nobody(except SUPER) can write
print current_time_us() . " Set read_only=1 on the orig master.. ";
$orig_master_handler->enable_read_only();
if ( $orig_master_handler->is_read_only() ) {
print "ok.\n";
}
else {
die "Failed!\n";
}
## Waiting for M * 100 milliseconds so that current update queries can complete
my $time_until_kill_threads = 5;
@threads = get_threads_util( $orig_master_handler->{dbh},
$orig_master_handler->{connection_id} );
while ( $time_until_kill_threads > 0 && $#threads >= 0 ) {
if ( $time_until_kill_threads % 5 == 0 ) {
printf
"%s Waiting all running %d queries are disconnected.. (max %d milliseconds)\n",
current_time_us(), $#threads + 1, $time_until_kill_threads * 100;
if ( $#threads < 5 ) {
print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . "\n"
foreach (@threads);
}
}
sleep_until();
$_tstart = [gettimeofday];
$time_until_kill_threads--;
@threads = get_threads_util( $orig_master_handler->{dbh},
$orig_master_handler->{connection_id} );
}
## Terminating all threads
print current_time_us() . " Killing all application threads..\n";
$orig_master_handler->kill_threads(@threads) if ( $#threads >= 0 );
print current_time_us() . " done.\n";
$orig_master_handler->enable_log_bin_local();
$orig_master_handler->disconnect();
## After finishing the script, MHA executes FLUSH TABLES WITH READ LOCK
eval {
`ssh -p$orig_master_ssh_port $orig_master_ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
};
if ($@) {
warn $@;
}
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
## Activating master ip on the new master
# 1. Create app user with write privileges
# 2. Moving backup script if needed
# 3. Register new master's ip to the catalog database
# We don't return error even though activating updatable accounts/ip failed so that we don't interrupt slaves' recovery.
# If exit code is 0 or 10, MHA does not abort
my $exit_code = 10;
eval {
my $new_master_handler = new MHA::DBHelper();
# args: hostname, port, user, password, raise_error_or_not
$new_master_handler->connect( $new_master_ip, $new_master_port,
$new_master_user, $new_master_password, 1 );
## Set read_only=0 on the new master
$new_master_handler->disable_log_bin_local();
print current_time_us() . " Set read_only=0 on the new master.\n";
$new_master_handler->disable_read_only();
## Creating an app user on the new master
print current_time_us() . " Creating app user on the new master..\n";
#FIXME_xxx_create_app_user($new_master_handler);
$new_master_handler->enable_log_bin_local();
$new_master_handler->disconnect();
## Update master ip on the catalog database, etc
`ssh -p$new_master_ssh_port $new_master_ssh_user\@$new_master_host \" $ssh_start_vip \"`;
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
# do nothing
exit 0;
}
else {
&usage();
exit 1;
}
}
sub usage {
print
"Usage: master_ip_online_change --command=start|stop|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
die;
}
說明能夠參考官網：
https://code.google.com/p/mysql-master-ha/wiki/Parameters#master_ip_online_change_script
二、修復宕機的Master
一般狀況下自動切換之後，原master可能已經廢棄掉，待原master主機修復後，若是數據完整的狀況下，可能想把原來master從新做爲新主庫的slave，這時咱們能夠藉助當時自動切換時刻的MHA日誌來完成對原master的修復。下面是提取相關日誌的命令：
從上面信息能夠看到：
[root]# grep -i "All other slaves should start" manager.log All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.2.129', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000005', MASTER_LOG_POS=61791, MASTER_USER='repl', MASTER_PASSWORD='xxx';
意思是說若是Master主機修復好了，能夠在修復好後的Master執行CHANGE MASTER操做,做爲新的slave庫。
Mysql高可用方案能夠必定程度上實現數據庫的高可用，好比MMM，heartbeat+drbd，Cluster等。還有percona的Galera Cluster等。這些高可用軟件各有優劣。在進行高可用方案選擇時，主要是看業務還有對數據一致性方面的要求。最後出於對數據庫的高可用和數據一致性的要求，推薦使用MHA架構。
參考資料：
https://code.google.com/p/mysql-master-ha/wiki/TableOfContents?tm=6
http://www.cnblogs.com/gomysql/p/3675429.html
http://www.cnblogs.com/xuanzhi201111/p/4231412.html
http://51power.blog.51cto.com/3549599/1672730
http://blog.csdn.net/wshxgxiaoli/article/category/1756667 mysql專題
相關文章
- 1. 搭建MySQL的高可用架構（MHA）
- 2. Mysql 的高可用之 MHA
- 3. MySQL高可用架構之MHA MySQL高可用架構之MHA
- 4. 搭建MySQL高可用架構MHA
- 5. MySQL 高可用之MHA（二）
- 6. MySQL高可用之MHA
- 7. MySQL高可用之MHA（二）
- 8. MySQL 高可用之MHA（一）
- 9. MySQL高可用架構之基於MHA的搭建
- 10. MYSQL高可用架構之MHA的搭建
- 更多相關文章...
- • PHP 可用的函數 - PHP參考手冊
- • Swift 環境搭建 - Swift 教程
- • 漫談MySQL的鎖機制
- • 適用於PHP初學者的學習線路和建議
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
歡迎關注本站公眾號,獲取更多信息
相關文章