mysql高可用架構之-MHA學習
此博文參考 博主：mysql高級DBA yayun 完成php
簡介：html
MHA（Master High Availability）目前在MySQL高可用方面是一個相對成熟的解決方案，它由日本DeNA公司youshimaton（現就任於Facebook公司）開發，是一套優秀的做爲MySQL高可用性環境下故障切換和主從提高的高可用軟件。在MySQL故障切換過程當中，MHA能作到在0~30秒以內自動完成數據庫的故障切換操做，而且在進行故障切換的過程當中，MHA能在最大程度上保證數據的一致性，以達到真正意義上的高可用。node
該軟件由兩部分組成：MHA Manager（管理節點）和MHA Node（數據節點）。MHA Manager能夠單獨部署在一臺獨立的機器上管理多個master-slave集羣，也能夠部署在一臺slave節點上。MHA Node運行在每臺MySQL服務器上，MHA Manager會定時探測集羣中的master節點，當master出現故障時，它能夠自動將最新數據的slave提高爲新的master，而後將全部其餘的slave從新指向新的master。整個故障轉移過程對應用程序徹底透明。mysql
在MHA自動故障切換過程當中，MHA試圖從宕機的主服務器上保存二進制日誌，最大程度的保證數據的不丟失，但這並不老是可行的。例如，若是主服務器硬件故障或沒法經過ssh訪問，MHA無法保存二進制日誌，只進行故障轉移而丟失了最新的數據。使用MySQL 5.5的半同步複製，能夠大大下降數據丟失的風險。MHA能夠與半同步複製結合起來。若是隻有一個slave已經收到了最新的二進制日誌，MHA能夠將最新的二進制日誌應用於其餘全部的slave服務器上，所以能夠保證全部節點的數據一致性。linux
目前MHA主要支持一主多從的架構，要搭建MHA,要求一個複製集羣中必須最少有三臺數據庫服務器，一主二從，即一臺充當master，一臺充當備用master，另一臺充當從庫，由於至少須要三臺服務器，出於機器成本的考慮，淘寶也在該基礎上進行了改造，目前淘寶TMHA已經支持一主一從。另外對於想快速搭建的能夠參考：MHA快速搭建c++
咱們本身使用其實也可使用1主1從，可是master主機宕機後沒法切換，以及沒法補全binlog。master的mysqld進程crash後，仍是能夠切換成功，以及補全binlog的。redis
官方介紹：https://code.google.com/p/mysql-master-ha/sql
圖01展現瞭如何經過MHA Manager管理多組主從複製。能夠將MHA工做原理總結爲以下：shell
每一組都是一組主從複製組數據庫
（1）從宕機崩潰的master保存二進制日誌事件（binlog events）;
（2）識別含有最新更新的slave；
（3）應用差別的中繼日誌（relay log）到其餘的slave；
（4）應用從master保存的二進制日誌事件（binlog events）；
（5）提高一個slave爲新的master；
（6）使其餘的slave鏈接新的master進行復制；
1.編譯安裝mysql 參考地址：
①cmake的安裝
yum install -y cmake
yum -y install make gcc-c++ cmake bison-devel ncurses-devel gcc autoconf automake zlib* fiex* libxml* libmcrypt* libtool-ltdl-devel*
②mysql源碼安裝
採用源碼安裝方式安裝，先將mysql-5.6.16.tar源碼拷貝至服務器解壓，解壓後進入mysql-5.6.16這個目錄中 使用cmake源碼安裝mysql（若是你打算安裝到不一樣的路徑，注意修改下面語句中/usr/local/mysql這個路徑！） cmake -DCMAKE_INSTALL_PREFIX=/usr/local/mysql \ -DMYSQL_UNIX_ADDR=/usr/local/mysql/mysql.sock \ -DDEFAULT_CHARSET=utf8 \ -DDEFAULT_COLLATION=utf8_general_ci \ -DWITH_MYISAM_STORAGE_ENGINE=1 \ -DWITH_INNOBASE_STORAGE_ENGINE=1 \ -DWITH_ARCHIVE_STORAGE_ENGINE=1 \ -DWITH_BLACKHOLE_STORAGE_ENGINE=1 \ -DWITH_MEMORY_STORAGE_ENGINE=1 \ -DWITH_READLINE=1 \ -DENABLED_LOCAL_INFILE=1 \ -DMYSQL_DATADIR=/mysql/data \ -DMYSQL_USER=mysql \ -DMYSQL_TCP_PORT=3306 \ -DENABLE_DOWNLOADS=1 上面的這些複製完，回車，而後就開始cmake的過程，通常時間不會很長。 cmake結束後開始編譯源碼，這一步時間會較長，請耐心等待。 make （注：此步驟可使用make -j多個CPU一塊兒編譯，提高速度，單核CPU不要加此參數） 安裝編譯好的程序 make install 注意：若是須要重裝mysql，在/usr/local/src/mysql-5.6.16在執行下make install就能夠了，不須要再cmake和make 清除安裝臨時文件 make clean
③初始化數據目錄
cd /usr/local/mysql/scripts/ 建立mysql用戶，將/mysql /user/local/mysql 目錄賦權給mysql useradd mysql chown -Rf mysql:mysql /mysql chown -Rf mysql:mysql /usr/local/mysql ./mysql_install_db --datadir=/mysql/data --basedir=/usr/local/mysql --user=mysql 出現兩個OK字樣，表示數據目錄初始化完畢在/mysql/data下
④mysql註冊服務並啓動
cd /usr/local/mysql/support-files cp mysql.server /etc/rc.d/init.d/mysql cp my-default.cnf /etc/my.cnf chkconfig --add mysql chkconfig mysql on service mysql start mysql -u mysql -S /usr/local/mysql/mysql.sock 鏈接須要sock信息，能夠-S指定路徑或者把socket路徑寫在my.cnf中，配置相應路徑，不然鏈接報錯，建議寫在配置文件裏 [mysqld] socket=/usr/local/mysql/mysql.sock [client] socket=/usr/local/mysql/mysql.sock 其餘my.cnf參數後續添加
# mysql主從複製
本實驗經過三臺服務器搭建主從複製環境
Master 10.0.0.50
cadicateMaster 10.0.0.60
slave 10.0.0.70
修改主庫和從庫的參數文件(/etc/my.cnf)
主庫：10.0.0.50
#[mysqld]標籤下追加 server_id =1 log-bin=/mysql/data/mysql-bin log-bin-index=/mysql/data/mysql-binlog.index log_slave_updates=1 sync-binlog = 1
從庫1：10.0.0.60 (備主)
#[mysqld]標籤下追加 server_id =2 log-bin=/mysql/data/mysql-bin log-bin-index=/mysql/data/mysql-binlog.index log_slave_updates=1 sync-binlog = 1
從庫2：10.0.0.70
#[mysqld]標籤下追加 read_only=1 server_id = 3
MHA軟件由兩部分組成，Manager工具包和Node工具包，具體的說明以下。
Manager工具包主要包括如下幾個工具：
masterha_check_ssh 檢查MHA的SSH配置情況
masterha_check_repl 檢查MySQL複製情況
masterha_manger 啓動MHA
masterha_check_status 檢測當前MHA運行狀態
masterha_master_monitor 檢測master是否宕機
masterha_master_switch 控制故障轉移（自動或者手動）
masterha_conf_host 添加或刪除配置的server信息
Node工具包（這些工具一般由MHA Manager的腳本觸發，無需人爲操做）主要包括如下幾個工具：
save_binary_logs 保存和複製master的二進制日誌
apply_diff_relay_logs 識別差別的中繼日誌事件並將其差別的事件應用於其餘的slave
filter_mysqlbinlog 去除沒必要要的ROLLBACK事件（MHA已再也不使用這個工具）
purge_relay_logs 清除中繼日誌（不會阻塞SQL線程）
注意：
爲了儘量的減小主庫硬件損壞宕機形成的數據丟失，所以在配置MHA的同時建議配置成MySQL 5.5的半同步複製。關於半同步複製原理各位本身進行查閱。（不是必須）
角色 ip地址 主機名 server_id 類型 Monitor host 10.0.0.20 manager - 監控複製組 Master 10.0.0.50 master 1 寫入 Candicate master 10.0.0.60 cadicate-master 2 讀 Slave 10.0.0.70 slave 3 讀
其中master對外提供寫服務，備選master（實際的slave，主機名Candicate master）提供讀服務，slave也提供相關的讀服務，一旦master宕機，將會把備選master提高爲新的master，slave指向新的master
（1）在全部節點安裝MHA node所需的perl模塊（DBD:mysql），安裝腳本以下：
[root@master ~]# cat install.sh #!/bin/bash wget http://xrl.us/cpanm --no-check-certificate mv cpanm /usr/bin chmod 755 /usr/bin/cpanm cat > /root/list << EOF install DBD::mysql EOF for package in `cat /root/list` do cpanm $package done
[root@master ~]bash install.sh
[root@master ~]scp install.sh root@10.0.0.50(60,70):~/ #傳到其餘節點 並執行安裝腳本
安裝報以下錯誤：
Can't locate ExtUtils/Manifest.pm in @INC (@INC contains: FatPacked::33970024=HASH(0x2065768) /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/bin/cpanm line 132.
解決方法： 安裝下面的包 yum install perl-ExtUtils-MakeMaker -y
若是有安裝epel源，也可使用yum安裝
yum install perl-DBD-MySQL -y
（2）在全部的節點安裝mha node：
wget http://mysql-master-ha.googlecode.com/files/mha4mysql-node-0.56.tar.gz tar xf mha4mysql-node-0.56.tar.gz cd mha4mysql-node-0.56 perl Makefile.PL make && make install
安裝完成後會在/usr/local/bin目錄下生成如下腳本文件：
[root@master bin]# pwd /usr/local/bin [root@master bin]# ll total 40 -r-xr-xr-x 1 root root 15498 Apr 20 10:05 apply_diff_relay_logs -r-xr-xr-x 1 root root 4807 Apr 20 10:05 filter_mysqlbinlog -r-xr-xr-x 1 root root 7401 Apr 20 10:05 purge_relay_logs -r-xr-xr-x 1 root root 7263 Apr 20 10:05 save_binary_logs [root@master bin]#
關於上面腳本的功能，上面已經介紹過了，這裏再也不重複了。
2.安裝MHA Manager
MHA Manager中主要包括了幾個管理員的命令行工具，例如master_manger，master_master_switch等。MHA Manger也依賴於perl模塊，具體以下：
（1）安裝MHA Node軟件包以前須要安裝依賴。我這裏使用yum完成，沒有epel源的可使用上面提到的腳本（epel源安裝也簡單）。注意：在MHA Manager的主機也是須要安裝MHA Node。
rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm yum install perl-DBD-MySQL -y
安裝MHA Node軟件包，和上面的方法同樣，以下：
wget http://mysql-master-ha.googlecode.com/files/mha4mysql-node-0.56.tar.gz tar xf mha4mysql-node-0.56.tar.gz cd mha4mysql-node-0.56 perl Makefile.PL
小坑：報錯，刪除編譯緩存文件。安裝缺乏的Curses-devel 包
CMake Error at cmake/readline.cmake:85 (MESSAGE):
Curses library not found. Please install appropriate package,
remove CMakeCache.txt and rerun cmake.On Debian/Ubuntu, package name is libncurses5-dev, on Redhat and derivates it is ncurses-devel.
Call Stack (most recent call first):
cmake/readline.cmake:128 (FIND_CURSES)
cmake/readline.cmake:202 (MYSQL_USE_BUNDLED_EDITLINE)
CMakeLists.txt:410 (MYSQL_CHECK_EDITLINE)
-- Configuring incomplete, errors occurred!
See also "/root/mysql-5.6.16/CMakeFiles/CMakeOutput.log".
See also "/root/mysql-5.6.16/CMakeFiles/CMakeError.log".
[root@cadicate-master mysql-5.6.16] yum install curses-devel -y
[root@cadicate-master mysql-5.6.16]# rm CMakeCache.txt
#報以下錯誤（一個老外的回答）
Can't locate CPAN.pm in @INC (@INC contains: inc /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .)
at inc/Module/AutoInstall.pm line 277.
Try to install CPAN using
yum -y install perl-CPAN
and then use command to install the required modulescpan
cpan XML::Writer
#有特麼報錯（這perl真是xxxxx）
Warning (usually harmless): 'YAML' not installed, will not store persistent state
perl -MCPAN -e 'install "YAML"'
make && make install
（2）安裝MHA Manager。首先安裝MHA Manger依賴的perl模塊（我這裏使用yum安裝）：
yum install perl-DBD-MySQL perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager perl-Time-HiRes -y
安裝MHA Manager軟件包：
wget http://mysql-master-ha.googlecode.com/files/mha4mysql-manager-0.56.tar.gz tar xf mha4mysql-manager-0.56.tar.gz cd mha4mysql-manager-0.56 perl Makefile.PL make && make install
安裝完成後會在/usr/local/bin目錄下面生成如下腳本文件，前面已經說過這些腳本的做用，這裏再也不重複
[root@manager bin]# pwd /usr/local/bin [root@manager bin]# ll total 76 -r-xr-xr-x 1 root root 15498 Apr 20 10:58 apply_diff_relay_logs -r-xr-xr-x 1 root root 4807 Apr 20 10:58 filter_mysqlbinlog -r-xr-xr-x 1 root root 1995 Apr 20 11:33 masterha_check_repl -r-xr-xr-x 1 root root 1779 Apr 20 11:33 masterha_check_ssh -r-xr-xr-x 1 root root 1865 Apr 20 11:33 masterha_check_status -r-xr-xr-x 1 root root 3201 Apr 20 11:33 masterha_conf_host -r-xr-xr-x 1 root root 2517 Apr 20 11:33 masterha_manager -r-xr-xr-x 1 root root 2165 Apr 20 11:33 masterha_master_monitor -r-xr-xr-x 1 root root 2373 Apr 20 11:33 masterha_master_switch -r-xr-xr-x 1 root root 3749 Apr 20 11:33 masterha_secondary_check -r-xr-xr-x 1 root root 1739 Apr 20 11:33 masterha_stop -r-xr-xr-x 1 root root 7401 Apr 20 10:58 purge_relay_logs -r-xr-xr-x 1 root root 7263 Apr 20 10:58 save_binary_logs [root@manager bin]#
複製相關腳本到/usr/local/bin目錄(軟件包解壓縮後就有了，不是必須，由於這些腳本不完整，須要本身修改，這是軟件開發者留給咱們本身發揮的,若是開啓下面的任何一個腳本對應的參數，而對應這裏的腳本又沒有修改，則會拋錯，本身被坑的很慘)
[root@manager scripts]# pwd /root/mha4mysql-manager-0.56/samples/scripts [root@manager scripts]# ll total 32 -rwxr-xr-x 1 root root 3443 Jan 8 2012 master_ip_failover #自動切換時vip管理的腳本，不是必須，若是咱們使用keepalived的，咱們能夠本身編寫腳本完成對vip的管理，好比監控mysql，若是mysql異常，咱們中止keepalived就行，這樣vip就會自動漂移 -rwxr-xr-x 1 root root 9186 Jan 8 2012 master_ip_online_change #在線切換時vip的管理，不是必須，一樣能夠能夠自行編寫簡單的shell完成 -rwxr-xr-x 1 root root 11867 Jan 8 2012 power_manager #故障發生後關閉主機的腳本，不是必須 -rwxr-xr-x 1 root root 1360 Jan 8 2012 send_report #因故障切換後發送報警的腳本，不是必須，可自行編寫簡單的shell完成。 [root@manager scripts]# cp * /usr/local/bin/ [root@manager scripts]#
3.配置SSH登陸無密碼驗證（使用key登陸，工做中經常使用）個人測試環境已是使用key登陸，服務器之間無需密碼驗證的。關於配置使用key登陸，我想我再也不重複。可是有一點須要注意：不能禁止 password 登錄，不然會出現錯誤
4.搭建主從複製環境
注意：binlog-do-db 和 replicate-ignore-db 設置必須相同。 MHA 在啓動時候會檢測過濾規則，若是過濾規則不一樣，MHA 不啓動監控和故障轉移。
（1）在master上執行備份（10.0.0.50）
[root@master ~]# mysqldump -uroot -p --master-data=2 --single-transaction -R --triggers -A > all.sql
其中--master-data=2表明備份時刻記錄master的Binlog位置和Position，--single-transaction意思是獲取一致性快照，-R意思是備份存儲過程和函數，--triggres的意思是備份觸發器，-A表明備份全部的庫。更多信息請自行mysqldump --help查看。
（2）在master上建立複製用戶：
mysql> grant replication slave on *.* to 'repl'@'10.0.0.%' identified by '123456'; Query OK, 0 rows affected (0.00 sec) mysql> flush privileges; Query OK, 0 rows affected (0.00 sec) mysql>
（3）查看主庫備份時的binlog名稱和位置，MASTER_LOG_FILE和MASTER_LOG_POS：
#這個DBA哥們的獲取文件和位置方法，有點問題。
[root@master ~]# head -n 30 all.sql |grep 'CHANGE MASTER TO' -- CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=106;
#咱們仍是來點傳統的獲取方式，在master主機查看master狀態信息。
mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000005 | 795 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
（4）把備份複製到candicateMaster和slave，也就是10.0.0.60和10.0.0.70
scp all.sql 10.0.0.60:/data/ scp all.sql 10.0.0.70:/data/
（5）導入備份到candicateMaster，執行復制相關命令
mysql -uroot -p < /data/all.sql
mysql> CHANGE MASTER TO MASTER_HOST='10.0.0.50',MASTER_USER='repl', MASTER_PASSWORD='123',MASTER_LOG_FILE='mysql-bin.000001',MASTER_LOG_POS=106; Query OK, 0 rows affected (0.02 sec) mysql> start slave; Query OK, 0 rows affected (0.01 sec)
查看複製狀態（能夠看見覆製成功）：
[root@cadicate-master ~]# mysql -uroot -p123 -e "show slave status\G" | grep -E "Slave_SQL|Slave_IO" Slave_IO_State: Waiting for master to send event Slave_IO_Running: Yes Slave_SQL_Running: Yes Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
（6）在slave（10.0.0.70）上搭建複製環境，操做和上面同樣。
mysql -uroot -p < /data/all.sql
mysql> change master to -> master_host='10.0.0.50', -> master_user='repl', -> master_password='123', -> master_log_file='mysql-bin.000005', -> master_log_pos=795; Query OK, 0 rows affected, 2 warnings (0.01 sec) mysql> start slave; Query OK, 0 rows affected (0.01 sec)
# 查看線程狀態
[root@slave ~]# mysql -uroot -p123 -e "show slave status\G" |grep -E "Slave_SQL|Slave_IO" Slave_IO_State: Waiting for master to send event Slave_IO_Running: Yes Slave_SQL_Running: Yes Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
（7）兩臺slave服務器設置read_only（從庫對外提供讀服務，只因此沒有寫進配置文件，是由於隨時slave會提高爲master）
[root@cadicate-master ~]# mysql -uroot -p123 -e 'set global read_only=1'
[root@slave ~]# mysql -uroot -p123 -e 'set global read_only=1'
（8）建立監控用戶（在master上執行，也就是10.0.0.50）：
mysql> grant all privileges on *.* to 'root'@'10.0.0.%' identified by '123'; Query OK, 0 rows affected (0.00 sec) mysql> flush privileges; Query OK, 0 rows affected (0.01 sec)
到這裏整個集羣環境已經搭建完畢，剩下的就是配置MHA軟件了。
5.配置MHA
（1）建立MHA的工做目錄，而且建立相關配置文件（在軟件包解壓後的目錄裏面有樣例配置文件）
[root@manager ~]# mkdir -p /etc/masterha [root@manager ~]# cp mha4mysql-manager-0.56/samples/conf/app1.cnf /etc/masterha/
修改app1.cnf配置文件，修改後的文件內容以下（注意，配置文件中的註釋須要去掉，我這裏是爲了解釋清楚）：
[root@10.0.0.20 ~]# cat /etc/masterha/app1.cnf [server default] manager_workdir=/var/log/masterha/app1.log //設置manager的工做目錄 manager_log=/var/log/masterha/app1/manager.log //設置manager的日誌 master_binlog_dir=/data/mysql //設置master 保存binlog的位置，以便MHA能夠找到master的日誌，我這裏的也就是mysql的數據目錄 master_ip_failover_script= /usr/local/bin/master_ip_failover //設置自動failover時候的切換腳本 master_ip_online_change_script= /usr/local/bin/master_ip_online_change //設置手動切換時候的切換腳本 password=123456 //設置mysql中root用戶的密碼，這個密碼是前文中建立監控用戶的那個密碼 user=root 設置監控用戶root ping_interval=1 //設置監控主庫，發送ping包的時間間隔，默認是3秒，嘗試三次沒有迴應的時候自動進行railover remote_workdir=/tmp //設置遠端mysql在發生切換時binlog的保存位置 repl_password=123456 //設置複製用戶的密碼 repl_user=repl //設置複製環境中的複製用戶名 report_script=/usr/local/send_report //設置發生切換後發送的報警的腳本 secondary_check_script= /usr/local/bin/masterha_secondary_check -s server03 -s server02 shutdown_script="" //設置故障發生後關閉故障主機腳本（該腳本的主要做用是關閉主機放在發生腦裂,這裏沒有使用） ssh_user=root //設置ssh的登陸用戶名 [server1] hostname=10.0.0.50 port=3306 [server2] hostname=10.0.0.60 port=3306 candidate_master=1 //設置爲候選master，若是設置該參數之後，發生主從切換之後將會將此從庫提高爲主庫，即便這個主庫不是集羣中事件最新的slave check_repl_delay=0 //默認狀況下若是一個slave落後master 100M的relay logs的話，MHA將不會選擇該slave做爲一個新的master，由於對於這個slave的恢復須要花費很長時間，經過設置check_repl_delay=0,MHA觸發切換在選擇一個新的master的時候將會忽略複製延時，這個參數對於設置了candidate_master=1的主機很是有用，由於這個候選主在切換的過程當中必定是新的master [server3] hostname=10.0.0.70 port=3306
（2）設置relay log的清除方式（在每一個slave節點上）：
[root@cadicate-master ~]# mysql -e 'set global relay_log_purge=0' [root@slave ~]# mysql -e 'set global relay_log_purge=0'
注意：
MHA在發生切換的過程當中，從庫的恢復過程當中依賴於relay log的相關信息，因此這裏要將relay log的自動清除設置爲OFF，採用手動清除relay log的方式。在默認狀況下，從服務器上的中繼日誌會在SQL線程執行完畢後被自動刪除。可是在MHA環境中，這些中繼日誌在恢復其餘從服務器時可能會被用到，所以須要禁用中繼日誌的自動刪除功能。按期清除中繼日誌須要考慮到複製延時的問題。在ext3的文件系統下，刪除大的文件須要必定的時間，會致使嚴重的複製延時。爲了不復制延時，須要暫時爲中繼日誌建立硬連接，由於在linux系統中經過硬連接刪除大文件速度會很快。（在mysql數據庫中，刪除大表時，一般也採用創建硬連接的方式）
MHA節點中包含了pure_relay_logs命令工具，它能夠爲中繼日誌建立硬連接，執行SET GLOBAL relay_log_purge=1,等待幾秒鐘以便SQL線程切換到新的中繼日誌，再執行SET GLOBAL relay_log_purge=0。
pure_relay_logs腳本參數以下所示：
--user mysql 用戶名 --password mysql 密碼 --port 端口號 --workdir 指定建立relay log的硬連接的位置，默認是/var/tmp，因爲系統不一樣分區建立硬連接文件會失敗，故須要執行硬連接具體位置，成功執行腳本後，硬連接的中繼日誌文件被刪除 --disable_relay_log_purge 默認狀況下，若是relay_log_purge=1，腳本會什麼都不清理，自動退出，經過設定這個參數，當relay_log_purge=1的狀況下會將relay_log_purge設置爲0。清理relay log以後，最後將參數設置爲OFF。
（3）設置按期清理relay腳本（兩臺slave服務器）
[root@cadicate-master ~]# cat purge_relay_log.sh #!/bin/bash user=root passwd=123 port=3306 log_dir='/data/masterha/log' work_dir='/data' purge='/usr/local/bin/purge_relay_logs' if [ ! -d $log_dir ] then mkdir $log_dir -p fi $purge --user=$user --password=$passwd --disable_relay_log_purge --port=$port --workdir=$work_dir >> $log_dir/purge_relay_logs.log 2>&1
添加到crontab按期執行
[root@cadicate-master ~]# crontab -l 0 4 * * * /bin/bash /root/purge_relay_log.sh
purge_relay_logs腳本刪除中繼日誌不會阻塞SQL線程。下面咱們手動執行看看什麼狀況。
[root@cadicate-master ~]# purge_relay_logs --user=root --password=123 --port=3306 -disable_relay_log_purge --workdir=/data/
2018-03-09 15:45:59: purge_relay_logs script started.
Found relay_log.info: /mysql/data/relay-log.info
Opening /mysql/data/cadicate-master-relay-bin.000001 ..
Opening /mysql/data/cadicate-master-relay-bin.000002 ..
Opening /mysql/data/cadicate-master-relay-bin.000003 ..
Executing SET GLOBAL relay_log_purge=1; FLUSH LOGS; sleeping a few seconds so that SQL thread can delete older relay log files (if it keeps up); SET GLOBAL relay_log_purge=0; .. ok.
2018-03-09 15:46:02: All relay log purging operations succeeded.
6.檢查SSH配置
檢查MHA Manger到全部MHA Node的SSH鏈接狀態：
[root@manager .ssh]# masterha_check_ssh --conf=/etc/masterha/app1.cnf
Fri Mar 9 16:29:14 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri Mar 9 16:29:14 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Fri Mar 9 16:29:14 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Fri Mar 9 16:29:14 2018 - [info] Starting SSH connection tests..
Fri Mar 9 16:29:14 2018 - [debug]
Fri Mar 9 16:29:14 2018 - [debug] Connecting via SSH from root@10.0.0.50(10.0.0.50:22) to root@10.0.0.60(10.0.0.60:22)..
Fri Mar 9 16:29:14 2018 - [debug] ok.
Fri Mar 9 16:29:14 2018 - [debug] Connecting via SSH from root@10.0.0.50(10.0.0.50:22) to root@10.0.0.70(10.0.0.70:22)..
Fri Mar 9 16:29:14 2018 - [debug] ok.
Fri Mar 9 16:29:15 2018 - [debug]
Fri Mar 9 16:29:14 2018 - [debug] Connecting via SSH from root@10.0.0.60(10.0.0.60:22) to root@10.0.0.50(10.0.0.50:22)..
Fri Mar 9 16:29:15 2018 - [debug] ok.
Fri Mar 9 16:29:15 2018 - [debug] Connecting via SSH from root@10.0.0.60(10.0.0.60:22) to root@10.0.0.70(10.0.0.70:22)..
Fri Mar 9 16:29:15 2018 - [debug] ok.
Fri Mar 9 16:29:15 2018 - [debug]
Fri Mar 9 16:29:15 2018 - [debug] Connecting via SSH from root@10.0.0.70(10.0.0.70:22) to root@10.0.0.50(10.0.0.50:22)..
Fri Mar 9 16:29:15 2018 - [debug] ok.
Fri Mar 9 16:29:15 2018 - [debug] Connecting via SSH from root@10.0.0.70(10.0.0.70:22) to root@10.0.0.60(10.0.0.60:22)..
Fri Mar 9 16:29:15 2018 - [debug] ok.
Fri Mar 9 16:29:15 2018 - [info] All SSH connection tests passed successfully.
# 所有爲ok 有error證實密鑰配對存在問題。
能夠看見各個節點ssh驗證都是ok的。
7.檢查整個複製環境情況。
經過masterha_check_repl腳本查看整個集羣的狀態
#這個折騰很久，這位dba大哥的binlog日誌路徑是/data/mysql 個人/mysql/data 好蛋疼，一頓瞎J8百度，徹底無卵用，正確姿式就是耐心分析日誌。
[root@manager ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf
Fri Mar 9 17:21:22 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri Mar 9 17:21:22 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Fri Mar 9 17:21:22 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Fri Mar 9 17:21:22 2018 - [info] MHA::MasterMonitor version 0.56.
Fri Mar 9 17:21:22 2018 - [info] GTID failover mode = 0
Fri Mar 9 17:21:22 2018 - [info] Dead Servers:
Fri Mar 9 17:21:22 2018 - [info] Alive Servers:
Fri Mar 9 17:21:22 2018 - [info] 10.0.0.50(10.0.0.50:3306)
Fri Mar 9 17:21:22 2018 - [info] 10.0.0.60(10.0.0.60:3306)
Fri Mar 9 17:21:22 2018 - [info] 10.0.0.70(10.0.0.70:3306)
Fri Mar 9 17:21:22 2018 - [info] Alive Slaves:
Fri Mar 9 17:21:22 2018 - [info] 10.0.0.60(10.0.0.60:3306) Version=5.6.16-log (oldest major version between slaves) log-bin:enabled
Fri Mar 9 17:21:22 2018 - [info] Replicating from 10.0.0.50(10.0.0.50:3306)
Fri Mar 9 17:21:22 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Fri Mar 9 17:21:22 2018 - [info] 10.0.0.70(10.0.0.70:3306) Version=5.6.16 (oldest major version between slaves) log-bin:disabled
Fri Mar 9 17:21:22 2018 - [info] Replicating from 10.0.0.50(10.0.0.50:3306)
Fri Mar 9 17:21:22 2018 - [info] Current Alive Master: 10.0.0.50(10.0.0.50:3306)
Fri Mar 9 17:21:22 2018 - [info] Checking slave configurations..
Fri Mar 9 17:21:22 2018 - [warning] log-bin is not set on slave 10.0.0.70(10.0.0.70:3306). This host cannot be a master.
Fri Mar 9 17:21:22 2018 - [info] Checking replication filtering settings..
Fri Mar 9 17:21:22 2018 - [info] binlog_do_db= , binlog_ignore_db=
Fri Mar 9 17:21:22 2018 - [info] Replication filtering check ok.
Fri Mar 9 17:21:22 2018 - [info] GTID (with auto-pos) is not supported
Fri Mar 9 17:21:22 2018 - [info] Starting SSH connection tests..
Fri Mar 9 17:21:23 2018 - [info] All SSH connection tests passed successfully.
Fri Mar 9 17:21:23 2018 - [info] Checking MHA Node version..
Fri Mar 9 17:21:24 2018 - [info] Version check ok.
Fri Mar 9 17:21:24 2018 - [info] Checking SSH publickey authentication settings on the current master..
Fri Mar 9 17:21:24 2018 - [info] HealthCheck: SSH to 10.0.0.50 is reachable.
Fri Mar 9 17:21:24 2018 - [info] Master MHA Node version is 0.56.
Fri Mar 9 17:21:24 2018 - [info] Checking recovery script configurations on 10.0.0.50(10.0.0.50:3306)..
Fri Mar 9 17:21:24 2018 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/mysql/data --output_file=/tmp/save_binary_logs_test --manager_version=0.56 --start_file=mysql-bin.000005
Fri Mar 9 17:21:24 2018 - [info] Connecting to root@10.0.0.50(10.0.0.50:22)..
Creating /tmp if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /mysql/data, up to mysql-bin.000005
Fri Mar 9 17:21:24 2018 - [info] Binlog setting check done.
Fri Mar 9 17:21:24 2018 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Fri Mar 9 17:21:24 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=10.0.0.60 --slave_ip=10.0.0.60 --slave_port=3306 --workdir=/tmp --target_version=5.6.16-log --manager_version=0.56 --relay_log_info=/mysql/data/relay-log.info --relay_dir=/mysql/data/ --slave_pass=xxx
Fri Mar 9 17:21:24 2018 - [info] Connecting to root@10.0.0.60(10.0.0.60:22)..
Checking slave recovery environment settings..
Opening /mysql/data/relay-log.info ... ok.
Relay log found at /mysql/data, up to cadicate-master-relay-bin.000004
Temporary relay log file is /mysql/data/cadicate-master-relay-bin.000004
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Fri Mar 9 17:21:25 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=10.0.0.70 --slave_ip=10.0.0.70 --slave_port=3306 --workdir=/tmp --target_version=5.6.16 --manager_version=0.56 --relay_log_info=/mysql/data/relay-log.info --relay_dir=/mysql/data/ --slave_pass=xxx
Fri Mar 9 17:21:25 2018 - [info] Connecting to root@10.0.0.70(10.0.0.70:22)..
Checking slave recovery environment settings..
Opening /mysql/data/relay-log.info ... ok.
Relay log found at /mysql/data, up to slave-relay-bin.000002
Temporary relay log file is /mysql/data/slave-relay-bin.000002
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Fri Mar 9 17:21:25 2018 - [info] Slaves settings check done.
Fri Mar 9 17:21:25 2018 - [info]
10.0.0.50(10.0.0.50:3306) (current master)
+--10.0.0.60(10.0.0.60:3306)
+--10.0.0.70(10.0.0.70:3306)
Fri Mar 9 17:21:25 2018 - [info] Checking replication health on 10.0.0.60..
Fri Mar 9 17:21:25 2018 - [info] ok.
Fri Mar 9 17:21:25 2018 - [info] Checking replication health on 10.0.0.70..
Fri Mar 9 17:21:25 2018 - [info] ok.
Fri Mar 9 17:21:25 2018 - [warning] master_ip_failover_script is not defined.
Fri Mar 9 17:21:25 2018 - [warning] shutdown_script is not defined.
Fri Mar 9 17:21:25 2018 - [info] Got exit code 0 (Not master dead).
MySQL Replication Health is OK. （這邊顯示ok纔是真是ok否則腳本不能作出正確判斷，它會亂來的你懂的）
若是發現以下錯誤：
Can't exec "mysqlbinlog": No such file or directory at /usr/local/share/perl5/MHA/BinlogManager.pm line 99. mysqlbinlog version not found!
Testing mysql connection and privileges..sh: mysql: command not found
解決方法以下，添加軟鏈接（全部節點）
ln -s /usr/local/mysql/bin/mysqlbinlog /usr/local/bin/mysqlbinlog
ln -s /usr/local/mysql/bin/mysql /usr/local/bin/mysql
因此先暫時註釋master_ip_failover_script= /usr/local/bin/master_ip_failover這個選項。後面引入keepalived後和修改該腳本之後再開啓該選項。
[root@manager ~]# grep master_ip_failover /etc/masterha/app1.cnf #master_ip_failover_script= /usr/local/bin/master_ip_failover
8.檢查MHA Manager的狀態：
經過master_check_status腳本查看Manager的狀態：
[root@manager ~]# masterha_check_status --conf=/etc/masterha/app1.cnf app1 is stopped(2:NOT_RUNNING).
注意：若是正常，會顯示"PING_OK"，不然會顯示"NOT_RUNNING"，這表明MHA監控沒有開啓。
9.開啓MHA Manager監控
[root@manager ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1.log/manager.log 2>&1 & [1] 2201
啓動參數介紹：
--remove_dead_master_conf 該參數表明當發生主從切換後，老的主庫的ip將會從配置文件中移除。
--manger_log 日誌存放位置
--ignore_last_failover 在缺省狀況下，若是MHA檢測到連續發生宕機，且兩次宕機間隔不足8小時的話，則不會進行Failover，之因此這樣限制是爲了不ping-pong效應。該參數表明忽略上次MHA觸發切換產生的文件，默認狀況下，MHA發生切換後會在日誌目錄，也就是上面我設置的/data產生app1.failover.complete文件，下次再次切換的時候若是發現該目錄下存在該文件將不容許觸發切換，除非在第一次切換後收到刪除該文件，爲了方便，這裏設置爲--ignore_last_failover。
# 查看MHA Manager監控是否正常：
[root@manager ~]# masterha_check_status --conf=/etc/masterha/app1.cnf app1 (pid:2211) is running(0:PING_OK), master:10.0.0.50
[root@manager ~]# masterha_check_status --conf=/etc/masterha/app1.cnf app1 (pid:2211) is running(0:PING_OK), master:10.0.0.50
[root@manager ~]# masterha_check_status --conf=/etc/masterha/app1.cnf app1 (pid:2211) is running(0:PING_OK), master:10.0.0.50
能夠看見已經在監控了，並且master的主機爲10.0.0.50
10.查看啓動日誌
[root@manager ~]# tail -n20 /var/log/masterha/app1/manager.log Checking slave recovery environment settings.. Opening /mysql/data/relay-log.info ... ok. Relay log found at /mysql/data, up to slave-relay-bin.000002 Temporary relay log file is /mysql/data/slave-relay-bin.000002 Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Fri Mar 9 17:34:40 2018 - [info] Slaves settings check done. Fri Mar 9 17:34:40 2018 - [info] 10.0.0.50(10.0.0.50:3306) (current master) +--10.0.0.60(10.0.0.60:3306) +--10.0.0.70(10.0.0.70:3306) Fri Mar 9 17:34:40 2018 - [warning] master_ip_failover_script is not defined. Fri Mar 9 17:34:40 2018 - [warning] shutdown_script is not defined. Fri Mar 9 17:34:40 2018 - [info] Set master ping interval 1 seconds. Fri Mar 9 17:34:40 2018 - [info] Set secondary check script: /usr/local/bin/masterha_secondary_check -s server03 -s server02 Fri Mar 9 17:34:40 2018 - [info] Starting ping health check on 10.0.0.50(10.0.0.50:3306).. Fri Mar 9 17:34:40 2018 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
其中"Ping(SELECT) succeeded, waiting until MySQL doesn't respond.."說明整個系統已經開始監控了。
11.關閉MHA Manage監控
關閉很簡單，使用masterha_stop命令完成。
[root@manager ~]# masterha_stop --conf=/etc/masterha/app1.cnf Stopped app1 successfully. [1]+ Exit 1 nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 [root@manager ~]# masterha_check_status --conf=/etc/masterha/app1.cnf app1 is stopped(2:NOT_RUNNING).
12.配置VIP
vip配置能夠採用兩種方式，一種經過keepalived的方式管理虛擬ip的浮動；另一種經過腳本方式啓動虛擬ip的方式（即不須要keepalived或者heartbeat相似的軟件）。
1.keepalived方式管理虛擬ip，keepalived配置方法以下：
（1）下載軟件進行並進行安裝（兩臺master，準確的說一臺是master，另一臺是備選master，在沒有切換之前是slave）：
[root@master ~]# wget http://www.keepalived.org/software/keepalived-1.2.12.tar.gz
tar xf keepalived-1.2.12.tar.gz cd keepalived-1.2.12 ./configure --prefix=/usr/local/keepalived make && make install cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/ cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/ mkdir /etc/keepalived cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/ cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
（2）配置keepalived的配置文件，在master上配置（10.0.0.50）
[root@master ~]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { saltstack@163.com } notification_email_from dba@dbserver.com smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id MySQL-HA } vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 51 priority 150 advert_int 1 nopreempt authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.0.10.88 } }
其中router_id MySQL HA表示設定keepalived組的名稱，將192.168.0.88這個虛擬ip綁定到該主機的eth1網卡上，而且設置了狀態爲backup模式，將keepalived的模式設置爲非搶佔模式（nopreempt），priority 150表示設置的優先級爲150。下面的配置略有不一樣，可是都是一個意思。
在候選master上配置（10.0.0.60）
[root@cadicate-master ~]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { saltstack@163.com } notification_email_from dba@dbserver.com smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id MySQL-HA } vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 51 priority 120 advert_int 1 nopreempt authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.0.0.88 } }
（3）啓動keepalived服務，在master上啓動並查看日誌
[root@master ~]# /etc/init.d/keepalived start Starting keepalived: [ OK ] [root@master ~]# tail -f /var/log/messages Mar 10 04:27:52 master kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes) Mar 10 04:27:52 master kernel: IPVS: ipvs loaded. Mar 10 04:27:52 master Keepalived_healthcheckers[12025]: Opening file '/etc/keepalived/keepalived.conf'. Mar 10 04:27:52 master Keepalived_healthcheckers[12025]: Configuration is using : 7241 Bytes Mar 10 04:27:52 master Keepalived_healthcheckers[12025]: Using LinkWatch kernel netlink reflector... Mar 10 04:27:55 master Keepalived_vrrp[12026]: VRRP_Instance(VI_1) Transition to MASTER STATE Mar 10 04:27:56 master Keepalived_vrrp[12026]: VRRP_Instance(VI_1) Entering MASTER STATE Mar 10 04:27:56 master Keepalived_vrrp[12026]: VRRP_Instance(VI_1) setting protocol VIPs. Mar 10 04:27:56 master Keepalived_vrrp[12026]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 10.0.0.88 Mar 10 04:27:56 master Keepalived_healthcheckers[12025]: Netlink reflector reports IP 10.0.0.88 added Mar 10 04:28:01 master Keepalived_vrrp[12026]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 10.0.0.88
發現已經將虛擬ip 10.0.0.88綁定了網卡eth0上。
（4）查看綁定狀況
[root@master ~]# ip addr |grep eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
inet 10.0.0.50/8 brd 10.255.255.255 scope global eth0
inet 10.0.0.88/32 scope global eth0
在另一臺服務器，候選master上啓動keepalived服務，並觀察
[root@cadicate-master ~]# /etc/init.d/keepalived start ; tail -f /var/log/messages
Starting keepalived: [ OK ]
Mar 11 10:21:24 cadicate-master ntpd[1159]: 0.0.0.0 c012 02 freq_set kernel -26.844 PPM
Mar 11 10:21:31 cadicate-master ntpd[1159]: 0.0.0.0 c615 05 clock_sync
Mar 11 10:24:56 cadicate-master Keepalived[1457]: Starting Keepalived v1.2.12 (03/09,2018)
Mar 11 10:24:56 cadicate-master Keepalived[1459]: Starting Healthcheck child process, pid=1461
Mar 11 10:24:56 cadicate-master Keepalived[1459]: Starting VRRP child process, pid=1463
Mar 11 10:24:56 cadicate-master Keepalived_vrrp[1463]: Netlink reflector reports IP 10.0.0.60 added
Mar 11 10:24:56 cadicate-master Keepalived_vrrp[1463]: Netlink reflector reports IP fe80::20c:29ff:fe39:2d82 added
Mar 11 10:24:56 cadicate-master Keepalived_vrrp[1463]: Registering Kernel netlink reflector
Mar 11 10:24:56 cadicate-master Keepalived_vrrp[1463]: Registering Kernel netlink command channel
Mar 11 10:24:56 cadicate-master Keepalived_vrrp[1463]: Registering gratuitous ARP shared channel
Mar 11 10:24:56 cadicate-master Keepalived_healthcheckers[1461]: Netlink reflector reports IP 10.0.0.60 added
Mar 11 10:24:56 cadicate-master Keepalived_healthcheckers[1461]: Netlink reflector reports IP fe80::20c:29ff:fe39:2d82 added
Mar 11 10:24:56 cadicate-master Keepalived_healthcheckers[1461]: Registering Kernel netlink reflector
Mar 11 10:24:56 cadicate-master Keepalived_healthcheckers[1461]: Registering Kernel netlink command channel
Mar 11 10:24:56 cadicate-master kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
Mar 11 10:24:56 cadicate-master kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
Mar 11 10:24:56 cadicate-master kernel: IPVS: ipvs loaded.
Mar 11 10:25:16 cadicate-master Keepalived_vrrp[1463]: Opening file '/etc/keepalived/keepalived.conf'.
Mar 11 10:25:16 cadicate-master Keepalived_vrrp[1463]: Configuration is using : 62858 Bytes
Mar 11 10:25:16 cadicate-master Keepalived_vrrp[1463]: Using LinkWatch kernel netlink reflector...
Mar 11 10:25:16 cadicate-master Keepalived_vrrp[1463]: VRRP_Instance(VI_1) Entering BACKUP STATE
Mar 11 10:25:16 cadicate-master Keepalived_vrrp[1463]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Mar 11 10:25:16 cadicate-master Keepalived_healthcheckers[1461]: Opening file '/etc/keepalived/keepalived.conf'.
Mar 11 10:25:16 cadicate-master Keepalived_healthcheckers[1461]: Configuration is using : 7113 Bytes
Mar 11 10:25:16 cadicate-master Keepalived_healthcheckers[1461]: Using LinkWatch kernel netlink reflector...
從上面的信息能夠看到keepalived已經配置成功。
注意：
上面兩臺服務器的keepalived都設置爲了BACKUP模式，在keepalived中2種模式，分別是master->backup模式和backup->backup模式。這兩種模式有很大區別。在master->backup模式下，一旦主庫宕機，虛擬ip會自動漂移到從庫，當主庫修復後，keepalived啓動後，還會把虛擬ip搶佔過來，即便設置了非搶佔模式（nopreempt）搶佔ip的動做也會發生。在backup->backup模式下，當主庫宕機後虛擬ip會自動漂移到從庫上，當原主庫恢復和keepalived服務啓動後，並不會搶佔新主的虛擬ip，即便是優先級高於從庫的優先級別，也不會發生搶佔。爲了減小ip漂移次數，一般是把修復好的主庫當作新的備庫。
（5）MHA引入keepalived（MySQL服務進程掛掉時經過MHA 中止keepalived）:
要想把keepalived服務引入MHA，咱們只須要修改切換是觸發的腳本文件master_ip_failover便可，在該腳本中添加在master發生宕機時對keepalived的處理。
編輯腳本/usr/local/bin/master_ip_failover，修改後以下，我對perl不熟悉，因此我這裏完整貼出該腳本（10.0.0.20）。#那位DBA哥們說是在master上修改，我怎麼感受不像不是那麼回事。
在MHA Manager修改腳本修改後的內容以下（參考資料比較少）：
#!/usr/bin/env perl use strict; use warnings FATAL => 'all'; use Getopt::Long; my ( $command, $ssh_user, $orig_master_host, $orig_master_ip, $orig_master_port, $new_master_host, $new_master_ip, $new_master_port ); my $vip = '10.0.0.88'; my $ssh_start_vip = "/etc/init.d/keepalived start"; my $ssh_stop_vip = "/etc/init.d/keepalived stop"; GetOptions( 'command=s' => \$command, 'ssh_user=s' => \$ssh_user, 'orig_master_host=s' => \$orig_master_host, 'orig_master_ip=s' => \$orig_master_ip, 'orig_master_port=i' => \$orig_master_port, 'new_master_host=s' => \$new_master_host, 'new_master_ip=s' => \$new_master_ip, 'new_master_port=i' => \$new_master_port, ); exit &main(); sub main { print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n"; if ( $command eq "stop" || $command eq "stopssh" ) { my $exit_code = 1; eval { print "Disabling the VIP on old master: $orig_master_host \n"; &stop_vip(); $exit_code = 0; }; if ($@) { warn "Got Error: $@\n"; exit $exit_code; } exit $exit_code; } elsif ( $command eq "start" ) { my $exit_code = 10; eval { print "Enabling the VIP - $vip on the new master - $new_master_host \n"; &start_vip(); $exit_code = 0; }; if ($@) { warn $@; exit $exit_code; } exit $exit_code; } elsif ( $command eq "status" ) { print "Checking the Status of the script.. OK \n"; #`ssh $ssh_user\@cluster1 \" $ssh_start_vip \"`; exit 0; } else { &usage(); exit 1; } } # A simple system call that enable the VIP on the new master sub start_vip() { `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`; } # A simple system call that disable the VIP on the old_master sub stop_vip() { return 0 unless ($ssh_user); `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`; } sub usage { print "Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n"; }
如今已經修改這個腳本了，咱們如今打開在上面提到過的參數，再檢查集羣狀態，看是否會報錯。
[root@manager ~]# grep 'master_ip_failover_script' /etc/masterha/app1.cnf master_ip_failover_script= /usr/local/bin/master_ip_failover
[root@manager ~]# grep 'master_ip_failover_script' /etc/masterha/app1.cnf
at Mar 10 05:22:36 2018 - [info] Slaves settings check done. Sat Mar 10 05:22:36 2018 - [info] 10.0.0.50(10.0.0.50:3306) (current master) +--10.0.0.60(10.0.0.60:3306) +--10.0.0.70(10.0.0.70:3306) Sat Mar 10 05:22:36 2018 - [info] Checking replication health on 10.0.0.60.. Sat Mar 10 05:22:36 2018 - [info] ok. Sat Mar 10 05:22:36 2018 - [info] Checking replication health on 10.0.0.70.. Sat Mar 10 05:22:36 2018 - [info] ok. Sat Mar 10 05:22:36 2018 - [info] Checking master_ip_failover_script status: Sat Mar 10 05:22:36 2018 - [info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=10.0.0.50 --orig_master_ip=10.0.0.50 --orig_master_port=3306 IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start=== Checking the Status of the script.. OK Sat Mar 10 05:22:36 2018 - [info] OK. Sat Mar 10 05:22:36 2018 - [warning] shutdown_script is not defined. Sat Mar 10 05:22:36 2018 - [info] Got exit code 0 (Not master dead). MySQL Replication Health is OK. [root@manager ~]#
能夠看見已經沒有報錯了。我也哈哈
/usr/local/bin/master_ip_failover添加或者修改的內容意思是當主庫數據庫發生故障時，會觸發MHA切換，MHA Manager會停掉主庫上的keepalived服務，觸發虛擬ip漂移到備選從庫，從而完成切換。固然能夠在keepalived裏面引入腳本，這個腳本監控mysql是否正常運行，若是不正常，則調用該腳本殺掉keepalived進程。
2.經過腳本的方式管理VIP。這裏是修改/usr/local/bin/master_ip_failover，也可使用其餘的語言完成，好比php語言。使用php腳本編寫的failover這裏就不介紹了。修改完成後內容以下，並且若是使用腳本管理vip的話，須要手動在master服務器上綁定一個vip
[root@master ~]# /sbin/ifconfig eth1:1 10.0.0.88/24
#!/usr/bin/env perl use strict; use warnings FATAL => 'all'; use Getopt::Long; my ( $command, $ssh_user, $orig_master_host, $orig_master_ip, $orig_master_port, $new_master_host, $new_master_ip, $new_master_port ); my $vip = '10.0.0.88/24'; my $key = '1'; my $ssh_start_vip = "/sbin/ifconfig eth1:$key $vip"; my $ssh_stop_vip = "/sbin/ifconfig eth1:$key down"; GetOptions( 'command=s' => \$command, 'ssh_user=s' => \$ssh_user, 'orig_master_host=s' => \$orig_master_host, 'orig_master_ip=s' => \$orig_master_ip, 'orig_master_port=i' => \$orig_master_port, 'new_master_host=s' => \$new_master_host, 'new_master_ip=s' => \$new_master_ip, 'new_master_port=i' => \$new_master_port, ); exit &main(); sub main { print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n"; if ( $command eq "stop" || $command eq "stopssh" ) { my $exit_code = 1; eval { print "Disabling the VIP on old master: $orig_master_host \n"; &stop_vip(); $exit_code = 0; }; if ($@) { warn "Got Error: $@\n"; exit $exit_code; } exit $exit_code; } elsif ( $command eq "start" ) { my $exit_code = 10; eval { print "Enabling the VIP - $vip on the new master - $new_master_host \n"; &start_vip(); $exit_code = 0; }; if ($@) { warn $@; exit $exit_code; } exit $exit_code; } elsif ( $command eq "status" ) { print "Checking the Status of the script.. OK \n"; exit 0; } else { &usage(); exit 1; } } sub start_vip() { `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`; } sub stop_vip() { return 0 unless ($ssh_user); `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`; } sub usage { print "Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n"; }
爲了防止腦裂發生，推薦生產環境採用腳本的方式來管理虛擬ip，而不是使用keepalived來完成。到此爲止，基本MHA集羣已經配置完畢。接下來就是實際的測試環節了。經過一些測試來看一下MHA究竟是如何進行工做的。下面將從MHA自動failover，咱們手動failover，在線切換三種方式來介紹MHA的工做狀況。
一.自動Failover（必須先啓動MHA Manager，不然沒法自動切換，固然手動切換不須要開啓MHA Manager監控。）
（1）使用sysbench生成測試數據（使用yum快速安裝）
yum install sysbench -y
也能夠編譯安裝，我這裏使用了編譯安裝的方式：http://www.cnblogs.com/benjamin77/p/8537226.html
在主庫（10.0.0.50）上進行sysbench數據生成，在sbtest庫下生成sbtest表，共100W記錄。
#哈哈 記得進數據庫提早建立一個叫sbtest的庫。（這名字有個性）
[root@master sysbench]# pwd
/root/sysbench-0.4.12-1.1/sysbench
[root@master sysbench]# ./sysbench --test=tests/db/oltp.lua --oltp-table-size=1000000 --oltp-read-only=off --init-rng=on --num-threads=16 --max-requests=0 --oltp-dist-type=uniform --max-time=1800 --mysql-user=root --mysql-socket=/usr/local/mysql/mysql.sock --mysql-password=123 --db-driver=mysql --mysql-table-engine=innodb --oltp-test-mode=complex prepare
sysbench 0.5: multi-threaded system evaluation benchmark
Creating table 'sbtest1'...
Inserting 1000000 records into 'sbtest1'
（2）停掉slave sql線程，模擬主從延時。（10.0.0.60）
mysql> stop slave io_thread; Query OK, 0 rows affected (0.08 sec) mysql>
另一臺slave咱們沒有中止io線程，因此還在繼續接收日誌。
（3）模擬sysbench壓力測試。
在主庫上（10.0.0.50）進行壓力測試，持續時間爲3分鐘，產生大量的binlog。
[root@master sysbench]# ./sysbench --test=tests/db/oltp.lua --oltp-table-size=1000000 --oltp-read-only=off --init-rng=on --num-threads=16
--max-requests=0 --oltp-dist-type=uniform --max-time=180 --mysql-user=root --mysql-socket=/usr/local/mysql/mysql.sock --mysql-password=123 --db-driver=mysql --mysql-table-engine=innodb --oltp-test-mode=complex run sysbench 0.5: multi-threaded system evaluation benchmark Running the test with following options: Number of threads: 16 Random number generator seed is 0 and will be ignored Threads started! OLTP test statistics: queries performed: read: 324660 write: 92760 other: 46380 total: 463800 transactions: 23190 (128.75 per sec.) deadlocks: 0 (0.00 per sec.) read/write requests: 417420 (2317.42 per sec.) other operations: 46380 (257.49 per sec.) General statistics: total time: 180.1229s total number of events: 23190 total time taken by event execution: 2879.0035s response time: min: 12.24ms avg: 124.15ms max: 2239.50ms approx. 95 percentile: 372.38ms Threads fairness: events (avg/stddev): 1449.3750/10.03 execution time (avg/stddev): 179.9377/0.01
（4）開啓slave（10.0.0.60）上的IO線程，追趕落後於master的binlog。
mysql> start slave io_thread; Query OK, 0 rows affected (0.00 sec)
（5）殺掉主庫mysql進程，模擬主庫發生故障，進行自動failover操做。
[root@master sysbench]# pkill -9 mysqld
（6）查看MHA切換日誌，瞭解整個切換過程，在10.0.0.20上查看日誌：
Sun Mar 11 13:55:07 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Mar 11 13:55:07 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Sun Mar 11 13:55:07 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Sun Mar 11 13:55:07 2018 - [info] MHA::MasterMonitor version 0.56. Sun Mar 11 13:55:07 2018 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 10.0.0.50:3306 from which slave 10.0.0.60(10.0.0.60:3306) replicates is not defined in the configuration file! Sun Mar 11 13:55:07 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln424] Error happened on checking configurations. at /usr/local/share/perl5/MHA/MasterMonitor.pm line 326 Sun Mar 11 13:55:07 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln523] Error happened on monitoring servers. Sun Mar 11 13:55:07 2018 - [info] Got exit code 1 (Not master dead). ^C [root@manager ~]# > /var/log/masterha/app1/manager.log [root@manager ~]# tail -f /var/log/masterha/app1/manager.log Sun Mar 11 13:55:48 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Mar 11 13:55:48 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Sun Mar 11 13:55:48 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Sun Mar 11 13:55:48 2018 - [info] MHA::MasterMonitor version 0.56. Sun Mar 11 13:55:49 2018 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 10.0.0.50:3306 from which slave 10.0.0.60(10.0.0.60:3306) replicates is not defined in the configuration file! Sun Mar 11 13:55:49 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln424] Error happened on checking configurations. at /usr/local/share/perl5/MHA/MasterMonitor.pm line 326 Sun Mar 11 13:55:49 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln523] Error happened on monitoring servers. Sun Mar 11 13:55:49 2018 - [info] Got exit code 1 (Not master dead). tail: /var/log/masterha/app1/manager.log: file truncated Sun Mar 11 14:02:04 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Mar 11 14:02:04 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Sun Mar 11 14:02:04 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Sun Mar 11 14:02:04 2018 - [info] MHA::MasterMonitor version 0.56. Sun Mar 11 14:02:04 2018 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 10.0.0.50:3306 from which slave 10.0.0.60(10.0.0.60:3306) replicates is not defined in the configuration file! Sun Mar 11 14:02:04 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln424] Error happened on checking configurations. at /usr/local/share/perl5/MHA/MasterMonitor.pm line 326 Sun Mar 11 14:02:04 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln523] Error happened on monitoring servers. Sun Mar 11 14:02:04 2018 - [info] Got exit code 1 (Not master dead). tail: /var/log/masterha/app1/manager.log: file truncated Sun Mar 11 14:03:50 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Mar 11 14:03:50 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Sun Mar 11 14:03:50 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Sun Mar 11 14:03:50 2018 - [info] MHA::MasterMonitor version 0.56. Sun Mar 11 14:03:50 2018 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 10.0.0.50:3306 from which slave 10.0.0.60(10.0.0.60:3306) replicates is not defined in the configuration file! Sun Mar 11 14:03:50 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln424] Error happened on checking configurations. at /usr/local/share/perl5/MHA/MasterMonitor.pm line 326 Sun Mar 11 14:03:50 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln523] Error happened on monitoring servers. Sun Mar 11 14:03:50 2018 - [info] Got exit code 1 (Not master dead). tail: /var/log/masterha/app1/manager.log: file truncated Sun Mar 11 14:15:27 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Mar 11 14:15:27 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Sun Mar 11 14:15:27 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Sun Mar 11 14:15:27 2018 - [info] MHA::MasterMonitor version 0.56. Sun Mar 11 14:15:27 2018 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln781] Multi-master configuration is detected, but two or more masters are either writable (read-only is not set) or dead! Check configurations for details. Master configurations are as below: Master 10.0.0.50(10.0.0.50:3306) Master 10.0.0.60(10.0.0.60:3306), replicating from 10.0.0.50(10.0.0.50:3306) Sun Mar 11 14:15:27 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln424] Error happened on checking configurations. at /usr/local/share/perl5/MHA/MasterMonitor.pm line 326 Sun Mar 11 14:15:27 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln523] Error happened on monitoring servers. Sun Mar 11 14:15:27 2018 - [info] Got exit code 1 (Not master dead). tail: /var/log/masterha/app1/manager.log: file truncated Sun Mar 11 14:18:58 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Mar 11 14:18:58 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Sun Mar 11 14:18:58 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Sun Mar 11 14:18:58 2018 - [info] MHA::MasterMonitor version 0.56. Sun Mar 11 14:18:58 2018 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln781] Multi-master configuration is detected, but two or more masters are either writable (read-only is not set) or dead! Check configurations for details. Master configurations are as below: Master 10.0.0.50(10.0.0.50:3306) Master 10.0.0.60(10.0.0.60:3306), replicating from 10.0.0.50(10.0.0.50:3306) Sun Mar 11 14:18:58 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln424] Error happened on checking configurations. at /usr/local/share/perl5/MHA/MasterMonitor.pm line 326 Sun Mar 11 14:18:58 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln523] Error happened on monitoring servers. Sun Mar 11 14:18:58 2018 - [info] Got exit code 1 (Not master dead). tail: /var/log/masterha/app1/manager.log: file truncated Sun Mar 11 14:35:38 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Mar 11 14:35:38 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Sun Mar 11 14:35:38 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Sun Mar 11 14:35:38 2018 - [info] MHA::MasterMonitor version 0.56. Sun Mar 11 14:35:38 2018 - [info] Multi-master configuration is detected. Current primary(writable) master is 10.0.0.50(10.0.0.50:3306) Sun Mar 11 14:35:38 2018 - [info] Master configurations are as below: Master 10.0.0.50(10.0.0.50:3306) Master 10.0.0.60(10.0.0.60:3306), replicating from 10.0.0.50(10.0.0.50:3306), read-only Sun Mar 11 14:35:38 2018 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln726] Slave 10.0.0.70(10.0.0.70:3306) replicates from 10.0.0.60:3306, but real master is 10.0.0.50(10.0.0.50:3306)! Sun Mar 11 14:35:38 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln424] Error happened on checking configurations. at /usr/local/share/perl5/MHA/MasterMonitor.pm line 326 Sun Mar 11 14:35:38 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln523] Error happened on monitoring servers. Sun Mar 11 14:35:38 2018 - [info] Got exit code 1 (Not master dead). tail: /var/log/masterha/app1/manager.log: file truncated Sun Mar 11 14:37:49 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Mar 11 14:37:49 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Sun Mar 11 14:37:49 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Sun Mar 11 14:37:49 2018 - [info] MHA::MasterMonitor version 0.56. Sun Mar 11 14:37:49 2018 - [info] Multi-master configuration is detected. Current primary(writable) master is 10.0.0.50(10.0.0.50:3306) Sun Mar 11 14:37:49 2018 - [info] Master configurations are as below: Master 10.0.0.50(10.0.0.50:3306) Master 10.0.0.60(10.0.0.60:3306), replicating from 10.0.0.50(10.0.0.50:3306), read-only Sun Mar 11 14:37:49 2018 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln726] Slave 10.0.0.70(10.0.0.70:3306) replicates from 10.0.0.60:3306, but real master is 10.0.0.50(10.0.0.50:3306)! Sun Mar 11 14:37:49 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln424] Error happened on checking configurations. at /usr/local/share/perl5/MHA/MasterMonitor.pm line 326 Sun Mar 11 14:37:49 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln523] Error happened on monitoring servers. Sun Mar 11 14:37:49 2018 - [info] Got exit code 1 (Not master dead). tail: /var/log/masterha/app1/manager.log: file truncated Sun Mar 11 14:38:57 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Mar 11 14:38:57 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Sun Mar 11 14:38:57 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Sun Mar 11 14:38:57 2018 - [info] MHA::MasterMonitor version 0.56. Sun Mar 11 14:38:57 2018 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln781] Multi-master configuration is detected, but two or more masters are either writable (read-only is not set) or dead! Check configurations for details. Master configurations are as below: Master 10.0.0.50(10.0.0.50:3306) Master 10.0.0.60(10.0.0.60:3306), replicating from 10.0.0.50(10.0.0.50:3306) Sun Mar 11 14:38:57 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln424] Error happened on checking configurations. at /usr/local/share/perl5/MHA/MasterMonitor.pm line 326 Sun Mar 11 14:38:57 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln523] Error happened on monitoring servers. Sun Mar 11 14:38:57 2018 - [info] Got exit code 1 (Not master dead). tail: /var/log/masterha/app1/manager.log: file truncated Sun Mar 11 14:41:47 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Mar 11 14:41:47 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Sun Mar 11 14:41:47 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Sun Mar 11 14:41:47 2018 - [info] MHA::MasterMonitor version 0.56. Sun Mar 11 14:41:47 2018 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln781] Multi-master configuration is detected, but two or more masters are either writable (read-only is not set) or dead! Check configurations for details. Master configurations are as below: Master 10.0.0.50(10.0.0.50:3306) Master 10.0.0.60(10.0.0.60:3306), replicating from 10.0.0.50(10.0.0.50:3306) Sun Mar 11 14:41:47 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln424] Error happened on checking configurations. at /usr/local/share/perl5/MHA/MasterMonitor.pm line 326 Sun Mar 11 14:41:47 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln523] Error happened on monitoring servers. Sun Mar 11 14:41:47 2018 - [info] Got exit code 1 (Not master dead). tail: /var/log/masterha/app1/manager.log: file truncated Sun Mar 11 14:44:29 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Mar 11 14:44:29 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Sun Mar 11 14:44:29 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Sun Mar 11 14:44:29 2018 - [info] MHA::MasterMonitor version 0.56. Sun Mar 11 14:44:29 2018 - [info] Multi-master configuration is detected. Current primary(writable) master is 10.0.0.50(10.0.0.50:3306) Sun Mar 11 14:44:29 2018 - [info] Master configurations are as below: Master 10.0.0.50(10.0.0.50:3306) Master 10.0.0.60(10.0.0.60:3306), replicating from 10.0.0.50(10.0.0.50:3306), read-only Sun Mar 11 14:44:29 2018 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln726] Slave 10.0.0.70(10.0.0.70:3306) replicates from 10.0.0.60:3306, but real master is 10.0.0.50(10.0.0.50:3306)! Sun Mar 11 14:44:29 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln424] Error happened on checking configurations. at /usr/local/share/perl5/MHA/MasterMonitor.pm line 326 Sun Mar 11 14:44:29 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln523] Error happened on monitoring servers. Sun Mar 11 14:44:29 2018 - [info] Got exit code 1 (Not master dead). tail: /var/log/masterha/app1/manager.log: file truncated Sun Mar 11 15:02:01 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Mar 11 15:02:01 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Sun Mar 11 15:02:01 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Sun Mar 11 15:02:01 2018 - [info] MHA::MasterMonitor version 0.56. Sun Mar 11 15:02:01 2018 - [info] GTID failover mode = 0 Sun Mar 11 15:02:01 2018 - [info] Dead Servers: Sun Mar 11 15:02:01 2018 - [info] Alive Servers: Sun Mar 11 15:02:01 2018 - [info] 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:02:01 2018 - [info] 10.0.0.60(10.0.0.60:3306) Sun Mar 11 15:02:01 2018 - [info] 10.0.0.70(10.0.0.70:3306) Sun Mar 11 15:02:01 2018 - [info] Alive Slaves: Sun Mar 11 15:02:01 2018 - [info] 10.0.0.60(10.0.0.60:3306) Version=5.6.16-log (oldest major version between slaves) log-bin:enabled Sun Mar 11 15:02:01 2018 - [info] Replicating from 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:02:01 2018 - [info] Primary candidate for the new Master (candidate_master is set) Sun Mar 11 15:02:01 2018 - [info] 10.0.0.70(10.0.0.70:3306) Version=5.6.16 (oldest major version between slaves) log-bin:disabled Sun Mar 11 15:02:01 2018 - [info] Replicating from 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:02:01 2018 - [info] Current Alive Master: 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:02:01 2018 - [info] Checking slave configurations.. Sun Mar 11 15:02:01 2018 - [warning] relay_log_purge=0 is not set on slave 10.0.0.60(10.0.0.60:3306). Sun Mar 11 15:02:01 2018 - [warning] relay_log_purge=0 is not set on slave 10.0.0.70(10.0.0.70:3306). Sun Mar 11 15:02:01 2018 - [warning] log-bin is not set on slave 10.0.0.70(10.0.0.70:3306). This host cannot be a master. Sun Mar 11 15:02:01 2018 - [info] Checking replication filtering settings.. Sun Mar 11 15:02:01 2018 - [info] binlog_do_db= , binlog_ignore_db= Sun Mar 11 15:02:01 2018 - [info] Replication filtering check ok. Sun Mar 11 15:02:01 2018 - [info] GTID (with auto-pos) is not supported Sun Mar 11 15:02:01 2018 - [info] Starting SSH connection tests.. Sun Mar 11 15:02:02 2018 - [info] All SSH connection tests passed successfully. Sun Mar 11 15:02:02 2018 - [info] Checking MHA Node version.. Sun Mar 11 15:02:03 2018 - [info] Version check ok. Sun Mar 11 15:02:03 2018 - [info] Checking SSH publickey authentication settings on the current master.. Sun Mar 11 15:02:04 2018 - [info] HealthCheck: SSH to 10.0.0.50 is reachable. Sun Mar 11 15:02:04 2018 - [info] Master MHA Node version is 0.56. Sun Mar 11 15:02:04 2018 - [info] Checking recovery script configurations on 10.0.0.50(10.0.0.50:3306).. Sun Mar 11 15:02:04 2018 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/mysql/data --output_file=/tmp/save_binary_logs_test --manager_version=0.56 --start_file=mysql-bin.000002 Sun Mar 11 15:02:04 2018 - [info] Connecting to root@10.0.0.50(10.0.0.50:22).. Creating /tmp if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /mysql/data, up to mysql-bin.000002 Sun Mar 11 15:02:04 2018 - [info] Binlog setting check done. Sun Mar 11 15:02:04 2018 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers.. Sun Mar 11 15:02:04 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=10.0.0.60 --slave_ip=10.0.0.60 --slave_port=3306 --workdir=/tmp --target_version=5.6.16-log --manager_version=0.56 --relay_log_info=/mysql/data/relay-log.info --relay_dir=/mysql/data/ --slave_pass=xxx Sun Mar 11 15:02:04 2018 - [info] Connecting to root@10.0.0.60(10.0.0.60:22).. Checking slave recovery environment settings.. Opening /mysql/data/relay-log.info ... ok. Relay log found at /mysql/data, up to cadicate-master-relay-bin.000005 Temporary relay log file is /mysql/data/cadicate-master-relay-bin.000005 Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sun Mar 11 15:02:05 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=10.0.0.70 --slave_ip=10.0.0.70 --slave_port=3306 --workdir=/tmp --target_version=5.6.16 --manager_version=0.56 --relay_log_info=/mysql/data/relay-log.info --relay_dir=/mysql/data/ --slave_pass=xxx Sun Mar 11 15:02:05 2018 - [info] Connecting to root@10.0.0.70(10.0.0.70:22).. Checking slave recovery environment settings.. Opening /mysql/data/relay-log.info ... ok. Relay log found at /mysql/data, up to slave-relay-bin.000002 Temporary relay log file is /mysql/data/slave-relay-bin.000002 Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sun Mar 11 15:02:05 2018 - [info] Slaves settings check done. Sun Mar 11 15:02:05 2018 - [info] 10.0.0.50(10.0.0.50:3306) (current master) +--10.0.0.60(10.0.0.60:3306) +--10.0.0.70(10.0.0.70:3306) Sun Mar 11 15:02:05 2018 - [info] Checking master_ip_failover_script status: Sun Mar 11 15:02:05 2018 - [info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=10.0.0.50 --orig_master_ip=10.0.0.50 --orig_master_port=3306 IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start=== Checking the Status of the script.. OK Sun Mar 11 15:02:05 2018 - [info] OK. Sun Mar 11 15:02:05 2018 - [warning] shutdown_script is not defined. Sun Mar 11 15:02:05 2018 - [info] Set master ping interval 1 seconds. Sun Mar 11 15:02:05 2018 - [info] Set secondary check script: /usr/local/bin/masterha_secondary_check -s server03 -s server02 Sun Mar 11 15:02:05 2018 - [info] Starting ping health check on 10.0.0.50(10.0.0.50:3306).. Sun Mar 11 15:02:05 2018 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond.. ^C [root@manager ~]# > /var/log/masterha/app1/manager.log [root@manager ~]# tail -f /var/log/masterha/app1/manager.log Sun Mar 11 15:12:20 2018 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away) Sun Mar 11 15:12:20 2018 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/mysql/data --output_file=/tmp/save_binary_logs_test --manager_version=0.56 --binlog_prefix=mysql-bin Sun Mar 11 15:12:20 2018 - [info] Executing secondary network check script: /usr/local/bin/masterha_secondary_check -s server03 -s server02 --user=root --master_host=10.0.0.50 --master_ip=10.0.0.50 --master_port=3306 --master_user=root --master_password=123 --ping_type=SELECT Sun Mar 11 15:12:20 2018 - [info] HealthCheck: SSH to 10.0.0.50 is reachable. Monitoring server server03 is reachable, Master is not reachable from server03. OK. Monitoring server server02 is reachable, Master is not reachable from server02. OK. Sun Mar 11 15:12:20 2018 - [info] Master is not reachable from all other monitoring servers. Failover should start. Sun Mar 11 15:12:21 2018 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111) Sun Mar 11 15:12:21 2018 - [warning] Connection failed 2 time(s).. Sun Mar 11 15:12:22 2018 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111) Sun Mar 11 15:12:22 2018 - [warning] Connection failed 3 time(s).. Sun Mar 11 15:12:23 2018 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111) Sun Mar 11 15:12:23 2018 - [warning] Connection failed 4 time(s).. Sun Mar 11 15:12:23 2018 - [warning] Master is not reachable from health checker! Sun Mar 11 15:12:23 2018 - [warning] Master 10.0.0.50(10.0.0.50:3306) is not reachable! Sun Mar 11 15:12:23 2018 - [warning] SSH is reachable. Sun Mar 11 15:12:23 2018 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/masterha/app1.cnf again, and trying to connect to all servers to check server status.. Sun Mar 11 15:12:23 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Mar 11 15:12:23 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Sun Mar 11 15:12:23 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Sun Mar 11 15:12:23 2018 - [info] GTID failover mode = 0 Sun Mar 11 15:12:23 2018 - [info] Dead Servers: Sun Mar 11 15:12:23 2018 - [info] 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:12:23 2018 - [info] Alive Servers: Sun Mar 11 15:12:23 2018 - [info] 10.0.0.60(10.0.0.60:3306) Sun Mar 11 15:12:23 2018 - [info] 10.0.0.70(10.0.0.70:3306) Sun Mar 11 15:12:23 2018 - [info] Alive Slaves: Sun Mar 11 15:12:23 2018 - [info] 10.0.0.60(10.0.0.60:3306) Version=5.6.16-log (oldest major version between slaves) log-bin:enabled Sun Mar 11 15:12:23 2018 - [info] Replicating from 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:12:23 2018 - [info] Primary candidate for the new Master (candidate_master is set) Sun Mar 11 15:12:23 2018 - [info] 10.0.0.70(10.0.0.70:3306) Version=5.6.16 (oldest major version between slaves) log-bin:disabled Sun Mar 11 15:12:23 2018 - [info] Replicating from 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:12:23 2018 - [info] Checking slave configurations.. Sun Mar 11 15:12:23 2018 - [warning] relay_log_purge=0 is not set on slave 10.0.0.60(10.0.0.60:3306). Sun Mar 11 15:12:23 2018 - [warning] relay_log_purge=0 is not set on slave 10.0.0.70(10.0.0.70:3306). Sun Mar 11 15:12:23 2018 - [warning] log-bin is not set on slave 10.0.0.70(10.0.0.70:3306). This host cannot be a master. Sun Mar 11 15:12:23 2018 - [info] Checking replication filtering settings.. Sun Mar 11 15:12:23 2018 - [info] Replication filtering check ok. Sun Mar 11 15:12:23 2018 - [info] Master is down! Sun Mar 11 15:12:23 2018 - [info] Terminating monitoring script. Sun Mar 11 15:12:23 2018 - [info] Got exit code 20 (Master dead). Sun Mar 11 15:12:23 2018 - [info] MHA::MasterFailover version 0.56. Sun Mar 11 15:12:23 2018 - [info] Starting master failover. Sun Mar 11 15:12:23 2018 - [info] Sun Mar 11 15:12:23 2018 - [info] * Phase 1: Configuration Check Phase.. Sun Mar 11 15:12:23 2018 - [info] Sun Mar 11 15:12:23 2018 - [info] GTID failover mode = 0 Sun Mar 11 15:12:23 2018 - [info] Dead Servers: Sun Mar 11 15:12:23 2018 - [info] 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:12:23 2018 - [info] Checking master reachability via MySQL(double check)... Sun Mar 11 15:12:23 2018 - [info] ok. Sun Mar 11 15:12:23 2018 - [info] Alive Servers: Sun Mar 11 15:12:23 2018 - [info] 10.0.0.60(10.0.0.60:3306) Sun Mar 11 15:12:23 2018 - [info] 10.0.0.70(10.0.0.70:3306) Sun Mar 11 15:12:23 2018 - [info] Alive Slaves: Sun Mar 11 15:12:23 2018 - [info] 10.0.0.60(10.0.0.60:3306) Version=5.6.16-log (oldest major version between slaves) log-bin:enabled Sun Mar 11 15:12:23 2018 - [info] Replicating from 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:12:23 2018 - [info] Primary candidate for the new Master (candidate_master is set) Sun Mar 11 15:12:23 2018 - [info] 10.0.0.70(10.0.0.70:3306) Version=5.6.16 (oldest major version between slaves) log-bin:disabled Sun Mar 11 15:12:23 2018 - [info] Replicating from 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:12:23 2018 - [info] Starting Non-GTID based failover. Sun Mar 11 15:12:23 2018 - [info] Sun Mar 11 15:12:23 2018 - [info] ** Phase 1: Configuration Check Phase completed. Sun Mar 11 15:12:23 2018 - [info] Sun Mar 11 15:12:23 2018 - [info] * Phase 2: Dead Master Shutdown Phase.. Sun Mar 11 15:12:23 2018 - [info] Sun Mar 11 15:12:23 2018 - [info] Forcing shutdown so that applications never connect to the current master.. Sun Mar 11 15:12:23 2018 - [info] Executing master IP deactivation script: Sun Mar 11 15:12:23 2018 - [info] /usr/local/bin/master_ip_failover --orig_master_host=10.0.0.50 --orig_master_ip=10.0.0.50 --orig_master_port=3306 --command=stopssh --ssh_user=root IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start=== Disabling the VIP on old master: 10.0.0.50 Sun Mar 11 15:12:23 2018 - [info] done. Sun Mar 11 15:12:23 2018 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master. Sun Mar 11 15:12:23 2018 - [info] * Phase 2: Dead Master Shutdown Phase completed. Sun Mar 11 15:12:23 2018 - [info] Sun Mar 11 15:12:23 2018 - [info] * Phase 3: Master Recovery Phase.. Sun Mar 11 15:12:23 2018 - [info] Sun Mar 11 15:12:23 2018 - [info] * Phase 3.1: Getting Latest Slaves Phase.. Sun Mar 11 15:12:23 2018 - [info] Sun Mar 11 15:12:23 2018 - [info] The latest binary log file/position on all slaves is mysql-bin.000002:120 Sun Mar 11 15:12:23 2018 - [info] Latest slaves (Slaves that received relay log files to the latest): Sun Mar 11 15:12:23 2018 - [info] 10.0.0.60(10.0.0.60:3306) Version=5.6.16-log (oldest major version between slaves) log-bin:enabled Sun Mar 11 15:12:23 2018 - [info] Replicating from 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:12:23 2018 - [info] Primary candidate for the new Master (candidate_master is set) Sun Mar 11 15:12:23 2018 - [info] 10.0.0.70(10.0.0.70:3306) Version=5.6.16 (oldest major version between slaves) log-bin:disabled Sun Mar 11 15:12:23 2018 - [info] Replicating from 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:12:23 2018 - [info] The oldest binary log file/position on all slaves is mysql-bin.000002:120 Sun Mar 11 15:12:23 2018 - [info] Oldest slaves: Sun Mar 11 15:12:23 2018 - [info] 10.0.0.60(10.0.0.60:3306) Version=5.6.16-log (oldest major version between slaves) log-bin:enabled Sun Mar 11 15:12:23 2018 - [info] Replicating from 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:12:23 2018 - [info] Primary candidate for the new Master (candidate_master is set) Sun Mar 11 15:12:23 2018 - [info] 10.0.0.70(10.0.0.70:3306) Version=5.6.16 (oldest major version between slaves) log-bin:disabled Sun Mar 11 15:12:23 2018 - [info] Replicating from 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:12:23 2018 - [info] Sun Mar 11 15:12:23 2018 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase.. Sun Mar 11 15:12:23 2018 - [info] Sun Mar 11 15:12:23 2018 - [info] Fetching dead master's binary logs.. Sun Mar 11 15:12:23 2018 - [info] Executing command on the dead master 10.0.0.50(10.0.0.50:3306): save_binary_logs --command=save --start_file=mysql-bin.000002 --start_pos=120 --binlog_dir=/mysql/data --output_file=/tmp/saved_master_binlog_from_10.0.0.50_3306_20180311151223.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56 Creating /tmp if not exists.. ok. Concat binary/relay logs from mysql-bin.000002 pos 120 to mysql-bin.000002 EOF into /tmp/saved_master_binlog_from_10.0.0.50_3306_20180311151223.binlog .. Binlog Checksum enabled Dumping binlog format description event, from position 0 to 120.. ok. No need to dump effective binlog data from /mysql/data/mysql-bin.000002 (pos starts 120, filesize 120). Skipping. Binlog Checksum enabled /tmp/saved_master_binlog_from_10.0.0.50_3306_20180311151223.binlog has no effective data events. Event not exists. Sun Mar 11 15:12:24 2018 - [info] Additional events were not found from the orig master. No need to save. Sun Mar 11 15:12:24 2018 - [info] Sun Mar 11 15:12:24 2018 - [info] * Phase 3.3: Determining New Master Phase.. Sun Mar 11 15:12:24 2018 - [info] Sun Mar 11 15:12:24 2018 - [info] Finding the latest slave that has all relay logs for recovering other slaves.. Sun Mar 11 15:12:24 2018 - [info] All slaves received relay logs to the same position. No need to resync each other. Sun Mar 11 15:12:24 2018 - [info] Searching new master from slaves.. Sun Mar 11 15:12:24 2018 - [info] Candidate masters from the configuration file: Sun Mar 11 15:12:24 2018 - [info] 10.0.0.60(10.0.0.60:3306) Version=5.6.16-log (oldest major version between slaves) log-bin:enabled Sun Mar 11 15:12:24 2018 - [info] Replicating from 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:12:24 2018 - [info] Primary candidate for the new Master (candidate_master is set) Sun Mar 11 15:12:24 2018 - [info] Non-candidate masters: Sun Mar 11 15:12:24 2018 - [info] 10.0.0.70(10.0.0.70:3306) Version=5.6.16 (oldest major version between slaves) log-bin:disabled Sun Mar 11 15:12:24 2018 - [info] Replicating from 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:12:24 2018 - [info] Searching from candidate_master slaves which have received the latest relay log events.. Sun Mar 11 15:12:24 2018 - [info] New master is 10.0.0.60(10.0.0.60:3306) Sun Mar 11 15:12:24 2018 - [info] Starting master failover.. Sun Mar 11 15:12:24 2018 - [info] From: 10.0.0.50(10.0.0.50:3306) (current master) +--10.0.0.60(10.0.0.60:3306) +--10.0.0.70(10.0.0.70:3306) To: 10.0.0.60(10.0.0.60:3306) (new master) +--10.0.0.70(10.0.0.70:3306) Sun Mar 11 15:12:24 2018 - [info] Sun Mar 11 15:12:24 2018 - [info] * Phase 3.3: New Master Diff Log Generation Phase.. Sun Mar 11 15:12:24 2018 - [info] Sun Mar 11 15:12:24 2018 - [info] This server has all relay logs. No need to generate diff files from the latest slave. Sun Mar 11 15:12:24 2018 - [info] Sun Mar 11 15:12:24 2018 - [info] * Phase 3.4: Master Log Apply Phase.. Sun Mar 11 15:12:24 2018 - [info] Sun Mar 11 15:12:24 2018 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed. Sun Mar 11 15:12:24 2018 - [info] Starting recovery on 10.0.0.60(10.0.0.60:3306).. Sun Mar 11 15:12:24 2018 - [info] This server has all relay logs. Waiting all logs to be applied.. Sun Mar 11 15:12:24 2018 - [info] done. Sun Mar 11 15:12:24 2018 - [info] All relay logs were successfully applied. Sun Mar 11 15:12:24 2018 - [info] Getting new master's binlog name and position.. Sun Mar 11 15:12:24 2018 - [info] mysql-bin.000002:120 Sun Mar 11 15:12:24 2018 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.0.0.60', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000002', MASTER_LOG_POS=120, MASTER_USER='repl', MASTER_PASSWORD='xxx'; Sun Mar 11 15:12:24 2018 - [info] Executing master IP activate script: Sun Mar 11 15:12:24 2018 - [info] /usr/local/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=10.0.0.50 --orig_master_ip=10.0.0.50 --orig_master_port=3306 --new_master_host=10.0.0.60 --new_master_ip=10.0.0.60 --new_master_port=3306 --new_master_user='root' --new_master_password='123' Unknown option: new_master_user Unknown option: new_master_password IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start=== Enabling the VIP - 10.0.0.88 on the new master - 10.0.0.60 Sun Mar 11 15:12:24 2018 - [info] OK. Sun Mar 11 15:12:24 2018 - [info] Setting read_only=0 on 10.0.0.60(10.0.0.60:3306).. Sun Mar 11 15:12:24 2018 - [info] ok. Sun Mar 11 15:12:24 2018 - [info] ** Finished master recovery successfully. Sun Mar 11 15:12:24 2018 - [info] * Phase 3: Master Recovery Phase completed. Sun Mar 11 15:12:24 2018 - [info] Sun Mar 11 15:12:24 2018 - [info] * Phase 4: Slaves Recovery Phase.. Sun Mar 11 15:12:24 2018 - [info] Sun Mar 11 15:12:24 2018 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase.. Sun Mar 11 15:12:24 2018 - [info] Sun Mar 11 15:12:24 2018 - [info] -- Slave diff file generation on host 10.0.0.70(10.0.0.70:3306) started, pid: 2291. Check tmp log /var/log/masterha/app1.log/10.0.0.70_3306_20180311151223.log if it takes time.. Sun Mar 11 15:12:24 2018 - [info] Sun Mar 11 15:12:24 2018 - [info] Log messages from 10.0.0.70 ... Sun Mar 11 15:12:24 2018 - [info] Sun Mar 11 15:12:24 2018 - [info] This server has all relay logs. No need to generate diff files from the latest slave. Sun Mar 11 15:12:24 2018 - [info] End of log messages from 10.0.0.70. Sun Mar 11 15:12:24 2018 - [info] -- 10.0.0.70(10.0.0.70:3306) has the latest relay log events. Sun Mar 11 15:12:24 2018 - [info] Generating relay diff files from the latest slave succeeded. Sun Mar 11 15:12:24 2018 - [info] Sun Mar 11 15:12:24 2018 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase.. Sun Mar 11 15:12:24 2018 - [info] Sun Mar 11 15:12:24 2018 - [info] -- Slave recovery on host 10.0.0.70(10.0.0.70:3306) started, pid: 2293. Check tmp log /var/log/masterha/app1.log/10.0.0.70_3306_20180311151223.log if it takes time.. Sun Mar 11 15:12:24 2018 - [info] Sun Mar 11 15:12:24 2018 - [info] Log messages from 10.0.0.70 ... Sun Mar 11 15:12:24 2018 - [info] Sun Mar 11 15:12:24 2018 - [info] Starting recovery on 10.0.0.70(10.0.0.70:3306).. Sun Mar 11 15:12:24 2018 - [info] This server has all relay logs. Waiting all logs to be applied.. Sun Mar 11 15:12:24 2018 - [info] done. Sun Mar 11 15:12:24 2018 - [info] All relay logs were successfully applied. Sun Mar 11 15:12:24 2018 - [info] Resetting slave 10.0.0.70(10.0.0.70:3306) and starting replication from the new master 10.0.0.60(10.0.0.60:3306).. Sun Mar 11 15:12:24 2018 - [info] Executed CHANGE MASTER. Sun Mar 11 15:12:24 2018 - [info] Slave started. Sun Mar 11 15:12:24 2018 - [info] End of log messages from 10.0.0.70. Sun Mar 11 15:12:24 2018 - [info] -- Slave recovery on host 10.0.0.70(10.0.0.70:3306) succeeded. Sun Mar 11 15:12:24 2018 - [info] All new slave servers recovered successfully. Sun Mar 11 15:12:24 2018 - [info] Sun Mar 11 15:12:24 2018 - [info] * Phase 5: New master cleanup phase.. Sun Mar 11 15:12:24 2018 - [info] Sun Mar 11 15:12:24 2018 - [info] Resetting slave info on the new master.. Sun Mar 11 15:12:24 2018 - [info] 10.0.0.60: Resetting slave info succeeded. Sun Mar 11 15:12:24 2018 - [info] Master failover to 10.0.0.60(10.0.0.60:3306) completed successfully. Sun Mar 11 15:12:24 2018 - [info] Deleted server1 entry from /etc/masterha/app1.cnf . Sun Mar 11 15:12:24 2018 - [info] ----- Failover Report ----- app1: MySQL Master failover 10.0.0.50(10.0.0.50:3306) to 10.0.0.60(10.0.0.60:3306) succeeded Master 10.0.0.50(10.0.0.50:3306) is down! Check MHA Manager logs at manager:/var/log/masterha/app1/manager.log for details. Started automated(non-interactive) failover. Invalidated master IP address on 10.0.0.50(10.0.0.50:3306) The latest slave 10.0.0.60(10.0.0.60:3306) has all relay logs for recovery. Selected 10.0.0.60(10.0.0.60:3306) as a new master. 10.0.0.60(10.0.0.60:3306): OK: Applying all logs succeeded. 10.0.0.60(10.0.0.60:3306): OK: Activated master IP address. 10.0.0.70(10.0.0.70:3306): This host has the latest relay log events. Generating relay diff files from the latest slave succeeded. 10.0.0.70(10.0.0.70:3306): OK: Applying all logs succeeded. Slave started, replicating from 10.0.0.60(10.0.0.60:3306) 10.0.0.60(10.0.0.60:3306): Resetting slave info succeeded. Master failover to 10.0.0.60(10.0.0.60:3306) completed successfully.
successfully
而有故障的主機會被剔出集羣。
查看 cadicate上的vip是否由master 飄過來
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:39:2d:82 brd ff:ff:ff:ff:ff:ff inet 10.0.0.60/8 brd 10.255.255.255 scope global eth0 inet 10.0.0.88/32 scope global eth0
從上面的輸出能夠看出整個MHA的切換過程，共包括如下的步驟：
1.配置文件檢查階段，這個階段會檢查整個集羣配置文件配置
2.宕機的master處理，這個階段包括虛擬ip摘除操做，主機關機操做（這個我這裏尚未實現，須要研究）
3.複製dead maste和最新slave相差的relay log，並保存到MHA Manger具體的目錄下
4.識別含有最新更新的slave
5.應用從master保存的二進制日誌事件（binlog events）
6.提高一個slave爲新的master進行復制
7.使其餘的slave鏈接新的master進行復制
最後啓動MHA Manger監控，查看集羣裏面如今誰是master（在切換後監控就中止了。。。還有東西沒搞對？）後來在官方網站看到這句話就明白了 。
上述模擬了master宕機的狀況下手動把10.0.0.60提高爲主庫的操做過程。
Running MHA Manager from daemontools
Currently MHA Manager process does not run as a daemon. If failover completed successfully or the master process was killed by accident, the manager stops working. To run as a daemon, daemontool. or any external daemon program can be used. Here is an example to run from daemontools.
二.手動Failover（MHA Manager必須沒有運行）
手動failover，這種場景意味着在業務上沒有啓用MHA自動切換功能，當主服務器故障時，人工手動調用MHA來進行故障切換操做，具體命令以下：
注意：若是，MHA manager檢測到沒有dead的server，將報錯，並結束failover：
進行手動切換命令以下：
[root@manager ~]# masterha_master_switch --master_state=dead --conf=/etc/masterha/app1.cnf --dead_master_host=10.0.0.50 --dead_master_port=3306 --new_master_host=10.0.0.60 --new_master_port=3306 --ignore_last_failover
輸出的信息會詢問你是否進行切換：
上述模擬了master宕機的狀況下手動把10.0.0.60提高爲主庫的操做過程。
三.在線進行切換
在許多狀況下， 須要將現有的主服務器遷移到另一臺服務器上。 好比主服務器硬件故障，RAID 控制卡須要重建，將主服務器移到性能更好的服務器上等等。維護主服務器引發性能降低， 致使停機時間至少沒法寫入數據。 另外， 阻塞或殺掉當前運行的會話會致使主主之間數據不一致的問題發生。 MHA 提供快速切換和優雅的阻塞寫入，這個切換過程只須要 0.5-2s 的時間，這段時間內數據是沒法寫入的。在不少狀況下，0.5-2s 的阻塞寫入是能夠接受的。所以切換主服務器不須要計劃分配維護時間窗口。
MHA在線切換的大概過程：
1.檢測複製設置和肯定當前主服務器
2.肯定新的主服務器
3.阻塞寫入到當前主服務器
4.等待全部從服務器遇上覆制
5.授予寫入到新的主服務器
6.從新設置從服務器
注意，在線切換的時候應用架構須要考慮如下兩個問題：
1.自動識別master和slave的問題（master的機器可能會切換），若是採用了vip的方式，基本能夠解決這個問題。
2.負載均衡的問題（能夠定義大概的讀寫比例，每臺機器可承擔的負載比例，當有機器離開集羣時，須要考慮這個問題）
爲了保證數據徹底一致性，在最快的時間內完成切換，MHA的在線切換必須知足如下條件纔會切換成功，不然會切換失敗。
1.全部slave的IO線程都在運行
2.全部slave的SQL線程都在運行
3.全部的show slave status的輸出中Seconds_Behind_Master參數小於或者等於running_updates_limit秒，若是在切換過程當中不指定running_updates_limit,那麼默認狀況下running_updates_limit爲1秒。
4.在master端，經過show processlist輸出，沒有一個更新花費的時間大於running_updates_limit秒。
在線切換步驟以下：
首先，停掉MHA監控：
[root@manager ~]# masterha_stop --conf=/etc/masterha/app1.cnf
其次，進行在線切換操做（模擬在線切換主庫操做，原主庫10.0.0.50變爲slave，10.0.0.60提高爲新的主庫）
[root@manager ~]# masterha_master_switch --conf=/etc/masterha/app1.cnf --master_state=alive --new_master_host=10.0.0.60 --new_master_port=3306 --orig_master_is_new_slave --running_updates_limit=10000
有個交互問你是否要切換
[root@manager ~]# masterha_master_switch --conf=/etc/masterha/app1.cnf --master_state=alive --new_master_host=10.0.0.60 --new_master_port=3306 --orig_master_is_new_slave --running_updates_limit=10000 Sun Mar 11 15:50:13 2018 - [info] MHA::MasterRotate version 0.56. Sun Mar 11 15:50:13 2018 - [info] Starting online master switch.. Sun Mar 11 15:50:13 2018 - [info] Sun Mar 11 15:50:13 2018 - [info] * Phase 1: Configuration Check Phase.. Sun Mar 11 15:50:13 2018 - [info] Sun Mar 11 15:50:13 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Mar 11 15:50:13 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Sun Mar 11 15:50:13 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Sun Mar 11 15:50:13 2018 - [info] GTID failover mode = 0 Sun Mar 11 15:50:13 2018 - [info] Current Alive Master: 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:50:13 2018 - [info] Alive Slaves: Sun Mar 11 15:50:13 2018 - [info] 10.0.0.60(10.0.0.60:3306) Version=5.6.16-log (oldest major version between slaves) log-bin:enabled Sun Mar 11 15:50:13 2018 - [info] Replicating from 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:50:13 2018 - [info] Primary candidate for the new Master (candidate_master is set) Sun Mar 11 15:50:13 2018 - [info] 10.0.0.70(10.0.0.70:3306) Version=5.6.16 (oldest major version between slaves) log-bin:disabled Sun Mar 11 15:50:13 2018 - [info] Replicating from 10.0.0.50(10.0.0.50:3306)
It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 10.0.0.50(10.0.0.50:3306)? (YES/no): yes
Sun Mar 11 15:51:11 2018 - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..
Sun Mar 11 15:51:11 2018 - [info] ok.
Sun Mar 11 15:51:11 2018 - [info] Checking MHA is not monitoring or doing failover..
Sun Mar 11 15:51:11 2018 - [info] Checking replication health on 10.0.0.60..
Sun Mar 11 15:51:11 2018 - [info] ok.
Sun Mar 11 15:51:11 2018 - [info] Checking replication health on 10.0.0.70..
Sun Mar 11 15:51:11 2018 - [info] ok.
Sun Mar 11 15:51:11 2018 - [info] 10.0.0.60 can be new master.
Sun Mar 11 15:51:11 2018 - [info]
From:
10.0.0.50(10.0.0.50:3306) (current master)
+--10.0.0.60(10.0.0.60:3306)
+--10.0.0.70(10.0.0.70:3306)
To:
10.0.0.60(10.0.0.60:3306) (new master)
+--10.0.0.70(10.0.0.70:3306)
+--10.0.0.50(10.0.0.50:3306)
Starting master switch from 10.0.0.50(10.0.0.50:3306) to 10.0.0.60(10.0.0.60:3306)? (yes/NO): ^CSun Mar 11 15:51:31 2018 - [info] Killing thread 9 on 10.0.0.50(10.0.0.50:3306)..
Sun Mar 11 15:51:31 2018 - [info] ok.
Sun Mar 11 15:51:31 2018 - [info] Killing thread 10 on 10.0.0.60(10.0.0.60:3306)..
Sun Mar 11 15:51:31 2018 - [info] ok.
Sun Mar 11 15:51:31 2018 - [info] Killing thread 10 on 10.0.0.70(10.0.0.70:3306)..
Sun Mar 11 15:51:31 2018 - [info] ok.
[root@manager ~]# masterha_master_switch --conf=/etc/masterha/app1.cnf --master_state=alive --new_master_host=10.0.0.60 --new_master_port=3306 --orig_master_is_new_slave --running_updates_limit=10000
Sun Mar 11 15:51:45 2018 - [info] MHA::MasterRotate version 0.56.
Sun Mar 11 15:51:45 2018 - [info] Starting online master switch..
Sun Mar 11 15:51:45 2018 - [info]
Sun Mar 11 15:51:45 2018 - [info] * Phase 1: Configuration Check Phase..
Sun Mar 11 15:51:45 2018 - [info]
Sun Mar 11 15:51:45 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sun Mar 11 15:51:45 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Sun Mar 11 15:51:45 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Sun Mar 11 15:51:45 2018 - [info] GTID failover mode = 0
Sun Mar 11 15:51:45 2018 - [info] Current Alive Master: 10.0.0.50(10.0.0.50:3306)
Sun Mar 11 15:51:45 2018 - [info] Alive Slaves:
Sun Mar 11 15:51:45 2018 - [info] 10.0.0.60(10.0.0.60:3306) Version=5.6.16-log (oldest major version between slaves) log-bin:enabled
Sun Mar 11 15:51:45 2018 - [info] Replicating from 10.0.0.50(10.0.0.50:3306)
Sun Mar 11 15:51:45 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Sun Mar 11 15:51:45 2018 - [info] 10.0.0.70(10.0.0.70:3306) Version=5.6.16 (oldest major version between slaves) log-bin:disabled
Sun Mar 11 15:51:45 2018 - [info] Replicating from 10.0.0.50(10.0.0.50:3306)
It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 10.0.0.50(10.0.0.50:3306)? (YES/no): yes
Sun Mar 11 15:51:50 2018 - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..
Sun Mar 11 15:51:50 2018 - [info] ok.
Sun Mar 11 15:51:50 2018 - [info] Checking MHA is not monitoring or doing failover..
Sun Mar 11 15:51:50 2018 - [info] Checking replication health on 10.0.0.60..
Sun Mar 11 15:51:50 2018 - [info] ok.
Sun Mar 11 15:51:50 2018 - [info] Checking replication health on 10.0.0.70..
Sun Mar 11 15:51:50 2018 - [info] ok.
Sun Mar 11 15:51:50 2018 - [info] 10.0.0.60 can be new master.
Sun Mar 11 15:51:50 2018 - [info]
From:
10.0.0.50(10.0.0.50:3306) (current master)
+--10.0.0.60(10.0.0.60:3306)
+--10.0.0.70(10.0.0.70:3306)
To:
10.0.0.60(10.0.0.60:3306) (new master)
+--10.0.0.70(10.0.0.70:3306)
+--10.0.0.50(10.0.0.50:3306)
Starting master switch from 10.0.0.50(10.0.0.50:3306) to 10.0.0.60(10.0.0.60:3306)? (yes/NO): yes
Sun Mar 11 15:52:04 2018 - [info] Checking whether 10.0.0.60(10.0.0.60:3306) is ok for the new master..
Sun Mar 11 15:52:04 2018 - [info] ok.
Sun Mar 11 15:52:04 2018 - [info] 10.0.0.50(10.0.0.50:3306): SHOW SLAVE STATUS returned empty result. To check replication filtering rules, temporarily executing CHANGE MASTER to a dummy host.
Sun Mar 11 15:52:04 2018 - [info] 10.0.0.50(10.0.0.50:3306): Resetting slave pointing to the dummy host.
Sun Mar 11 15:52:04 2018 - [info] ** Phase 1: Configuration Check Phase completed.
Sun Mar 11 15:52:04 2018 - [info]
Sun Mar 11 15:52:04 2018 - [info] * Phase 2: Rejecting updates Phase..
Sun Mar 11 15:52:04 2018 - [info]
Sun Mar 11 15:52:04 2018 - [info] Executing master ip online change script to disable write on the current master:
Sun Mar 11 15:52:04 2018 - [info] /usr/local/bin/master_ip_online_change --command=stop --orig_master_host=10.0.0.50 --orig_master_ip=10.0.0.50 --orig_master_port=3306 --orig_master_user='root' --orig_master_password='123' --new_master_host=10.0.0.60 --new_master_ip=10.0.0.60 --new_master_port=3306 --new_master_user='root' --new_master_password='123' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_is_new_slave
Unknown option: orig_master_password
Unknown option: new_master_password
Unknown option: orig_master_ssh_user
Unknown option: new_master_ssh_user
Unknown option: orig_master_is_new_slave
Sun Mar 11 15:52:05 2018 035158 Set read_only on the new master.. ok.
Sun Mar 11 15:52:05 2018 037347 Set read_only=1 on the orig master.. ok.
Disabling the VIP on old master: 10.0.0.50
eth1:1: unknown interface: No such device
Sun Mar 11 15:52:05 2018 194450 Killing all application threads..
Sun Mar 11 15:52:05 2018 194507 done.
Sun Mar 11 15:52:05 2018 - [info] ok.
Sun Mar 11 15:52:05 2018 - [info] Locking all tables on the orig master to reject updates from everybody (including root):
Sun Mar 11 15:52:05 2018 - [info] Executing FLUSH TABLES WITH READ LOCK..
Sun Mar 11 15:52:05 2018 - [info] ok.
Sun Mar 11 15:52:05 2018 - [info] Orig master binlog:pos is mysql-bin.000002:120.
Sun Mar 11 15:52:05 2018 - [info] Waiting to execute all relay logs on 10.0.0.60(10.0.0.60:3306)..
Sun Mar 11 15:52:05 2018 - [info] master_pos_wait(mysql-bin.000002:120) completed on 10.0.0.60(10.0.0.60:3306). Executed 0 events.
Sun Mar 11 15:52:05 2018 - [info] done.
Sun Mar 11 15:52:05 2018 - [info] Getting new master's binlog name and position..
Sun Mar 11 15:52:05 2018 - [info] mysql-bin.000003:120
Sun Mar 11 15:52:05 2018 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.0.0.60', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000003', MASTER_LOG_POS=120, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Sun Mar 11 15:52:05 2018 - [info] Executing master ip online change script to allow write on the new master:
Sun Mar 11 15:52:05 2018 - [info] /usr/local/bin/master_ip_online_change --command=start --orig_master_host=10.0.0.50 --orig_master_ip=10.0.0.50 --orig_master_port=3306 --orig_master_user='root' --orig_master_password='123' --new_master_host=10.0.0.60 --new_master_ip=10.0.0.60 --new_master_port=3306 --new_master_user='root' --new_master_password='123' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_is_new_slave
Unknown option: orig_master_password
Unknown option: new_master_password
Unknown option: orig_master_ssh_user
Unknown option: new_master_ssh_user
Unknown option: orig_master_is_new_slave
Sun Mar 11 15:52:05 2018 332345 Set read_only=0 on the new master.
Enabling the VIP - 10.0.0.88/24 on the new master - 10.0.0.60
SIOCSIFADDR: No such device
SIOCSIFNETMASK: No such device
SIOCGIFADDR: No such device
SIOCSIFBROADCAST: No such device
eth1:1: unknown interface: No such device
Sun Mar 11 15:52:05 2018 - [info] ok.
Sun Mar 11 15:52:05 2018 - [info]
Sun Mar 11 15:52:05 2018 - [info] * Switching slaves in parallel..
Sun Mar 11 15:52:05 2018 - [info]
Sun Mar 11 15:52:05 2018 - [info] -- Slave switch on host 10.0.0.70(10.0.0.70:3306) started, pid: 1679
Sun Mar 11 15:52:05 2018 - [info]
Sun Mar 11 15:52:06 2018 - [info] Log messages from 10.0.0.70 ...
Sun Mar 11 15:52:06 2018 - [info]
Sun Mar 11 15:52:05 2018 - [info] Waiting to execute all relay logs on 10.0.0.70(10.0.0.70:3306)..
Sun Mar 11 15:52:05 2018 - [info] master_pos_wait(mysql-bin.000002:120) completed on 10.0.0.70(10.0.0.70:3306). Executed 0 events.
Sun Mar 11 15:52:05 2018 - [info] done.
Sun Mar 11 15:52:05 2018 - [info] Resetting slave 10.0.0.70(10.0.0.70:3306) and starting replication from the new master 10.0.0.60(10.0.0.60:3306)..
Sun Mar 11 15:52:05 2018 - [info] Executed CHANGE MASTER.
Sun Mar 11 15:52:06 2018 - [info] Slave started.
Sun Mar 11 15:52:06 2018 - [info] End of log messages from 10.0.0.70 ...
Sun Mar 11 15:52:06 2018 - [info]
Sun Mar 11 15:52:06 2018 - [info] -- Slave switch on host 10.0.0.70(10.0.0.70:3306) succeeded.
Sun Mar 11 15:52:06 2018 - [info] Unlocking all tables on the orig master:
Sun Mar 11 15:52:06 2018 - [info] Executing UNLOCK TABLES..
Sun Mar 11 15:52:06 2018 - [info] ok.
Sun Mar 11 15:52:06 2018 - [info] Starting orig master as a new slave..
Sun Mar 11 15:52:06 2018 - [info] Resetting slave 10.0.0.50(10.0.0.50:3306) and starting replication from the new master 10.0.0.60(10.0.0.60:3306)..
Sun Mar 11 15:52:06 2018 - [info] Executed CHANGE MASTER.
Sun Mar 11 15:52:06 2018 - [info] Slave started.
Sun Mar 11 15:52:06 2018 - [info] All new slave servers switched successfully.
Sun Mar 11 15:52:06 2018 - [info]
Sun Mar 11 15:52:06 2018 - [info] * Phase 5: New master cleanup phase..
Sun Mar 11 15:52:06 2018 - [info]
Sun Mar 11 15:52:06 2018 - [info] 10.0.0.60: Resetting slave info succeeded.
Sun Mar 11 15:52:06 2018 - [info] Switching master to 10.0.0.60(10.0.0.60:3306) completed successfully.
最後查看日誌，瞭解切換過程，輸出信息以下：
Sun Mar 11 15:45:41 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Mar 11 15:45:41 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Sun Mar 11 15:45:41 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Sun Mar 11 15:45:41 2018 - [info] MHA::MasterMonitor version 0.56. Sun Mar 11 15:45:41 2018 - [warning] /var/log/masterha/app1.log/app1.master_status.health already exists. You might have killed manager with SIGKILL(-9), may run two or more monitoring process for the same application, or use the same working directory. Check for details, and consider setting --workdir separately. Sun Mar 11 15:45:42 2018 - [info] GTID failover mode = 0 Sun Mar 11 15:45:42 2018 - [info] Dead Servers: Sun Mar 11 15:45:42 2018 - [info] Alive Servers: Sun Mar 11 15:45:42 2018 - [info] 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:45:42 2018 - [info] 10.0.0.60(10.0.0.60:3306) Sun Mar 11 15:45:42 2018 - [info] 10.0.0.70(10.0.0.70:3306) Sun Mar 11 15:45:42 2018 - [info] Alive Slaves: Sun Mar 11 15:45:42 2018 - [info] 10.0.0.60(10.0.0.60:3306) Version=5.6.16-log (oldest major version between slaves) log-bin:enabled Sun Mar 11 15:45:42 2018 - [info] Replicating from 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:45:42 2018 - [info] Primary candidate for the new Master (candidate_master is set) Sun Mar 11 15:45:42 2018 - [info] 10.0.0.70(10.0.0.70:3306) Version=5.6.16 (oldest major version between slaves) log-bin:disabled Sun Mar 11 15:45:42 2018 - [info] Replicating from 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:45:42 2018 - [info] Current Alive Master: 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:45:42 2018 - [info] Checking slave configurations.. Sun Mar 11 15:45:42 2018 - [info] read_only=1 is not set on slave 10.0.0.60(10.0.0.60:3306). Sun Mar 11 15:45:42 2018 - [warning] relay_log_purge=0 is not set on slave 10.0.0.60(10.0.0.60:3306). Sun Mar 11 15:45:42 2018 - [warning] relay_log_purge=0 is not set on slave 10.0.0.70(10.0.0.70:3306). Sun Mar 11 15:45:42 2018 - [warning] log-bin is not set on slave 10.0.0.70(10.0.0.70:3306). This host cannot be a master. Sun Mar 11 15:45:42 2018 - [info] Checking replication filtering settings.. Sun Mar 11 15:45:42 2018 - [info] binlog_do_db= , binlog_ignore_db= Sun Mar 11 15:45:42 2018 - [info] Replication filtering check ok. Sun Mar 11 15:45:42 2018 - [info] GTID (with auto-pos) is not supported Sun Mar 11 15:45:42 2018 - [info] Starting SSH connection tests.. Sun Mar 11 15:45:43 2018 - [info] All SSH connection tests passed successfully. Sun Mar 11 15:45:43 2018 - [info] Checking MHA Node version.. Sun Mar 11 15:45:45 2018 - [info] Version check ok. Sun Mar 11 15:45:45 2018 - [info] Checking SSH publickey authentication settings on the current master.. Sun Mar 11 15:45:45 2018 - [info] HealthCheck: SSH to 10.0.0.50 is reachable. Sun Mar 11 15:45:45 2018 - [info] Master MHA Node version is 0.56. Sun Mar 11 15:45:45 2018 - [info] Checking recovery script configurations on 10.0.0.50(10.0.0.50:3306).. Sun Mar 11 15:45:45 2018 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/mysql/data --output_file=/tmp/save_binary_logs_test --manager_version=0.56 --start_file=mysql-bin.000002 Sun Mar 11 15:45:45 2018 - [info] Connecting to root@10.0.0.50(10.0.0.50:22).. Creating /tmp if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /mysql/data, up to mysql-bin.000002 Sun Mar 11 15:45:46 2018 - [info] Binlog setting check done. Sun Mar 11 15:45:46 2018 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers.. Sun Mar 11 15:45:46 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=10.0.0.60 --slave_ip=10.0.0.60 --slave_port=3306 --workdir=/tmp --target_version=5.6.16-log --manager_version=0.56 --relay_log_info=/mysql/data/relay-log.info --relay_dir=/mysql/data/ --slave_pass=xxx Sun Mar 11 15:45:46 2018 - [info] Connecting to root@10.0.0.60(10.0.0.60:22).. Checking slave recovery environment settings.. Opening /mysql/data/relay-log.info ... ok. Relay log found at /mysql/data, up to cadicate-master-relay-bin.000007 Temporary relay log file is /mysql/data/cadicate-master-relay-bin.000007 Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sun Mar 11 15:45:47 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=10.0.0.70 --slave_ip=10.0.0.70 --slave_port=3306 --workdir=/tmp --target_version=5.6.16 --manager_version=0.56 --relay_log_info=/mysql/data/relay-log.info --relay_dir=/mysql/data/ --slave_pass=xxx Sun Mar 11 15:45:47 2018 - [info] Connecting to root@10.0.0.70(10.0.0.70:22).. Checking slave recovery environment settings.. Opening /mysql/data/relay-log.info ... ok. Relay log found at /mysql/data, up to slave-relay-bin.000005 Temporary relay log file is /mysql/data/slave-relay-bin.000005 Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sun Mar 11 15:45:47 2018 - [info] Slaves settings check done. Sun Mar 11 15:45:47 2018 - [info] 10.0.0.50(10.0.0.50:3306) (current master) +--10.0.0.60(10.0.0.60:3306) +--10.0.0.70(10.0.0.70:3306) Sun Mar 11 15:45:47 2018 - [info] Checking master_ip_failover_script status: Sun Mar 11 15:45:47 2018 - [info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=10.0.0.50 --orig_master_ip=10.0.0.50 --orig_master_port=3306 IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start=== Checking the Status of the script.. OK Sun Mar 11 15:45:47 2018 - [info] OK. Sun Mar 11 15:45:47 2018 - [warning] shutdown_script is not defined. Sun Mar 11 15:45:47 2018 - [info] Set master ping interval 1 seconds. Sun Mar 11 15:45:47 2018 - [info] Set secondary check script: /usr/local/bin/masterha_secondary_check -s server03 -s server02 Sun Mar 11 15:45:47 2018 - [info] Starting ping health check on 10.0.0.50(10.0.0.50:3306).. Sun Mar 11 15:45:47 2018 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond.. Sun Mar 11 15:49:49 2018 - [info] Got terminate signal. Exit. ^C [root@manager ~]# > /var/log/masterha/app1/manager.log [root@manager ~]# tail -f /var/log/masterha/app1/manager.log Sun Mar 11 15:53:04 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Mar 11 15:53:04 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Sun Mar 11 15:53:04 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Sun Mar 11 15:53:04 2018 - [info] MHA::MasterMonitor version 0.56. Sun Mar 11 15:53:04 2018 - [info] GTID failover mode = 0 Sun Mar 11 15:53:04 2018 - [info] Dead Servers: Sun Mar 11 15:53:04 2018 - [info] Alive Servers: Sun Mar 11 15:53:04 2018 - [info] 10.0.0.50(10.0.0.50:3306) Sun Mar 11 15:53:04 2018 - [info] 10.0.0.60(10.0.0.60:3306) Sun Mar 11 15:53:04 2018 - [info] 10.0.0.70(10.0.0.70:3306) Sun Mar 11 15:53:04 2018 - [info] Alive Slaves: Sun Mar 11 15:53:04 2018 - [info] 10.0.0.50(10.0.0.50:3306) Version=5.6.16-log (oldest major version between slaves) log-bin:enabled Sun Mar 11 15:53:04 2018 - [info] Replicating from 10.0.0.60(10.0.0.60:3306) Sun Mar 11 15:53:04 2018 - [info] 10.0.0.70(10.0.0.70:3306) Version=5.6.16 (oldest major version between slaves) log-bin:disabled Sun Mar 11 15:53:04 2018 - [info] Replicating from 10.0.0.60(10.0.0.60:3306) Sun Mar 11 15:53:04 2018 - [info] Current Alive Master: 10.0.0.60(10.0.0.60:3306) Sun Mar 11 15:53:04 2018 - [info] Checking slave configurations.. Sun Mar 11 15:53:04 2018 - [warning] relay_log_purge=0 is not set on slave 10.0.0.70(10.0.0.70:3306). Sun Mar 11 15:53:04 2018 - [warning] log-bin is not set on slave 10.0.0.70(10.0.0.70:3306). This host cannot be a master. Sun Mar 11 15:53:04 2018 - [info] Checking replication filtering settings.. Sun Mar 11 15:53:04 2018 - [info] binlog_do_db= , binlog_ignore_db= Sun Mar 11 15:53:04 2018 - [info] Replication filtering check ok. Sun Mar 11 15:53:04 2018 - [info] GTID (with auto-pos) is not supported Sun Mar 11 15:53:04 2018 - [info] Starting SSH connection tests.. Sun Mar 11 15:53:06 2018 - [info] All SSH connection tests passed successfully. Sun Mar 11 15:53:06 2018 - [info] Checking MHA Node version.. Sun Mar 11 15:53:06 2018 - [info] Version check ok. Sun Mar 11 15:53:06 2018 - [info] Checking SSH publickey authentication settings on the current master.. Sun Mar 11 15:53:06 2018 - [info] HealthCheck: SSH to 10.0.0.60 is reachable. Sun Mar 11 15:53:06 2018 - [info] Master MHA Node version is 0.56. Sun Mar 11 15:53:06 2018 - [info] Checking recovery script configurations on 10.0.0.60(10.0.0.60:3306).. Sun Mar 11 15:53:06 2018 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/mysql/data --output_file=/tmp/save_binary_logs_test --manager_version=0.56 --start_file=mysql-bin.000003 Sun Mar 11 15:53:06 2018 - [info] Connecting to root@10.0.0.60(10.0.0.60:22).. Creating /tmp if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /mysql/data, up to mysql-bin.000003 Sun Mar 11 15:53:06 2018 - [info] Binlog setting check done. Sun Mar 11 15:53:06 2018 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers.. Sun Mar 11 15:53:06 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=10.0.0.50 --slave_ip=10.0.0.50 --slave_port=3306 --workdir=/tmp --target_version=5.6.16-log --manager_version=0.56 --relay_log_info=/mysql/data/relay-log.info --relay_dir=/mysql/data/ --slave_pass=xxx Sun Mar 11 15:53:06 2018 - [info] Connecting to root@10.0.0.50(10.0.0.50:22).. Checking slave recovery environment settings.. Opening /mysql/data/relay-log.info ... ok. Relay log found at /mysql/data, up to master-relay-bin.000002 Temporary relay log file is /mysql/data/master-relay-bin.000002 Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sun Mar 11 15:53:07 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=10.0.0.70 --slave_ip=10.0.0.70 --slave_port=3306 --workdir=/tmp --target_version=5.6.16 --manager_version=0.56 --relay_log_info=/mysql/data/relay-log.info --relay_dir=/mysql/data/ --slave_pass=xxx Sun Mar 11 15:53:07 2018 - [info] Connecting to root@10.0.0.70(10.0.0.70:22).. Checking slave recovery environment settings.. Opening /mysql/data/relay-log.info ... ok. Relay log found at /mysql/data, up to slave-relay-bin.000002 Temporary relay log file is /mysql/data/slave-relay-bin.000002 Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sun Mar 11 15:53:07 2018 - [info] Slaves settings check done. Sun Mar 11 15:53:07 2018 - [info] 10.0.0.60(10.0.0.60:3306) (current master) +--10.0.0.50(10.0.0.50:3306) +--10.0.0.70(10.0.0.70:3306) Sun Mar 11 15:53:07 2018 - [info] Checking master_ip_failover_script status: Sun Mar 11 15:53:07 2018 - [info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=10.0.0.60 --orig_master_ip=10.0.0.60 --orig_master_port=3306 IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start=== Checking the Status of the script.. OK Sun Mar 11 15:53:07 2018 - [info] OK. Sun Mar 11 15:53:07 2018 - [warning] shutdown_script is not defined. Sun Mar 11 15:53:07 2018 - [info] Set master ping interval 1 seconds. Sun Mar 11 15:53:07 2018 - [info] Set secondary check script: /usr/local/bin/masterha_secondary_check -s server03 -s server02 Sun Mar 11 15:53:07 2018 - [info] Starting ping health check on 10.0.0.60(10.0.0.60:3306).. Sun Mar 11 15:53:07 2018 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
其中參數的意思：
--orig_master_is_new_slave 切換時加上此參數是將原 master 變爲 slave 節點，若是不加此參數，原來的 master 將不啓動
--running_updates_limit=10000,故障切換時,候選master 若是有延遲的話， mha 切換不能成功，加上此參數表示延遲在此時間範圍內均可切換（單位爲s），可是切換的時間長短是由recover 時relay 日誌的大小決定
注意：因爲在線進行切換須要調用到master_ip_online_change這個腳本，可是因爲該腳本不完整，須要本身進行相應的修改，我google到後發現仍是有問題，腳本中new_master_password這個變量獲取不到，致使在線切換失敗，因此進行了相關的硬編碼，直接把mysql的root用戶密碼賦值給變量new_master_password，若是有哪位大牛知道緣由，請指點指點。這個腳本還能夠管理vip。下面貼出腳本：
#!/usr/bin/env perl # Copyright (C) 2011 DeNA Co.,Ltd. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA ## Note: This is a sample script and is not complete. Modify the script based on your environment. use strict; use warnings FATAL => 'all'; use Getopt::Long; use MHA::DBHelper; use MHA::NodeUtil; use Time::HiRes qw( sleep gettimeofday tv_interval ); use Data::Dumper; my $_tstart; my $_running_interval = 0.1; my ( $command, $orig_master_host, $orig_master_ip, $orig_master_port, $orig_master_user, $new_master_host, $new_master_ip, $new_master_port, $new_master_user, ); my $vip = '10.0.0.88/24'; # Virtual IP my $key = "1"; my $ssh_start_vip = "/sbin/ifconfig eth1:$key $vip"; my $ssh_stop_vip = "/sbin/ifconfig eth1:$key down"; my $ssh_user = "root"; my $new_master_password='123'; my $orig_master_password='123'; GetOptions( 'command=s' => \$command, #'ssh_user=s' => \$ssh_user, 'orig_master_host=s' => \$orig_master_host, 'orig_master_ip=s' => \$orig_master_ip, 'orig_master_port=i' => \$orig_master_port, 'orig_master_user=s' => \$orig_master_user, #'orig_master_password=s' => \$orig_master_password, 'new_master_host=s' => \$new_master_host, 'new_master_ip=s' => \$new_master_ip, 'new_master_port=i' => \$new_master_port, 'new_master_user=s' => \$new_master_user, #'new_master_password=s' => \$new_master_password, ); exit &main(); sub current_time_us { my ( $sec, $microsec ) = gettimeofday(); my $curdate = localtime($sec); return $curdate . " " . sprintf( "%06d", $microsec ); } sub sleep_until { my $elapsed = tv_interval($_tstart); if ( $_running_interval > $elapsed ) { sleep( $_running_interval - $elapsed ); } } sub get_threads_util { my $dbh = shift; my $my_connection_id = shift; my $running_time_threshold = shift; my $type = shift; $running_time_threshold = 0 unless ($running_time_threshold); $type = 0 unless ($type); my @threads; my $sth = $dbh->prepare("SHOW PROCESSLIST"); $sth->execute(); while ( my $ref = $sth->fetchrow_hashref() ) { my $id = $ref->{Id}; my $user = $ref->{User}; my $host = $ref->{Host}; my $command = $ref->{Command}; my $state = $ref->{State}; my $query_time = $ref->{Time}; my $info = $ref->{Info}; $info =~ s/^\s*(.*?)\s*$/$1/ if defined($info); next if ( $my_connection_id == $id ); next if ( defined($query_time) && $query_time < $running_time_threshold ); next if ( defined($command) && $command eq "Binlog Dump" ); next if ( defined($user) && $user eq "system user" ); next if ( defined($command) && $command eq "Sleep" && defined($query_time) && $query_time >= 1 ); if ( $type >= 1 ) { next if ( defined($command) && $command eq "Sleep" ); next if ( defined($command) && $command eq "Connect" ); } if ( $type >= 2 ) { next if ( defined($info) && $info =~ m/^select/i ); next if ( defined($info) && $info =~ m/^show/i ); } push @threads, $ref; } return @threads; } sub main { if ( $command eq "stop" ) { ## Gracefully killing connections on the current master # 1. Set read_only= 1 on the new master # 2. DROP USER so that no app user can establish new connections # 3. Set read_only= 1 on the current master # 4. Kill current queries # * Any database access failure will result in script die. my $exit_code = 1; eval { ## Setting read_only=1 on the new master (to avoid accident) my $new_master_handler = new MHA::DBHelper(); # args: hostname, port, user, password, raise_error(die_on_error)_or_not $new_master_handler->connect( $new_master_ip, $new_master_port, $new_master_user, $new_master_password, 1 ); print current_time_us() . " Set read_only on the new master.. "; $new_master_handler->enable_read_only(); if ( $new_master_handler->is_read_only() ) { print "ok.\n"; } else { die "Failed!\n"; } $new_master_handler->disconnect(); # Connecting to the orig master, die if any database error happens my $orig_master_handler = new MHA::DBHelper(); $orig_master_handler->connect( $orig_master_ip, $orig_master_port, $orig_master_user, $orig_master_password, 1 ); ## Drop application user so that nobody can connect. Disabling per-session binlog beforehand #$orig_master_handler->disable_log_bin_local(); #print current_time_us() . " Drpping app user on the orig master..\n"; #FIXME_xxx_drop_app_user($orig_master_handler); ## Waiting for N * 100 milliseconds so that current connections can exit my $time_until_read_only = 15; $_tstart = [gettimeofday]; my @threads = get_threads_util( $orig_master_handler->{dbh}, $orig_master_handler->{connection_id} ); while ( $time_until_read_only > 0 && $#threads >= 0 ) { if ( $time_until_read_only % 5 == 0 ) { printf "%s Waiting all running %d threads are disconnected.. (max %d milliseconds)\n", current_time_us(), $#threads + 1, $time_until_read_only * 100; if ( $#threads < 5 ) { print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . "\n" foreach (@threads); } } sleep_until(); $_tstart = [gettimeofday]; $time_until_read_only--; @threads = get_threads_util( $orig_master_handler->{dbh}, $orig_master_handler->{connection_id} ); } ## Setting read_only=1 on the current master so that nobody(except SUPER) can write print current_time_us() . " Set read_only=1 on the orig master.. "; $orig_master_handler->enable_read_only(); if ( $orig_master_handler->is_read_only() ) { print "ok.\n"; } else { die "Failed!\n"; } ## Waiting for M * 100 milliseconds so that current update queries can complete my $time_until_kill_threads = 5; @threads = get_threads_util( $orig_master_handler->{dbh}, $orig_master_handler->{connection_id} ); while ( $time_until_kill_threads > 0 && $#threads >= 0 ) { if ( $time_until_kill_threads % 5 == 0 ) { printf "%s Waiting all running %d queries are disconnected.. (max %d milliseconds)\n", current_time_us(), $#threads + 1, $time_until_kill_threads * 100; if ( $#threads < 5 ) { print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . "\n" foreach (@threads); } } sleep_until(); $_tstart = [gettimeofday]; $time_until_kill_threads--; @threads = get_threads_util( $orig_master_handler->{dbh}, $orig_master_handler->{connection_id} ); } print "Disabling the VIP on old master: $orig_master_host \n"; &stop_vip(); ## Terminating all threads print current_time_us() . " Killing all application threads..\n"; $orig_master_handler->kill_threads(@threads) if ( $#threads >= 0 ); print current_time_us() . " done.\n"; #$orig_master_handler->enable_log_bin_local(); $orig_master_handler->disconnect(); ## After finishing the script, MHA executes FLUSH TABLES WITH READ LOCK $exit_code = 0; }; if ($@) { warn "Got Error: $@\n"; exit $exit_code; } exit $exit_code; } elsif ( $command eq "start" ) { ## Activating master ip on the new master # 1. Create app user with write privileges # 2. Moving backup script if needed # 3. Register new master's ip to the catalog database # We don't return error even though activating updatable accounts/ip failed so that we don't interrupt slaves' recovery. # If exit code is 0 or 10, MHA does not abort my $exit_code = 10; eval { my $new_master_handler = new MHA::DBHelper(); # args: hostname, port, user, password, raise_error_or_not $new_master_handler->connect( $new_master_ip, $new_master_port, $new_master_user, $new_master_password, 1 ); ## Set read_only=0 on the new master #$new_master_handler->disable_log_bin_local(); print current_time_us() . " Set read_only=0 on the new master.\n"; $new_master_handler->disable_read_only(); ## Creating an app user on the new master #print current_time_us() . " Creating app user on the new master..\n"; #FIXME_xxx_create_app_user($new_master_handler); #$new_master_handler->enable_log_bin_local(); $new_master_handler->disconnect(); ## Update master ip on the catalog database, etc print "Enabling the VIP - $vip on the new master - $new_master_host \n"; &start_vip(); $exit_code = 0; }; if ($@) { warn "Got Error: $@\n"; exit $exit_code; } exit $exit_code; } elsif ( $command eq "status" ) { # do nothing exit 0; } else { &usage(); exit 1; } } # A simple system call that enable the VIP on the new master sub start_vip() { `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`; } # A simple system call that disable the VIP on the old_master sub stop_vip() { `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`; } sub usage { print "Usage: master_ip_online_change --command=start|stop|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n"; die; }
四.修復宕機的Master
一般狀況下自動切換之後，原master可能已經廢棄掉，待原master主機修復後，若是數據完整的狀況下，可能想把原來master從新做爲新主庫的slave，這時咱們能夠藉助當時自動切換時刻的MHA日誌來完成對原master的修復。下面是提取相關日誌的命令：
[root@manager app1]# grep -i "All other slaves should start" manager.log
獲取上述信息之後，就能夠直接在修復後的master上執行change master to相關操做，從新做爲從庫了。
最後補充一下郵件發送腳本send_report ，這個腳本在詢問一位朋友後可使用，以下：
#!/usr/bin/perl # Copyright (C) 2011 DeNA Co.,Ltd. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA ## Note: This is a sample script and is not complete. Modify the script based on your environment. use strict; use warnings FATAL => 'all'; use Mail::Sender; use Getopt::Long; #new_master_host and new_slave_hosts are set only when recovering master succeeded my ( $dead_master_host, $new_master_host, $new_slave_hosts, $subject, $body ); my $smtp='smtp.163.com'; my $mail_from='xxxx'; my $mail_user='xxxxx'; my $mail_pass='xxxxx'; my $mail_to=['xxxx','xxxx']; GetOptions( 'orig_master_host=s' => \$dead_master_host, 'new_master_host=s' => \$new_master_host, 'new_slave_hosts=s' => \$new_slave_hosts, 'subject=s' => \$subject, 'body=s' => \$body, ); mailToContacts($smtp,$mail_from,$mail_user,$mail_pass,$mail_to,$subject,$body); sub mailToContacts { my ( $smtp, $mail_from, $user, $passwd, $mail_to, $subject, $msg ) = @_; open my $DEBUG, "> /tmp/monitormail.log" or die "Can't open the debug file:$!\n"; my $sender = new Mail::Sender { ctype => 'text/plain; charset=utf-8', encoding => 'utf-8', smtp => $smtp, from => $mail_from, auth => 'LOGIN', TLS_allowed => '0', authid => $user, authpwd => $passwd, to => $mail_to, subject => $subject, debug => $DEBUG }; $sender->MailMsg( { msg => $msg, debug => $DEBUG } ) or print $Mail::Sender::Error; return 1; } # Do whatever you want here exit 0;
最後切換之後發送告警的郵件示例，注意，這個是我後續的測試，和上面環境出現的ip不一致不要在乎。
總結：
目前高可用方案能夠必定程度上實現數據庫的高可用，好比前面文章介紹的MMM，heartbeat+drbd，Cluster等。還有percona的Galera Cluster等。這些高可用軟件各有優劣。在進行高可用方案選擇時，主要是看業務還有對數據一致性方面的要求。最後出於對數據庫的高可用和數據一致性的要求，推薦使用MHA架構。
- 1. MySQL高可用架構之MHA MySQL高可用架構之MHA
- 2. 【MySQL】MySQL高可用架構之MHA
- 3. MySQL高可用架構MHA
- 4. mysql高可用架構MHA
- 5. MySQL高可用架構之MHA（轉）
- 6. MySQL高可用架構之MHA
- 7. mysql實現高可用架構之MHA
- 8. MySQL高可用架構之MHA（一）
- 9. Mysql高可用架構之MHA
- 10. MySQL集羣高可用架構之MHA
- 更多相關文章...
- • Docker 架構 - Docker教程
- • 您已經學習了 XML Schema，下一步學習什麼呢？ - XML Schema 教程
- • 適用於PHP初學者的學習線路和建議
- • Tomcat學習筆記（史上最全tomcat學習筆記）
-
每一个你不满意的现在，都有一个你没有努力的曾经。