基於MHA的MYSQL高可用方案

時間 2019-12-17

標籤基於 mha mysql 可用方案欄目 MySQL 简体版

原文原文鏈接

1、MHA 簡介node

MHA （Master High Availability ）目前在 MySQL 高可用方面是一個相對成熟的解決方案,它由日本 DeNA 公司的 youshimaton 員工（現就任於 Facebook 公司）開發，是一套優秀的做爲 MySQL 高可用性環境下故障切換和主從角色提高的高可用軟件。在 MySQL 故障切換過程當中，MHA 能作到在 0~30 秒以內自動完成數據庫的主從故障切換操做，而且在進行故障切換的過程當中，MHA 能在最大程度上保證數據的一致性，以達到真正意義上的高可用。MHA ：由兩部分組成：MHA Manager （管理節點）和 MHA Node ）（數據節點）。MHA Manager能夠單獨部署在一臺獨立的機器上管理多個 master-slave 集羣，也能夠部署在一臺 slave 節點上。MHA Node 運行在每臺 MySQL 服務器及 Manager 服務器上，MHA Manager 會定時探測集羣中的 master 節點，當 master 出現故障時，它能夠自動將擁有最新數據的 slave 提高爲新的 master，而後將全部其餘的 slave 從新指向新提高的 master。整個故障轉移過程對應用程序層面徹底透明。在 MHA 自動故障切換過程當中，MHA 會試圖從宕機的主服務器上保存二進制日誌，最大程度的保證數據不丟失，但這種操做是有機率性的。例如，若是主服務器硬件故障或沒法經過 ssh 訪問，MHA無法保存二進制日誌，只進行故障轉移從而丟失了最新的數據。使用MySQL5.5 的半同步複製，能夠下降數據丟失的風險。MHA 能夠與半同步複製結合起來。若是隻有一個 slave 已經收到了最新的二進制日誌，MHA 能夠將最新的二進制日誌應用於其餘全部的slave 服務器上，所以能夠保證全部節點的數據一致性。目前 MHA 主要支持一主多從的架構，要搭建 MHA, 要求一個複製集羣中必須最少有三臺數據庫服務器，一主二從，即一臺充當 master，一臺充當備用 master，另一臺充當從庫，由於至少須要三臺服務器，出於機器成本的考慮，淘寶也在該基礎上進行了改造，目前淘寶 TMHA 已經支持一主一從。另外對於想快速搭建的能夠參考：MHA 快速搭建咱們本身使用其實也可使用 1 主 1 從，可是 master 主機宕機後沒法切換，以及沒法補全binlog。master 的 mysqld 進程 crash 後，仍是能夠切換成功，以及補全 binlog 的。mysql

1.2 、 MHA 工具介紹
MHA 軟件由兩部分組成，Manager 工具包和 Node 工具包，具體的說明以下。
Manager 工具包主要包括如下幾個工具：
➢ masterha_check_ssh 檢查 MHA 的 SSH 配置情況
➢ masterha_check_repl 檢查 MySQL 複製情況
➢ masterha_manger 啓動 MHA
➢ masterha_check_status 檢測當前 MHA 運行狀態
➢ masterha_master_monitor 檢測 master 是否宕機
➢ masterha_master_switch 控制故障轉移（自動或者手動）
➢ masterha_conf_host 添加或刪除配置的 server 信息
Node 工具包（這些工具一般由 MHA Manager 的腳本觸發，無需人爲操做）主要包括如下幾
個工具：
➢ save_binary_logs 保存和複製 master 的二進制日誌
➢ apply_diff_relay_logs 識別差別的中繼日誌事件並將其差別的事件應用於其餘的
slave
➢ filter_mysqlbinlog 去除沒必要要的 ROLLBACK 事件（MHA 已再也不使用這個工具）
➢ purge_relay_logs 清除中繼日誌（不會阻塞 SQL 線程) linux

注意：爲了儘量的減小主庫硬件損壞宕機形成的數據丟失，所以在配置 MHA 的同時建議
成配置成 MySQL 5.5 sql

MHA環境圖：一主三從模式（經過VIP漂移）數據庫

角色	IP地址	主機名	ServerID	數據庫類型
Primary Master	192.168.200.111	server01	1	寫入
Slave1	192.168.200.112	server02	2	讀
Slave2	192.168.200.113	server03	3	讀
Slave3	192.168.200.114	server04	4	讀
Manager	192.168.200.115	server05	-	監控複製組
LVS Matser	192.168.200.116	server06	-	-
LVS Backup	192.168.200.117	server07	-	-

其中 Primary Master 對外提供寫服務，備選 Secondary Master （實際的 slave 提供讀服務，
slave1和slave2也提供相關的讀服務，一旦Primary Master宕機，將會把備選Secondary Master
提高爲新的 Primary Master，slave1 和 slave2 指向新的 master。vim

1.4 實驗拓撲環境安全

2、前期環境部署
2.1 1 、配置全部主機名稱bash

master1 主機:
hostname server01
bash
master2 主機:
hostname server02
bash
slave1 主機:
hostname server03
bash
slave2 主機:
hostname server04
bash
manager 主機:
hostname server05
bash

.2.2 2 、配置全部主機名及映射關係
[root@server01~]# vim /etc/hosts服務器

192.168.200.111 server01
192.168.200.112 server02
192.168.200.113 server03
192.168.200.114 server04
192.168.200.115 server05

經過SCP發送hosts映射到另外幾臺主機

scp /etc/hosts 192.168.200.111:/etc/ scp /etc/hosts 192.168.200.112:/etc/ scp /etc/hosts 192.168.200.113:/etc/ scp /etc/hosts 192.168.200.114:/etc/

2.3 3 、全部主機關閉防火牆和安全機制架構

systemctl stop iptables
systemctl stop firewalld
setenforce 0

3、安裝 MHA node
3.一、全部主機裝安裝 MHA node 及相關 perl 依賴包
安裝 epel 源:

1：導入epel源碼包

2：解壓rpm -ivh epel-release-latest-7.noarch.rpm

安裝依賴包
yum install -y perl-DBD-MySQL.x86_64 perl-DBI.x86_64 perl-CPAN perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker


注意：安裝後建議檢查一下所需軟件包是否所有安裝 rpm -q perl-DBD-MySQL.x86_64 perl-DBI.x86_64 perl-CPAN perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker perl-DBD-MySQL-4.023-6.el7.x86_64 perl-DBI-1.627-4.el7.x86_64

perl-CPAN-1.9800-292.el7.noarch
perl-ExtUtils-CBuilder-0.28.2.6-292.el7.noarch
perl-ExtUtils-MakeMaker-6.68-3.el7.noarch

3.二、全部主機裝上安裝 MHA Node

tar xf mha4mysql-node-0.56.tar.gz
cd mha4mysql-node-0.56/
perl Makefile.PL
make && make install

3.3 、MHA Node 安裝完後會在 /usr/local/bin 生成如下腳本

[root@server04 mha4mysql-node-0.56]# ls -l /usr/local/bin/
總用量 40
-r-xr-xr-x. 1 root root 16346 10月 22 13:27 apply_diff_relay_logs
-r-xr-xr-x. 1 root root 4807 10月 22 13:27 filter_mysqlbinlog
-r-xr-xr-x. 1 root root 7401 10月 22 13:27 purge_relay_logs
-r-xr-xr-x. 1 root root 7395 10月 22 13:27 save_binary_logs

4、192.168.200.115上安裝 MHA Manger以及perl-Config-Tiny-2.14-7.el7.noarch其他四臺不須要裝

注意：115安裝 MHA Manger 以前也須要安裝 MHA Node

4.一、首先安裝 MHA Manger 依賴的 perl 模塊（我這裏使用 yum 安裝）

192.168.200.115導入以下四個源碼包

[root@server05 ~]# ls
perl-Config-Tiny-2.14-7.el7.noarch.rpm mha4mysql-manager-0.56.tar.gz
epel-release-latest-7.noarch.rpm mha4mysql-node-0.56.tar.gz

安裝依賴包:
yum install -y perl perl-Log-Dispatch perl-Parallel-ForkManager perl-DBD-MySQL perl-DBI perl-Time-HiRes
yum -y install perl-Config-Tiny-2.14-7.el7.noarch.rpm

注意：安裝後建議檢查一下所需軟件包是否所有安裝

rpm -q perl cpan perl-Log-Dispatch perl-Parallel-ForkManager perl-DBD-MySQL perl-DBI perl-Time-HiRes perl-Config-Tiny perl-5.16.3-292.el7.x86_64 perl-Log-Dispatch-2.41-1.el7.1.noarch perl-Parallel-ForkManager-1.18-2.el7.noarch perl-DBD-MySQL-4.023-6.el7.x86_64 perl-DBI-1.627-4.el7.x86_64 perl-Time-HiRes-1.9725-3.el7.x86_64 perl-Config-Tiny-2.14-7.el7.noarch

注意：以前時候 perl-Config-Tiny.noarch 沒有安裝成功,後來用 cpan(cpan install Config::Tiny )

4.二、安裝 MHA Manger

tar xf mha4mysql-manager-0.56.tar.gz
cd mha4mysql-manager-0.56/
perl Makefile.PL
make && make install

4.三、安裝完成後會有如下腳本文件

[root@server05 ~]# ls -l /usr/local/bin/
總用量 76
-r-xr-xr-x 1 root root 16346 12 月 29 15:08 apply_diff_relay_logs -r-xr-xr-x 1 root root 4807 12 月 29 15:08 filter_mysqlbinlog -r-xr-xr-x 1 root root 1995 12 月 29 15:37 masterha_check_repl
-r-xr-xr-x 1 root root 1779 12 月 29 15:37 masterha_check_ssh -r-xr-xr-x 1 root root 1865 12 月 29 15:37 masterha_check_status -r-xr-xr-x 1 root root 3201 12 月 29 15:37 masterha_conf_host -r-xr-xr-x 1 root root 2517 12 月 29 15:37 masterha_manager -r-xr-xr-x 1 root root 2165 12 月 29 15:37 masterha_master_monitor -r-xr-xr-x 1 root root 2373 12 月 29 15:37 masterha_master_switch -r-xr-xr-x 1 root root 3879 12 月 29 15:37 masterha_secondary_check -r-xr-xr-x 1 root root 1739 12 月 29 15:37 masterha_stop -r-xr-xr-x 1 root root 7401 12 月 29 15:08 purge_relay_logs -r-xr-xr-x 1 root root 7395 12 月 29 15:08 save_binary_logs

5、置配置 SSH 密鑰對驗證
服務器之間須要實現密鑰對驗證。關於配置密鑰對驗證可看下面步驟。可是有一點須要
注意：不能禁止 password 登錄，不然會出現錯誤

1. 服務器先生成一個密鑰對
2. 把本身的公鑰傳給對方

5.1 、Server05(192.168.200.115) 上操做:

[root@server05 ~]# ssh-keygen -t rsa
[root@server05 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.111
[root@server05 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.112
[root@server05 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.113
[root@server05 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.114

注意：Server05 須要 鏈接 每一個主機測試，由於第一次 鏈接入 的時候須要輸入 yes ，影響後期故
障切換時，對於每的 個主機的 SSH 控制。

檢測祕鑰對是否能夠鏈接：

[root@server05 ~]# ssh server01

[root@server01 ~]# exit
登出
Connection to server01 closed.

[root@server05 ~]# ssh server02

[root@server05 ~]# ssh server03

[root@server05 ~]# ssh server04

5.2 、Primary Master(192.168.200.111)上操做:

[root@server01 ~]# ssh-keygen -t rsa
[root@server01 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.112 [root@server01 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.113 [root@server01 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.114

5.3 、Secondary Master(192.168.200.112)上操做:

[root@server02 ~]# ssh-keygen -t rsa
[root@server02 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.111 [root@server02 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.113 [root@server02 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.114

5.4 、slave1(192.168.200.113)上操做:

[root@server03 ~]# ssh-keygen -t rsa
[root@server03 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.111 [root@server03 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.112 [root@server03 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.114

5.5 、slave2(192.168.200.114)上操做:

[root@server04 ~]# ssh-keygen -t rsa
[root@server04 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.111 [root@server04 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.112 [root@server04 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.113

6、安裝 mysql：不作字符集，作字符集後期會報錯，安裝時，/etc/my.cnf不作任何改動
111-114 全部主機上的操做：

yum -y install mariadb mariadb-server mariadb-devel
systemctl start mariadb
netstat -lnpt | grep :3306


設置數據庫初始密碼（後續操做中使用）
mysqladmin -u root password 123123

7、搭建主從複製環境

注意：binlog-do-db 和 replicate-ignore-db 設置必須相同。 MHA 在啓動時候會檢測過濾規
則，若是過濾規則不一樣，MHA 將不啓動監控和故障轉移功能。

7.1 1 、修改 mysql 主機的配置文件
Primary Master(192.168.200.111)上操做:

vim /etc/my.cnf
[mysqld]
server-id = 1
log-bin=master-bin
log-slave-updates=true
relay_log_purge=0

systemctl restart mariadb
netstat -anpt | grep 3306

Secondary Master(192.168.200.112)上操做:

vim /etc/my.cnf
[mysqld]
server-id=2
log-bin=master-bin
log-slave-updates=true
relay_log_purge=0



systemctl restart mariadb
netstat -anpt | grep 3306

slave1(192.168.200.113)上操做:

vim /etc/my.cnf
[mysqld]
server-id=3 
log-bin=mysql-bin
relay-log=slave-relay-bin
log-slave-updates=true
relay_log_purge=0



systemctl restart mariadb
netstat -anpt | grep 3306

slave2(192.168.200.114)上操做:

vim /etc/my.cnf
[mysqld]
server-id=4 
log-bin=mysql-bin
relay-log=slave-relay-bin
log-slave-updates=true
relay_log_purge=0



systemctl restart mariadb
netstat -anpt | grep 3306

7.2 、在 Primary Master( 192.168 .200.111) ) 上對舊數據進行備份

mysqldump --master-data=2 --single-transaction -R --triggers -A > all.sql
參數解釋：
--master-data=2 錄備份時刻記錄 master 的的 Binlog 位置和 Position
--single-transaction 獲取一致性快照
-R 備份存儲過程和函數
-triggres 備份觸發器
-A 備份全部的庫。

沒有舊數據就不用了執行本次7.2的操做

7.三、4 、mysql服務器建立複製受權用戶及 查看主庫備份時的 binlog 名稱和位置：192.168.200.111上操做

[root@server01 ~]# mysql -uroot -p123123
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 2
Server version: 5.5.60-MariaDB MariaDB Server

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> grant replication slave on *.* to 'repl'@'192.168.200.%' identified by '123123';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.01 sec)

7.5 、把數據備份複製其餘主機

scp all.sql 192.168.200.112:/tmp/
scp all.sql 192.168.200.113:/tmp/
scp all.sql 192.168.200.114:/tmp/

7.6 、導入舊數據備份數據到到 112-114 主機上、而且112-114執行復制相關命令

mysql -uroot -p123456< /tmp/all.sql           #舊數據備份、沒有舊數據就不用執行這一段代碼

MariaDB [(none)]> stop slave;
Query OK, 0 rows affected, 1 warning (0.01 sec)

MariaDB [(none)]> CHANGE MASTER TO
-> MASTER_HOST='192.168.200.111',
-> MASTER_USER='repl',
-> MASTER_PASSWORD='123123',
-> MASTER_LOG_FILE='master-bin.000001',
-> MASTER_LOG_POS=474;
Query OK, 0 rows affected (0.03 sec)

MariaDB [(none)]> start slave;
Query OK, 0 rows affected (0.01 sec)

MariaDB [(none)]> show slave status\G

# 檢查 IO 和 和 SQL 線程是否爲：yes
Slave_IO_Running: Yes
Slave_SQL_Running: Yes

7. 7 、主從同步故障處理：若是七、6報錯誤就執行這段代碼，沒報錯忽略

Slave_IO_Running: No
Slave_SQL_Running: Yes
----------------------------------- 忽略部分信息-----------------------------------
Last_IO_Errno: 1236
Last_IO_Error: Got fatal error 1236 from master when reading data from
binary log: 'Could not find first log file name in binary log index file'
----------------------------------- 忽略部分信息-----------------------------------
處理方式：
stop slave;
reset slave;
set global sql_slave_skip_counter =1 ;
start slave;

7.8 、三臺 slave(112-114) 服務置器設置 read_y only 狀態
從庫對外只進提供讀服務，只因此沒有寫進 mysql 配置文件，是由於時隨時 server02 會提高爲master

方法一：在主頁面輸入

[root@server02 ~]# mysql -uroot -p123123 -e 'set global read_only=1'
[root@server03 ~]# mysql -uroot -p123123 -e 'set global read_only=1'
[root@server04 ~]# mysql -uroot -p123123 -e 'set global read_only=1'

方法二：三臺slave都在數據庫內輸入
MariaDB [(none)]> set global read_only=1;

7.9 、建立監控用戶（111-114 主機上的操做）：作監控主要爲了給115更加方便的監控master(111)主機有沒有Down，一旦111主機down了就經過VIP漂移把VIP給112主機

，112主機從slave變成master

grant all privileges on *.* to 'root'@'192.168.200.%' identified by '123123';

flush privileges;



每一個主機爲本身的主機名受權在111-114上執行：          #此處跳過不寫一旦後期報錯時再執行，以避免影響服務
grant all privileges on *.* to 'root'@'server號（01-04）' identified by '123123'; flush privileges;

**到這裏整個 mysql 主從集羣環境已經搭建完畢**

8、配置 MHA 環境：都在192.168.200.115下操做
8.1 、建立 MHA 的工做目錄及相關配置文件
Server05(192.168.200.115)：在軟件包解壓後的目錄裏面有樣例配置文件

[root@server05 ~]# mkdir /etc/masterha
[root@server05 ~]# cp mha4mysql-manager-0.56/samples/conf/app1.cnf /etc/masterha

8.2 、修改 app1.cnf 配置文件
/usr/local/bin/master_ip_failover 腳本須要根據本身環境修改 ip 和網卡名稱等。

vim /etc/masterha/app1.cnf
[server default]
# 設置 manager 的工做日誌
manager_workdir=/var/log/masterha/app1
# 設置 manager 的日誌, 這兩條都是默認存在的
manager_log=/var/log/masterha/app1/manager.log
# 設置 master 默認保存 binlog 的位置, 以便 MHA 能夠找到 master 日誌 master_binlog_dir=/var/lib/mysql # 設置自動 failover 時候的切換腳本 master_ip_failover_script= /usr/local/bin/master_ip_failover # 設置 mysql 中root 用戶的密碼 password=123123 user=root #ping 包的時間間隔 ping_interval=1 # 設置遠端 mysql 在發生切換時保存 binlog 的具體位置 remote_workdir=/tmp # 設置複製用戶的密碼和用戶名 repl_password=123123 repl_user=repl
[server1]
hostname=server01
port=3306
[server2]
hostname=server02
candidate_master=1
port=3306
check_repl_delay=0
[server3]
hostname=server03
port=3306
[server4]
hostname=server04
port=3306

8.3 、配置故障轉移腳本

[root@server05 ~]# vim /usr/local/bin/master_ip_failover
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port,
);
my $vip = '192.168.200.100'; # 寫入 VIP my $key = "1"; #非keepalived 方式切換腳本使用的 my $ssh_start_vip = "/sbin/ifconfig ens33:$key $vip"; my $ssh_stop_vip = "/sbin/ifconfig ens33:$key down"; # 那麼這裏寫服務的開關命令
$ssh_user = "root";
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);
exit &main();
sub main {
print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";
if ( $command eq "stop" || $command eq "stopssh" ) {
# $orig_master_host, $orig_master_ip, $orig_master_port are passed.
# If you manage master ip address at global catalog database,
# invalidate orig_master_ip here.
my $exit_code = 1;
#eval {
# print "Disabling the VIP on old master: $orig_master_host \n";
# &stop_vip();
# $exit_code = 0;
#};
eval {
print "Disabling the VIP on old master: $orig_master_host \n";
#my $ping=`ping -c 1 10.0.0.13 | grep "packet loss" | awk -F',' '{print $3}' | awk '{print $1}'`;
#if ( $ping le "90.0%"&& $ping gt "0.0%" ){
#$exit_code = 0;
#}
#else {
&stop_vip();
# updating global catalog, etc
$exit_code = 0;
#}
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
# all arguments are passed.
# If you manage master ip address at global catalog database,
# activate new_master_ip here.
# You can also grant write access (create user, set read_only=0, etc) here.
my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
`ssh $ssh_user\@$orig_master_ip \" $ssh_start_vip \"`;
exit 0;
}
else {
&usage();
exit 1;
}
}
# A simple system call that enable the VIP on the new master
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host
--orig_master_ip=ip --orig_master_port=port --
new_master_host=host --new_master_ip=ip --new_master_port=port\n"; }

[root@server05 ~]# chmod +x /usr/local/bin/master_ip_failover

8.4 、設置從庫 relay log 的清除方式( 112- - 114 ) ：若是八、7報錯時在112-114上執行這條命令

mysql -uroot -p123123 -e 'set global relay_log_purge=0;'

注意：

MHA 在故障切換的過程當中，從庫的恢復過程依賴於 relay log 的相關信息，因此這裏要
將 relay log 的自動清除設置爲 OFF，採用手動清除 relay log 的方式。在默認狀況下，從服務
器上的中繼日誌會在 SQL 線程執行完畢後被自動刪除。可是在 MHA 環境中，這些中繼日誌
在恢復其餘從服務器時可能會被用到，所以須要禁用中繼日誌的自動清除功能。按期清除中
繼日誌須要考慮到複製延時的問題。在 ext3 的文件系統下，刪除大的文件須要必定的時間，
會致使嚴重的複製延時。爲了不復制延時，須要暫時爲中繼日誌建立硬連接，由於在 linux
系統中經過硬連接刪除大文件速度會很快。（在 mysql 數據庫中，刪除大表時，一般也採用
創建硬連接的方式）

8.5 、配置從庫（ 112- - 114 ）relay_log 清除腳本加入計劃任務我的實驗時能夠不作這條操做

MHA 節點中包含了 pure_relay_logs 命令工具，它能夠爲中繼日誌建立硬連接，執行 SET
GLOBAL relay_log_purge=1,等待幾秒鐘以便 SQL 線程切換到新的中繼日誌，再執行 SET
GLOBAL relay_log_purge=0。

vim purge_relay_log.sh
#!/bin/bash
user=root
passwd=123123 # 注意：數據庫要有 密碼, 填本身所設置的密碼就能夠 ，前面設置過
port=3306
log_dir='/tmp'
work_dir='/tmp'
purge='/usr/local/bin/purge_relay_logs'
if [ ! -d $log_dir ]
then
mkdir $log_dir -p
fi
$purge --user=$user --password=$passwd --disable_relay_log_purge --port=$port
--workdir=$work_dir >> $log_dir/purge_relay_logs.log 2>&1
chmod +x purge_relay_log.sh
crontab -e
0 4 * * * /bin/bash /root/purge_relay_log.sh

pure_relay_logs 腳本參數以下所示：
--user mysql 用戶名
--password mysql 密碼
--port 端口號
--workdir 指定建立 relay log 的硬連接的位置，默認是/var/tmp，因爲系統不一樣
分區建立硬連接文件會失敗，故須要執行硬連接具體位置，成功執行腳本後，硬連接的中繼
日誌文件被刪除
--disable_relay_log_purge 默認狀況下，若是 relay_log_purge=1，腳本會什麼都不清理，自
動退出，經過設定這個參數，當 relay_log_purge=1 的狀況下會將 relay_log_purge 設置爲 0。
清理 relay log 以後，最後將參數設置爲 OFF。

8.6 、手動清除中繼日誌能夠不作這條操做
purge_relay_logs --user=root --password=123456 --disable_relay_log_purge --port=3306
--workdir=/tmp
2017-08-31 21:33:52: purge_relay_logs script started.
Found relay_log.info: /usr/local/mysql/data/relay-log.info
Removing hard linked relay log files slave-relay-bin* under /tmp.. done.
Current relay log file: /usr/local/mysql/data/slave-relay-bin.000002
Archiving unused relay log files (up to /usr/local/mysql/data/slave-relay-bin.000001) ...
Creating hard link for /usr/local/mysql/data/slave-relay-bin.000001 under
/tmp/slave-relay-bin.000001 .. ok.
Creating hard links for unused relay log files completed.
Executing SET GLOBAL relay_log_purge=1; FLUSH LOGS; sleeping a few seconds so that SQL
thread can delete older relay log files (if i
t keeps up); SET GLOBAL relay_log_purge=0; .. ok. Removing hard linked relay log files
slave-relay-bin* under /tmp.. done.
2017-08-31 21:33:56: All relay log purging operations succeeded.

8.7 、檢查 MHA ssh 通訊狀態必須執行的操做

[root@server05 ~]# masterha_check_ssh --conf=/etc/masterha/app1.cnf

Sat Dec 29 16:03:57 2018 - [warning] Global configuration file /etc/masterha_default.cnf not
found. Skipping.
Sat Dec 29 16:03:57 2018 - [info] Reading application default configurations from
/etc/masterha/app1.cnf..
Sat Dec 29 16:03:57 2018 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Sat Dec 29 16:03:57 2018 - [info] Starting SSH connection tests..
Sat Dec 29 16:04:02 2018 - [debug]
Sat Dec 29 16:03:58 2018 - [debug] Connecting via SSH from
root@server02(192.168.200.112:22) to root@server01(192.168.200.111:22)..
Sat Dec 29 16:03:59 2018 - [debug] ok.
----------------
---------------------
Sat Dec 29 16:04:01 2018 - [debug] Connecting via SSH from
root@server04(192.168.200.114:22) to root@server03(192.168.200.113:22)..
Sat Dec 29 16:04:02 2018 - [debug] ok.
Sat Dec 29 16:04:02 2018 - [info] All SSH connection tests passed successfully.           #有這段話表示暫時也沒有問題

8.9 、檢查整個主從複製集羣的狀態

[root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf

Tue Oct 22 17:28:43 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Oct 22 17:28:43 2019 - [info] Reading application default configurations from /etc/masterha/app1.cnf..
Tue Oct 22 17:28:43 2019 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Tue Oct 22 17:28:43 2019 - [info] MHA::MasterMonitor version 0.56.
Tue Oct 22 17:28:44 2019 - [info] Dead Servers:
Tue Oct 22 17:28:44 2019 - [info] Alive Servers:
Tue Oct 22 17:28:44 2019 - [info] server01(192.168.200.111:3306) Tue Oct 22 17:28:44 2019 - [info] server02(192.168.200.112:3306) Tue Oct 22 17:28:44 2019 - [info] server03(192.168.200.113:3306)  #有四個活着的主機 Tue Oct 22 17:28:44 2019 - [info] server04(192.168.200.114:3306)
Tue Oct 22 17:28:44 2019 - [info] Alive Slaves:
Tue Oct 22 17:28:44 2019 - [info] server02(192.168.200.112:3306) Version=5.5.60-MariaDB (oldest major version between slaves) log-bin:enabled
Tue Oct 22 17:28:44 2019 - [info] Replicating from 192.168.200.111(192.168.200.111:3306)
Tue Oct 22 17:28:44 2019 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Oct 22 17:28:44 2019 - [info] server03(192.168.200.113:3306) Version=5.5.60-MariaDB (oldest major version between slaves) log-bin:enabled
Tue Oct 22 17:28:44 2019 - [info] Replicating from 192.168.200.111(192.168.200.111:3306)
Tue Oct 22 17:28:44 2019 - [info] server04(192.168.200.114:3306) Version=5.5.64-MariaDB (oldest major version between slaves) log-bin:enabled
Tue Oct 22 17:28:44 2019 - [info] Replicating from 192.168.200.111(192.168.200.111:3306)
Tue Oct 22 17:28:44 2019 - [info] Current Alive Master: server01(192.168.200.111:3306)
Tue Oct 22 17:28:44 2019 - [info] Checking slave configurations..
Tue Oct 22 17:28:44 2019 - [warning] relay_log_purge=0 is not set on slave server02(192.168.200.112:3306). Tue Oct 22 17:28:44 2019 - [warning] relay_log_purge=0 is not set on slave server03(192.168.200.113:3306). #報錯點三處 Tue Oct 22 17:28:44 2019 - [warning] relay_log_purge=0 is not set on slave server04(192.168.200.114:3306).
Tue Oct 22 17:28:44 2019 - [info] Checking replication filtering settings..
Tue Oct 22 17:28:44 2019 - [info] binlog_do_db= , binlog_ignore_db= 
Tue Oct 22 17:28:44 2019 - [info] Replication filtering check ok.
Tue Oct 22 17:28:44 2019 - [error][/usr/local/share/perl5/MHA/Server.pm, ln383] server04(192.168.200.114:3306): #報server04上沒有主從複製的用戶

User repl does not exist or does not have REPLICATION SLAVE privilege! Other slaves can not start replication from this host.
Tue Oct 22 17:28:44 2019 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln401] Error happend on checking configurations.
 at /usr/local/share/perl5/MHA/ServerManager.pm line 1354.
Tue Oct 22 17:28:44 2019 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln500] Error happened on monitoring servers.
Tue Oct 22 17:28:44 2019 - [info] Got exit code 1 (Not master dead).

MySQL Replication Health is NOT OK!

若是出現四個活主機三個報錯點這種報錯就執行八、4清除中繼日誌在（112-114）上執行：

[root@server02 ~]# mysql -uroot -p123123 -e 'set global relay_log_purge=0;'

[root@server03 ~]# mysql -uroot -p123123 -e 'set global relay_log_purge=0;'

[root@server04~]# mysql -uroot -p123123 -e 'set global relay_log_purge=0;'

115主機再次查看日誌：

[root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf
Tue Oct 22 19:18:56 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Oct 22 19:18:56 2019 - [info] Reading application default configurations from /etc/masterha/app1.cnf..
Tue Oct 22 19:18:56 2019 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Tue Oct 22 19:18:56 2019 - [info] MHA::MasterMonitor version 0.56.
Tue Oct 22 19:18:58 2019 - [info] Dead Servers:
Tue Oct 22 19:18:58 2019 - [info] Alive Servers:
Tue Oct 22 19:18:58 2019 - [info] server01(192.168.200.111:3306) Tue Oct 22 19:18:58 2019 - [info] server02(192.168.200.112:3306) Tue Oct 22 19:18:58 2019 - [info] server03(192.168.200.113:3306) Tue Oct 22 19:18:58 2019 - [info] server04(192.168.200.114:3306)
Tue Oct 22 19:18:58 2019 - [info] Alive Slaves:
Tue Oct 22 19:18:58 2019 - [info] server02(192.168.200.112:3306) Version=5.5.60-MariaDB (oldest major version between slaves) log-bin:enabled
Tue Oct 22 19:18:58 2019 - [info] Replicating from 192.168.200.111(192.168.200.111:3306)
Tue Oct 22 19:18:58 2019 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Oct 22 19:18:58 2019 - [info] server03(192.168.200.113:3306) Version=5.5.60-MariaDB (oldest major version between slaves) log-bin:enabled
Tue Oct 22 19:18:58 2019 - [info] Replicating from 192.168.200.111(192.168.200.111:3306)
Tue Oct 22 19:18:58 2019 - [info] server04(192.168.200.114:3306) Version=5.5.64-MariaDB (oldest major version between slaves) log-bin:enabled
Tue Oct 22 19:18:58 2019 - [info] Replicating from 192.168.200.111(192.168.200.111:3306)
Tue Oct 22 19:18:58 2019 - [info] Current Alive Master: server01(192.168.200.111:3306)
Tue Oct 22 19:18:58 2019 - [info] Checking slave configurations..
Tue Oct 22 19:18:58 2019 - [info] Checking replication filtering settings..
Tue Oct 22 19:18:58 2019 - [info] binlog_do_db= , binlog_ignore_db= 
Tue Oct 22 19:18:58 2019 - [info] Replication filtering check ok.
Tue Oct 22 19:18:58 2019 - [error][/usr/local/share/perl5/MHA/Server.pm, ln383] server04(192.168.200.114:3306): #報server04上沒有主從複製的用戶

User repl does not exist or does not have REPLICATION SLAVE privilege! Other slaves can not start replication from this host.
Tue Oct 22 19:18:58 2019 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln401] Error happend on checking configurations.
at /usr/local/share/perl5/MHA/ServerManager.pm line 1354.
Tue Oct 22 19:18:58 2019 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln500] Error happened on monitoring servers.
Tue Oct 22 19:18:58 2019 - [info] Got exit code 1 (Not master dead).

MySQL Replication Health is NOT OK!

報server04上的錯誤就再次在114上執行用戶權限配置：

MariaDB [(none)]> grant replication slave on *.* to 'repl'@'192.168.200.%' identified by '123123';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.00 sec)

115主機再次查看日誌：

[root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf
Tue Oct 22 19:30:32 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Oct 22 19:30:32 2019 - [info] Reading application default configurations from /etc/masterha/app1.cnf..
Tue Oct 22 19:30:32 2019 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Tue Oct 22 19:30:32 2019 - [info] MHA::MasterMonitor version 0.56.
Tue Oct 22 19:30:33 2019 - [info] Dead Servers:
Tue Oct 22 19:30:33 2019 - [info] Alive Servers:
Tue Oct 22 19:30:33 2019 - [info]   server01(192.168.200.111:3306)
Tue Oct 22 19:30:33 2019 - [info]   server02(192.168.200.112:3306)
Tue Oct 22 19:30:33 2019 - [info]   server03(192.168.200.113:3306)
Tue Oct 22 19:30:33 2019 - [info]   server04(192.168.200.114:3306)
Tue Oct 22 19:30:33 2019 - [info] Alive Slaves:
Tue Oct 22 19:30:33 2019 - [info]   server02(192.168.200.112:3306)  Version=5.5.60-MariaDB (oldest major version between slaves) log-bin:enabled
Tue Oct 22 19:30:33 2019 - [info]     Replicating from 192.168.200.111(192.168.200.111:3306)
Tue Oct 22 19:30:33 2019 - [info]     Primary candidate for the new Master (candidate_master is set)
Tue Oct 22 19:30:33 2019 - [info]   server03(192.168.200.113:3306)  Version=5.5.60-MariaDB (oldest major version between slaves) log-bin:enabled
Tue Oct 22 19:30:33 2019 - [info]     Replicating from 192.168.200.111(192.168.200.111:3306)
Tue Oct 22 19:30:33 2019 - [info]   server04(192.168.200.114:3306)  Version=5.5.64-MariaDB (oldest major version between slaves) log-bin:enabled
Tue Oct 22 19:30:33 2019 - [info]     Replicating from 192.168.200.111(192.168.200.111:3306)
Tue Oct 22 19:30:33 2019 - [info] Current Alive Master: server01(192.168.200.111:3306)
Tue Oct 22 19:30:33 2019 - [info] Checking slave configurations..
Tue Oct 22 19:30:33 2019 - [info] Checking replication filtering settings..
Tue Oct 22 19:30:33 2019 - [info]  binlog_do_db= , binlog_ignore_db= 
Tue Oct 22 19:30:33 2019 - [info]  Replication filtering check ok.
Tue Oct 22 19:30:33 2019 - [info] Starting SSH connection tests..
Tue Oct 22 19:30:39 2019 - [info] All SSH connection tests passed successfully.
Tue Oct 22 19:30:39 2019 - [info] Checking MHA Node version..
Tue Oct 22 19:30:41 2019 - [info]  Version check ok.
Tue Oct 22 19:30:41 2019 - [info] Checking SSH publickey authentication settings on the current master..
Tue Oct 22 19:30:41 2019 - [info] HealthCheck: SSH to server01 is reachable.
Tue Oct 22 19:30:42 2019 - [info] Master MHA Node version is 0.56.
Tue Oct 22 19:30:42 2019 - [info] Checking recovery script configurations on the current master..
Tue Oct 22 19:30:42 2019 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql 
--output_file=/tmp/save_binary_logs_test
 --manager_version=0.56 --start_file=master-bin.000002 
Tue Oct 22 19:30:42 2019 - [info]   Connecting to root@server01(server01).. 
  Creating /tmp if not exists..    ok.
  Checking output directory is accessible or not..
   ok.
  Binlog found at /var/lib/mysql, up to master-bin.000002
Tue Oct 22 19:30:42 2019 - [info] Master setting check done.
Tue Oct 22 19:30:42 2019 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Tue Oct 22 19:30:42 2019 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=server02 
--slave_ip=192.168.200.112 --slave_port=3306 
--workdir=/tmp --target_version=5.5.60-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info  
--relay_dir=/var/lib/mysql/  --slave_pass=xxx
Tue Oct 22 19:30:42 2019 - [info] Connecting to root@192.168.200.112(server02:22)..  #server02報受權錯誤
mysqlbinlog: unknown variable 'default-character-set=utf8'
mysqlbinlog version command failed with rc 7:0, please verify PATH, LD_LIBRARY_PATH, and client options
 at /usr/local/bin/apply_diff_relay_logs line 492.
Tue Oct 22 19:30:43 2019 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln201] Slaves settings check failed!
Tue Oct 22 19:30:43 2019 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln390] Slave configuration failed.
Tue Oct 22 19:30:43 2019 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln401] Error happend on checking configurations. 
 at /usr/local/bin/masterha_check_repl line 48.
Tue Oct 22 19:30:43 2019 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln500] Error happened on monitoring servers.
Tue Oct 22 19:30:43 2019 - [info] Got exit code 1 (Not master dead).

報受權錯誤時每一個主機爲本身的主機名受權：在111-114上執行：        
grant all privileges on *.* to 'root'@'server號（01-04）' identified by '123123'; flush privileges;

115主機再次執行日誌：

[root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf
Tue Oct 22 20:47:24 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Oct 22 20:47:24 2019 - [info] Reading application default configurations from /etc/masterha/app1.cnf..
Tue Oct 22 20:47:24 2019 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Tue Oct 22 20:47:24 2019 - [info] MHA::MasterMonitor version 0.56.
Tue Oct 22 20:47:25 2019 - [info] Dead Servers:
Tue Oct 22 20:47:25 2019 - [info] Alive Servers:
Tue Oct 22 20:47:25 2019 - [info] server01(192.168.200.111:3306)
Tue Oct 22 20:47:25 2019 - [info] server02(192.168.200.112:3306)
Tue Oct 22 20:47:25 2019 - [info] server03(192.168.200.113:3306)
Tue Oct 22 20:47:25 2019 - [info] server04(192.168.200.114:3306)
Tue Oct 22 20:47:25 2019 - [info] Alive Slaves:
Tue Oct 22 20:47:25 2019 - [info] server02(192.168.200.112:3306) Version=5.5.60-MariaDB (oldest major version between slaves) log-bin:enabled
Tue Oct 22 20:47:25 2019 - [info] Replicating from 192.168.200.111(192.168.200.111:3306)
Tue Oct 22 20:47:25 2019 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Oct 22 20:47:25 2019 - [info] server03(192.168.200.113:3306) Version=5.5.60-MariaDB (oldest major version between slaves) log-bin:enabled
Tue Oct 22 20:47:25 2019 - [info] Replicating from 192.168.200.111(192.168.200.111:3306)
Tue Oct 22 20:47:25 2019 - [info] server04(192.168.200.114:3306) Version=5.5.64-MariaDB (oldest major version between slaves) log-bin:enabled
Tue Oct 22 20:47:25 2019 - [info] Replicating from 192.168.200.111(192.168.200.111:3306)
Tue Oct 22 20:47:25 2019 - [info] Current Alive Master: server01(192.168.200.111:3306)
Tue Oct 22 20:47:25 2019 - [info] Checking slave configurations..
Tue Oct 22 20:47:25 2019 - [info] read_only=1 is not set on slave server02(192.168.200.112:3306).
Tue Oct 22 20:47:25 2019 - [info] read_only=1 is not set on slave server03(192.168.200.113:3306).
Tue Oct 22 20:47:25 2019 - [info] read_only=1 is not set on slave server04(192.168.200.114:3306).
Tue Oct 22 20:47:25 2019 - [info] Checking replication filtering settings..
Tue Oct 22 20:47:25 2019 - [info] binlog_do_db= , binlog_ignore_db= 
Tue Oct 22 20:47:25 2019 - [info] Replication filtering check ok.
Tue Oct 22 20:47:25 2019 - [info] Starting SSH connection tests..
Tue Oct 22 20:47:31 2019 - [info] All SSH connection tests passed successfully.
Tue Oct 22 20:47:31 2019 - [info] Checking MHA Node version..
Tue Oct 22 20:47:33 2019 - [info] Version check ok.
Tue Oct 22 20:47:33 2019 - [info] Checking SSH publickey authentication settings on the current master..
Tue Oct 22 20:47:33 2019 - [info] HealthCheck: SSH to server01 is reachable.
Tue Oct 22 20:47:34 2019 - [info] Master MHA Node version is 0.56.
Tue Oct 22 20:47:34 2019 - [info] Checking recovery script configurations on the current master..
Tue Oct 22 20:47:34 2019 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/tmp/save_binary_logs_test --manager_version=0.56 --start_file=master-bin.000004 
Tue Oct 22 20:47:34 2019 - [info] Connecting to root@server01(server01).. 
Creating /tmp if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /var/lib/mysql, up to master-bin.000004
Tue Oct 22 20:47:34 2019 - [info] Master setting check done.
Tue Oct 22 20:47:34 2019 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Tue Oct 22 20:47:34 2019 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=server02 --slave_ip=192.168.200.112 --slave_port=3306 --workdir=/tmp --target_version=5.5.60-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx
Tue Oct 22 20:47:34 2019 - [info] Connecting to root@192.168.200.112(server02:22).. 
Checking slave recovery environment settings..
Opening /var/lib/mysql/relay-log.info ... ok.
Relay log found at /var/lib/mysql, up to mariadb-relay-bin.000011
Temporary relay log file is /var/lib/mysql/mariadb-relay-bin.000011
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Tue Oct 22 20:47:35 2019 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=server03 --slave_ip=192.168.200.113 --slave_port=3306 --workdir=/tmp --target_version=5.5.60-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx
Tue Oct 22 20:47:35 2019 - [info] Connecting to root@192.168.200.113(server03:22).. 
Checking slave recovery environment settings..
Opening /var/lib/mysql/relay-log.info ... ok.
server01 (current master) +--server02 +--server03 +--server04
--------
--------------
MySQL Replication Health is OK.


    ** 返回 OK 表示沒有問題**

9、VIP 配置管理
Master vip 配置有兩種方式,一種是經過 keepalived 或者 heartbeat 相似的軟件的方式管
理 VIP 的漂移,另外一種爲經過命令方式管理。

經過命令方式管理 VIP 地址：
打開115在前面編輯過的文件/etc/masterha/app1.cnf，檢查以下行是否正確，再檢查集羣狀態。

Server05(192.168.200.115)修改故障轉移腳本
[root@server05 ~]# grep -n 'master_ip_failover_script' /etc/masterha/app1.cnf
master_ip_failover_script= /usr/local/bin/master_ip_failover #用於存放腳本的位置

Server05(192.168.200.115)修改故障轉移腳本
[root@server05 ~]# head -13 /usr/local/bin/master_ip_failover

#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port,
);
my $vip = '192.168.200.100'; # 寫入 VIP
my $key = "1"; #非 非 keepalived 方式切換腳本使用的
my $ssh_start_vip = "/sbin/ifconfig ens33:$key $vip"; # 如果使用 keepalived
my $ssh_stop_vip = "/sbin/ifconfig ens33:$key down"; # 那麼這裏寫服務的 開關命令

/usr/local/bin/master_ip_failover 文件的內容意思是當主庫發生故障時，會觸發 MHA 切
換，MHA manager 會停掉主庫上的 ens32:1 接口，觸發虛擬 ip 漂移到備選從庫，從而完成
切換。

Server05(192.168.200.115) 檢查 manager 狀態

[root@server05 ~]# masterha_check_status --conf=/etc/masterha/app1.cnf
app1 is stopped(2:NOT_RUNNING).

 注意：若是正常會顯示"PING_OK"，不然會顯示"NOT_RUNNING"，表明 MHA 監控沒有開啓。

Server05(192.168.200.115) 開啓 manager 監控

[root@server05 ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dea
d_master_conf --ignore_last_failover< /dev/null >/var/log/masterha/app1/manager.log 2>&1&
[1] 11275

啓動參數介紹：
--remove_dead_master_conf 該參數表明當發生主從切換後，老的主庫的 ip 將會從配置
文件中移除。
--manger_log 日誌存放位置
--ignore_last_failover 在缺省狀況下，若是 MHA 檢測到連續發生宕機，且兩次宕
機間隔不足 8 小時的話，則不會進行 Failover，之因此這樣限制是爲了不 ping-pong 效應。
該參數表明忽略上次 MHA 觸發切換產生的文件，默認狀況下，MHA 發生切換後會在日誌目
錄，也就是上面我設置的/data 產生 app1.failover.complete 文件，下次再次切換的時候若是
發現該目錄下存在該文件將不容許觸發切換，除非在第一次切換後收到刪除該文件，爲了方
便，這裏設置爲--ignore_last_failover。

Server05(192.168.200.115)再次查看 Server05 監控是否正常：

[root@monitor ~]# masterha_check_status --conf=/etc/masterha/app1.cnf
 app1 (pid:65837) is running(0:PING_OK), master:server01
 
 有PING_OK、表示已經在監控了

Primary Master(192.168.200.111)上操做：

[root@server01 ~]# ip a | grep ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 192.168.200.111/24 brd 192.168.200.255 scope global noprefixroute ens33
inet 192.168.200.100/24 brd 192.168.200.255 scope global secondary ens33:1

此時111做爲master有了192.168.200.100的VIP

Server05(192.168.200.115)查看啓動日誌

[root@server05 ~]# cat /var/log/masterha/app1/manager.log
Wed Oct 23 09:31:46 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Wed Oct 23 09:31:46 2019 - [info] Reading application default configurations from /etc/masterha/app1.cnf..
Wed Oct 23 09:31:46 2019 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Wed Oct 23 09:31:46 2019 - [info] MHA::MasterMonitor version 0.56.
Wed Oct 23 09:31:47 2019 - [info] Dead Servers:
Wed Oct 23 09:31:47 2019 - [info] Alive Servers:
Wed Oct 23 09:31:47 2019 - [info] server01(192.168.200.111:3306)
Wed Oct 23 09:31:47 2019 - [info] server02(192.168.200.112:3306)
Wed Oct 23 09:31:47 2019 - [info] server03(192.168.200.113:3306)
Wed Oct 23 09:31:47 2019 - [info] server04(192.168.200.114:3306)
-------

-------------
Wed Oct 23 09:31:59 2019 - [info]
server01 (current master)
+--server02
+--server03
+--server04

Wed Oct 23 09:32:00 2019 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..

**注意：其中"Ping(SELECT) succeeded, waiting until MySQL doesn't respond.."說明整個系統已經開始監控了**

Primary Master(192.168.200.111) 模擬主庫故障
[root@server01 ~]# systemctl stop mariadb
[root@server01 ~]# netstat -lnpt | grep :3306
[root@server01 ~]# ip a | grep ens32

2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group
default qlen 1000
inet 192.168.200.111/24 brd 192.168.200.255 scope global ens32

在11三、114上都查看狀態：

****若是當down了111時在113-114上看不到Master_Host: 192.168.200.112 且爲111時而且報IO線程問題，就查看一下115可否經過ssh server02祕鑰鏈接到112上

若是要輸入密碼就輸入yes才能進入，就反覆重啓111-114的mariadb服務（[root@server03 ~]# systemctl restart mariadb）***

像這樣就是IO線程的ssh祕鑰沒問題

步驟：

[root@server03 ~]# systemctl restart mariadb #111-114都要重啓

root@server05 ~]# ssh server02
Last login: Wed Oct 23 13:42:18 2019 from 192.168.200.2

[root@server05 ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover< /dev/null >/var/log/masterha/app1/manager.log 2>&1&
[6] 20465

[root@server05 ~]# masterha_check_status --conf=/etc/masterha/app1.cnf
app1 (pid:20465) is running(0:PING_OK), master:server01

[root@server02 ~]# ip a | grep ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 192.168.200.112/24 brd 192.168.200.255 scope global noprefixroute ens33
inet 192.168.200.100/24 brd 192.168.200.255 scope global secondary ens33:1

slave1(192.168.200.11三、192.168.200.114)查看狀態：

MariaDB [(none)]> show slave status\G *************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.200.112  #發現此時VIP已經漂移到192.168.200.112上，112成爲了master
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: master-bin.000001
Read_Master_Log_Pos: 1372
Relay_Log_File: slave-relay-bin.000002
Relay_Log_Pos: 530
Relay_Master_Log_File: master-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes

Server05(192.168.200.115) 查看監控配置文件已經發生了變化（[server01] 的配置已被刪除）：

[root@server05 ~]# cat /etc/masterha/app1.cnf
[server default]
manager_log=/var/log/masterha/app1/manager.log
manager_workdir=/var/log/masterha/app1
master_binlog_dir=/var/lib/mysql
master_ip_failover_script=/usr/local/bin/master_ip_failover
password=123123
ping_interval=1
remote_workdir=/tmp
repl_password=123123
repl_user=repl
user=root

[server2]
candidate_master=1
check_repl_delay=0
hostname=server02
port=3306

[server3]
hostname=server03
port=3306

[server4]
hostname=server04
port=3306

Server05(192.168.200.115) 故障切換過程當中的日誌文件內容以下：

[root@server05 ~]# tail -f /var/log/masterha/app1/manager.log
Selected server02 as a new master.
server02: OK: Applying all logs succeeded.
server02: OK: Activated master IP address.
server04: This host has the latest relay log events.
server03: This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
server04: OK: Applying all logs succeeded. Slave started, replicating from server02. server03: OK: Applying all logs succeeded. Slave started, replicating from server02.
server02: Resetting slave info succeeded.
Master failover to server02(192.168.200.112:3306) completed successfully.

故障主庫修復及及 VIP 切回測試
Primary Master(192.168.200.111)上操做：

[root@server01 ~]# systemctl start mariadb
[root@server01 ~]# netstat -lnpt | grep :3306
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN
6131/mysqld
Primary Master(192.168.200.111) 指向新的master庫
[root@server01 ~]# mysql -u root -p123123
MariaDB [(none)]> start slave;
MariaDB [(none)]>CHANGE MASTER TO

>MASTER_HOST='192.168.200.112',
>MASTER_USER='repl',
>MASTER_PASSWORD='123123';
MariaDB [(none)]> start slave;
MariaDB [(none)]> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.200.112
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: master-bin.000001
Read_Master_Log_Pos: 1372
Relay_Log_File: mariadb-relay-bin.000002
Relay_Log_Pos: 1208
Relay_Master_Log_File: master-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes

Server05(192.168.200.115) 修改監控配置文件從新添加 server1 配置：

[root@server05 ~]# vim /etc/masterha/app1.cnf
[server01]
hostname=server01
port=3306

Server05(192.168.200.115) 檢查集羣狀態：

[root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf
----------------------------------- 忽略部分信息-----------------------------------
Thu Aug 31 22:20:30 2017 - [info] Alive Servers:
Thu Aug 31 22:20:30 2017 - [info] server01(192.168.200.111:3306)
Thu Aug 31 22:20:30 2017 - [info] server02(192.168.200.112:3306)
Thu Aug 31 22:20:30 2017 - [info] server03(192.168.200.113:3306)
Thu Aug 31 22:20:30 2017 - [info] server04(192.168.200.114:3306)
----------------------------------- 忽略部分信息-----------------------------------
server02 (current master)
+--server01
+--server03
+--server04
----------------------------------- 忽略部分信息-----------------------------------
MySQL Replication Health is OK.

Server05(192.168.200.115) 開啓監控

[root@server05 ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover< /dev/null >/var/log/masterha/app1/manager.log 2>&1&
[7] 22551

Secondary Master(192.168.200.112) 關閉現有主庫 mysql

[root@server02 ~]# systemctl stop mariadb

[root@server02 ~]# ip a | grep ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 192.168.200.112/24 brd 192.168.200.255 scope global noprefixroute ens33

Primary Master(192.168.200.111)：查看VIP有沒有回來

[root@server01 ~]# ip a | grep ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 192.168.200.111/24 brd 192.168.200.255 scope global noprefixroute ens33
inet 192.168.200.100/24 brd 192.168.200.255 scope global secondary ens33:1

slave1(192.168.200.11三、192.168.200.114)查看狀態：

MariaDB [(none)]> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.200.111
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: master-bin.000012
          Read_Master_Log_Pos: 1586
               Relay_Log_File: slave-relay-bin.000002
                Relay_Log_Pos: 530
        Relay_Master_Log_File: master-bin.000012
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes

Server05(192.168.200.115) 配置文件變化（已經移除故障機 server2 配置）：

[root@server05 ~]# cat /etc/masterha/app1.cnf
[server default]
manager_log=/var/log/masterha/app1/manager.log
manager_workdir=/var/log/masterha/app1
master_binlog_dir=/var/lib/mysql
master_ip_failover_script=/usr/local/bin/master_ip_failover
password=123456
ping_interval=1
remote_workdir=/tmp
repl_password=123456
repl_user=repl
user=root
[server1]
hostname=server01
port=3306
[server3]
hostname=server03
port=3306
[server4]
hostname=server04
port=3306

10、配置讀請求負載均衡
10.1 、安裝 LVS 和 Keepalived

192.168.200.116 和 192.168.200.117 主機：

yum -y install ipvsadm kernel-devel openssl-devel keepalived

10.二、 master 主機設置

192.168.200.116 lvs master 主機配置：

[root@server06 ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   notification_email {
   route_id 192.168.200.116   
}
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}


vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }

    virtual_ipaddress {
        192.168.200.200
    }
}
virtual_server 192.168.200.200 3306 {
delay_loop 6
lb_algo wrr
lb_kind DR
protocol TCP
real_server 192.168.200.113 3306 {
weight 1
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 3306
}
}
real_server 192.168.200.114 3306 {
weight 1
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 3306
}
}
}

[root@server06 ~]# systemctl restart keepalived

檢查VIP地址：

[root@server06 ~]# ip a | grep ens33
5: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.200.116/24 brd 192.168.200.255 scope global noprefixroute ens33
    inet 192.168.200.200/32 scope global ens33

查看負載策略：

[root@server06 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.200.200:3306 wrr
  -> 192.168.200.113:3306         Route   1      0          0         
  -> 192.168.200.114:3306         Route   1      0          0

10.3 、192.168.200.117 backup 主機設置：

[root@server07 ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   notification_email {
   route_id 192.168.200.122   
}
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 51
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }   
    virtual_ipaddress {
        192.168.200.200
    }
}
virtual_server 192.168.200.200 3306 {
delay_loop 6
lb_algo wrr
lb_kind DR
protocol TCP
real_server 192.168.200.113 3306 {
weight 1
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 3306
}
}
real_server 192.168.200.114 3306 {
weight 1
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 3306
}
}
}

10.4 、爲數據庫節點設置 VIP 地址及配置 ARP 相關參數：

slave1(192.168.200.113)和 slave2(192.168.200.114)主機配置 realserver 腳本：

[root@server03 ~]# vim realserver.sh
#!/bin/bash
SNS_VIP=192.168.200.200
ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP

/sbin/route add -host $SNS_VIP dev lo:0
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p >/dev/null 2>&1

realserver1（192.168.200.113）檢測 VIP 地址：

[root@server03 ~]# ip a | grep lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 192.168.200.200/32 brd 192.168.200.200 scope global lo:0
    inet 192.168.200.113/24 brd 192.168.200.255 scope global noprefixroute ens33
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0

    遠程複製到 realserver2 主機上：192.168.200.114：

[root@server03 ~]# scp realserver.sh 192.168.200.114:/root/
realserver.sh 100% 378 183.1KB/s 00:00

realserver1（192.168.200.113）檢測 VIP 地址：

[root@server04 ~]# bash realserver.sh 
[root@server04 ~]# ip a | grep lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 192.168.200.200/32 brd 192.168.200.200 scope global lo:0
    inet 192.168.200.114/24 brd 192.168.200.255 scope global noprefixroute ens33
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0

10.5 、數據庫讀請求測試：

Server05(192.168.200.115)擔任 client 角色測試，鏈接 vip

[root@server05 ~]#  mysql -uroot -p123123 -h192.168.200.200 -P3306
Enter password:                 ###密碼爲123123
ERROR 1049 (42000): Unknown database '3306'
[root@server05 ~]#  mysql -uroot -p123123 -h192.168.200.200 -P3306
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 458
Server version: 5.5.60-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> exit
Bye


斷開後再鏈接測試一次爲了更好的看清請求分流狀況。。。。

[root@server05 ~]# mysql -uroot -p123123 -h192.168.200.200 -P3306
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 522
Server version: 5.5.64-MariaDB MariaDB Server

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]>

查看分流狀況：

[root@server06 ~]# ipvsadm -Lnc
IPVS connection entries
pro expire state       source             virtual            destination
TCP 00:54 FIN_WAIT 192.168.200.115:42558 192.168.200.200:3306 192.168.200.113:3306 TCP 13:56 ESTABLISHED 192.168.200.115:42560 192.168.200.200:3306 192.168.200.114:3306

查看此時111的VIP狀態: #能夠作一個週期計劃用於關閉防火牆機制（111-114-116-117）：

[root@server01 ~]# ip a | grep ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.200.111/24 brd 192.168.200.255 scope global noprefixroute ens33
    inet 192.168.200.100/24 brd 192.168.200.255 scope global secondary ens33:1