MySQL高可用之MHA（二）

時間 2020-07-09

標籤 mysql 用之 mha 欄目 MySQL 简体版

原文原文鏈接

注：本文基於MySQL高可用之MHA
配置VIPphp

vip配置能夠採用兩種方式：
一、經過keepalived的方式管理虛擬ip的浮動；
二、經過腳本方式啓動虛擬ip 的方式（即不須要keepalived或者heartbeat相似的軟件）mysql

一、keepalived方式管理虛擬ipsql

#在編譯安裝 Keepalived以前，必須先安裝內核開發包kernel-devel以及openssl-devel、popt-devel等支持庫
[root@master ~]# yum -y install kernel-devel popt-devel openssl-devel 
#在兩臺master上進行安裝
[root@master ~]# wget https://www.keepalived.org/software/keepalived-2.1.3.tar.gz
[root@master ~]# tar zxf keepalived-2.1.3.tar.gz 
[root@master ~]# cd keepalived-2.1.3/
[root@master keepalived-2.1.3]# ./configure --prefix=/ && make && make install 
#修改配置文件
[root@master ~]# vim /etc/keepalived/keepalived.conf 
! Configuration File for keepalived

global_defs {
   router_id mysql-ha1
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 51
    priority 100
    nopreempt
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.171.250
    }
}
[root@master ~]# scp /etc/keepalived/keepalived.conf root@192.168.171.152:/etc/keepalived/
#master2配置修改
[root@slave1 ~]# vim /etc/keepalived/keepalived.conf 
! Configuration File for keepalived

global_defs {
   router_id mysql-ha2
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 51
    priority 50
    nopreempt
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.171.250
    }
}
[root@master ~]# systemctl start keepalived         # 啓動服務
[root@master ~]# ip a | grep ens33         # 檢測IP  能夠發現 VIP已經搶佔
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    inet 192.168.171.151/24 brd 192.168.171.255 scope global ens33
    inet 192.168.171.250/32 scope global ens33

注意：上面兩臺服務器的keepalived都設置爲了BACKUP模式，在keepalived中2種模式，分別是master>backup模式和backup->backup模式。這兩種模式有很大區別。在master->backup模式下，一旦主庫宕機，虛擬ip會自動漂移到從庫，當主庫修復後，keepalived啓動後，還會把虛擬ip搶佔過來，即便設置了非搶佔模式（nopreempt）搶佔ip的動做也會發生。在backup->backup模式下，當主庫宕機後虛擬ip 會自動漂移到從庫上，當原主庫恢復和keepalived服務啓動後，並不會搶佔新主的虛擬ip，即便是優先級高於從庫的優先級別，也不會發生搶佔。爲了減小ip漂移次數，一般是把修復好的主庫當作新的備庫。數據庫

二、MHA引入keepalived（MySQL服務進程掛掉時經過MHA 中止keepalived）
要想把keepalived服務引入 MHA，咱們只須要修改切換時觸發的腳本文件master_ip_failover便可，在該腳本中添加在master發生宕機時對 keepalived的處理vim

#編輯腳本，修改以下
[root@manager ~]# vim /scripts/master_ip_failover 
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';

use Getopt::Long;

my (
    $command,$ssh_user,$orig_master_host,$orig_master_ip,$orig_master_port,
$new_master_host,$new_master_ip,$new_master_port
);
my $vip = '192.168.171.250';
my $ssh_start_vip = "systemctl start keepalived.service";
my $ssh_stop_vip = "systemctl stop keepalived.service";
GetOptions(
    'command=s'          => \$command,
    'ssh_user=s'         => \$ssh_user,
    'orig_master_host=s' => \$orig_master_host,
    'orig_master_ip=s'   => \$orig_master_ip,
    'orig_master_port=i' => \$orig_master_port,
    'new_master_host=s'  => \$new_master_host,
    'new_master_ip=s'    => \$new_master_ip,
    'new_master_port=i'  => \$new_master_port,
);
exit &main();

sub main {

    print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";

    if ( $command eq "stop" || $command eq "stopssh" ) {

        my $exit_code = 1;
        eval {
            print "Disabling the VIP on old master: $orig_master_host \n";
            &stop_vip();
            $exit_code = 0;
        };
        if ($@) {
            warn "Got Error: $@\n";
                                    exit $exit_code;
        }
        exit $exit_code;
    }
    elsif ( $command eq "start" ) {

        my $exit_code = 10;
        eval {
            print "Enabling the VIP - $vip on the new master - $new_master_host \n";
            &start_vip();
            $exit_code = 0;
        };
        if ($@) {
            warn $@;
            exit $exit_code;
        }
        exit $exit_code;
    }
    elsif ( $command eq "status" ) {
        print "Checking the Status of the script.. OK \n";
        #`ssh $ssh_user\@cluster1 \" $ssh_start_vip \"`; 
                exit 0;
    }
    else {
        &usage();
        exit 1;
    }
}
# A simple system call that enable the VIP on the new master 
sub start_vip() {
    `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master 
sub stop_vip() {
     return 0  unless  ($ssh_user);
    `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
sub usage {
    print
    "Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}
#在配置文件中添加以下內容
[root@manager ~]# vim /etc/masterha/app1.cnf       # 在[server default]下面添加
master_ip_failover_script=/scripts/master_ip_failover
[root@manager ~]# masterha_stop --conf=/etc/masterha/app1.cnf                # 中止MHA服務
Stopped app1 successfully.
[root@manager ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf &>/tmp/mha_manager.log  &          # 再啓動
[root@manager ~]# masterha_check_status --conf=/etc/masterha/app1.cnf                      # 查看狀態
app1 (pid:8560) is running(0:PING_OK), master:192.168.171.151

測試：關閉master1，模擬宕機安全

#slave上查看，已經轉移主
mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Reconnecting after a failed master event read
                  Master_Host: 192.168.171.152
                  Master_User: mharep
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000006
          Read_Master_Log_Pos: 154
               Relay_Log_File: relay-bin.000002
                Relay_Log_Pos: 320
        Relay_Master_Log_File: mysql-bin.000006
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
 #查看VIP綁定
 [root@slave1 ~]# ip a |grep ens33             # 能夠看到VIP地址已經漂移到了master2上
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    inet 192.168.171.152/24 brd 192.168.171.255 scope global ens33
    inet 192.168.171.250/32 scope global ens33

主從切換後續工做--重構：重構就是你的主掛了，切換到Candicate master上，Candicate master變成了主，所以重構的一種方案原主庫修復成一個新的slave 主庫切換後，把原主庫修復成新從庫，原主庫數據文件完整的狀況下，可經過如下方式找出最後執行的CHANGE MASTER命令：服務器

[root@manager ~]#  grep "CHANGE MASTER TO MASTER"  /masterha/app1/manager.log | tail -1 
Fri Jul  3 09:38:31 2020 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.171.152', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=746, MASTER_USER='mharep', MASTER_PASSWORD='xxx';

將原主庫修復成從庫架構

[root@master ~]# systemctl start mysqld
mysql> change master to master_host='192.168.171.152',master_port=3306,master_log_file='mysql-bin.000001',master_log_pos=746,master_user='mharep',master_password='123';
mysql> start slave;
mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.171.152
                  Master_User: mharep
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000001
          Read_Master_Log_Pos: 746
               Relay_Log_File: relay-bin.000002
                Relay_Log_Pos: 320
        Relay_Master_Log_File: mysql-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
[root@master ~]# systemctl start keepalived       # 將Keepalived啓動
#啓動mha manager
[root@manager ~]# rm -rf /masterha/app1/app1.failover.complete           # 首先須要將這個failover刪除掉，要不啓動不了
[root@manager ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf  --ignore_fail_on_start &>/tmp/mha_manager.log  & 
[root@manager ~]# masterha_check_status --conf=/etc/masterha/app1.cnf 
app1 (pid:49782) is running(0:PING_OK), master:192.168.171.152
[root@manager ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf 
#而後這樣就恢復原樣了，當備master宕機後，原主master會再次啓動搶佔VIP

二、經過腳本實現VIP切換
修改/scripts/master_ip_failover，也可使用其餘的語言完成，好比php語言。使用php腳本編寫的failover這裏就不介紹了。修改完成後內容以下：app

#若是使用腳本管理vip的話，須要 手動在master服務器上綁定一個vip
[root@master ~]# ifconfig ens33:0 192.168.171.250/24
[root@master ~]# ip a | grep ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    inet 192.168.171.151/24 brd 192.168.171.255 scope global ens33
    inet 192.168.171.250/24 brd 192.168.171.255 scope global secondary ens33:0
#修改/scripts/ master_ip_failover
[root@manager ~]# vim /scripts/master_ip_failover 
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';

use Getopt::Long;

my (
    $command,$ssh_user,$orig_master_host,$orig_master_ip,$orig_master_port,
$new_master_host,$new_master_ip,$new_master_port
);
my $vip = '192.168.171.250';
my $key = '0';
my $ssh_start_vip = "/sbin/ifconfig ens33:$key $vip";               # 照以前的腳本就修改了幾個變量，上下
my $ssh_stop_vip = "/sbin/ifconfig ens33:$key down";
GetOptions(
    'command=s'          => \$command,
    'ssh_user=s'         => \$ssh_user,
    'orig_master_host=s' => \$orig_master_host,
    'orig_master_ip=s'   => \$orig_master_ip,
    'orig_master_port=i' => \$orig_master_port,
    'new_master_host=s'  => \$new_master_host,
    'new_master_ip=s'    => \$new_master_ip,
    'new_master_port=i'  => \$new_master_port,
);
exit &main();

sub main {

    print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";

    if ( $command eq "stop" || $command eq "stopssh" ) {

        my $exit_code = 1;
        eval {
            print "Disabling the VIP on old master: $orig_master_host \n";
            &stop_vip();
            $exit_code = 0;
        };
        if ($@) {
            warn "Got Error: $@\n";
                                    exit $exit_code;
        }
        exit $exit_code;
    }
    elsif ( $command eq "start" ) {

        my $exit_code = 10;
        eval {
            print "Enabling the VIP - $vip on the new master - $new_master_host \n";
            &start_vip();
            $exit_code = 0;
        };
        if ($@) {
            warn $@;
            exit $exit_code;
        }
        exit $exit_code;
    }
    elsif ( $command eq "status" ) {
        print "Checking the Status of the script.. OK \n";
        #`ssh $ssh_user\@cluster1 \" $ssh_start_vip \"`; 
                        exit 0;
                            }
                                else {
                                        &usage();
                                                exit 1;
                                                    }
                                                    }
                                                    # A simple system call that enable the VIP on the new master 
                                                    sub start_vip() {
                                                        `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
                                                        }
                                                        # A simple system call that disable the VIP on the old_master 
                                                        sub stop_vip() {
                                                            return 0  unless  ($ssh_user);
                                                                 `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
                                                                 }
                                                                 sub usage {
                                                                     print
                                                                         "Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
                                                                         }
[root@manager ~]# grep "master_ip_failover_script" /etc/masterha/app1.cnf 
master_ip_failover_script=/scripts/master_ip_failover
#在/etc/masterha/app1.cnf 填入上方返回內容，其實在以前已經添加過了，這裏提示只作了腳本的人切記添加

中止MHA：less

[root@manager ~]# masterha_stop  --conf=/etc/masterha/app1.cnf

啓動MHA

[root@manager ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf &>/tmp/mha_manager.log  & 
[root@manager ~]# masterha_check_status --conf=/etc/masterha/app1.cnf 
app1 (pid:50564) is running(0:PING_OK), master:192.168.171.151
#再檢查集羣狀態，看是否會報錯
[root@manager ~]# masterha_check_repl  --conf=/etc/masterha/app1.cnf

測試：在master上停掉mysql服務

#在slave主機上查看狀態
mysql> show slave status\G         # 發現已經變爲了備主IP
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.171.152
                  Master_User: mharep
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000002
          Read_Master_Log_Pos: 154
               Relay_Log_File: relay-bin.000002
                Relay_Log_Pos: 320
        Relay_Master_Log_File: mysql-bin.000002
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
#在備主上查看IP
[root@slave1 ~]# ip a | grep ens33       # 能夠看到VIP已經漂移過來了
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    inet 192.168.171.152/24 brd 192.168.171.255 scope global ens33
    inet 192.168.171.250/24 brd 192.168.171.255 scope global secondary ens33:0

注：爲了防止腦裂發生，推薦生產環境採用腳本的方式來管理虛擬ip，而不是使用keepalived來完成。
總結：

MHA軟件由兩部分組成，Manager工具包和Node工具包，具體的說明以下：
Manager工具包主要包括如下幾個工具：
masterha_check_ssh 檢查MHA的SSH配置情況
masterha_check_repl 檢查MySQL複製情況
masterha_manger 啓動MHA
masterha_check_status 檢測當前MHA運行狀態
masterha_master_monitor 檢測 master是否宕機
masterha_master_switch 控制故障轉移（自動或者手動）
masterha_conf_host 添加或刪除配置的server信息

Node工具包（這些工具一般由MHA Manager的腳本觸發，無需人爲操做）主要包括如下幾個工具：
save_binary_logs 保存和複製master的二進制日誌
apply_diff_relay_logs 識別差別的中繼日誌事件並將其差異的事件應用於其餘的slave
filter_mysqlbinlog 去除沒必要要的ROLLBACK事件（MHA已再也不使用這個工具）
purge_relay_logs 清除中繼日誌（不會阻塞SQL線程）

mysql必備技能掌握：一、MySQL架構:對mysql的架構，總體有個印象，才能不斷的加深對mysql的理解和後繼的學習。二、用各類姿式備份MySQL數據庫數據備份是DBA或運維工程師平常工做之一，若是讓你來備份，你會用什麼方式備份，在時間時間備份，使用什麼策略備份三、mysql主從複製及讀寫分離 mysql的主從複製及讀寫分離是DBA必備技能之一四、MySQL/MariaDB數據庫基於SSL實現主從複製增強主從複製的安全性五、 MySQL高可用數據的高可用如何保證六、數據庫Sharding的基本思想和切分策略隨着數據量的不斷攀升，從性能和可維護的角度，須要進行一些Sharding，也就是數據庫的切分，有垂直切分和水平切分七、 MySQL/MariaDB 性能調整和優化技巧掌握優化思路和技巧，對數據庫的不斷優化是一項長期工程