MHA+keepalived集羣環境搭建

整個MHA+keepalived集羣環境搭建

1.1. 環境簡介
1.1.一、vmvare虛擬機，系統版本CentOS6.5 x86_64位最小化安裝，mysql的版本5.7.21，
1.1.二、虛擬機器的ssh端口均爲默認22,
1.1.三、虛擬機的iptables所有關閉，
1.1.四、虛擬機的selinux所有關閉，
1.1.五、虛擬機服務器時間所有一致 ntpdate 0.asia.pool.ntp.org
1.1.六、3臺機器的ssh端口爲22

1.二、這次試驗採用的是3臺機器，機器具體部署以下：
角色             IP地址（內網）    主機名稱      節點機器部署服務                             業務用途

Monitor         192.168.52.250       db250         mha4mysql-manager-0.56-0.el6
Master          192.168.52.251       db251          mha4mysql-node-0.56-0.el6                      寫入(keepalived)

--------------------------------------------------------------------------------------------

slave(備master) 192.168.52.252      db252         mha4mysql-node-0.56-0.el6                  keepalived

Slave           192.168.52.253       db253          mha4mysql-node-0.56-0.el6      

讀+備份數據

1.3 說明介紹：

server03和server04是server02的slave從庫，複製環境搭建後面會簡單演示,其中master對外提供寫服務，備選master（實際的slave，主機名server03）提供讀服務，slave也提供相關的讀服務，一旦master宕機，將會把備 
選備master提高爲新的master，slave指向新的master
server04上部署Monitor（MHA Manager監控），主要是監控主從複製的集羣中主庫master是否正常，一旦master掛掉，MHA Manager會自動完成主庫和slave從庫的自動切換

1.4 安裝mysql

Mater和slave都須要安裝
wget http://dev.mysql.com/get/mysql57-community-release-el7-8.noarch.rpm
yum localinstall mysql57-community-release-el7-8.noarch.rpm
yum install mysql-community-server -y
systemctl enable mysqld
systemctl daemon-reload
systemctl start mysqld
systemctl status mysqld
grep 'temporary password' /var/log/mysqld.log
mysql -uroot -p
[root@server04 ~]# mysql -u root
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
service mysqld restart
192.168.52.251

vim /etc/my.cnf

[mysqld]

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

character-set-server=utf8

server-id=1

log-bin=master-log

relay-log=relay-log

innodb_file_per_table = ON

skip_name_resolve = ON

max_connections = 5000#不在配置文件中限定只讀，可是要記得在slave上限制只讀

 

symbolic-links=0

log-error=/var/log/mysqld.log

pid-file=/var/run/mysqld/mysqld.pid

192.168.52.252

vim /etc/my.cnf

[mysqld]

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

character-set-server=utf8

server-id=2

log-bin=master-log

relay-log=relay-log

relay_log_purge=0

read_only=1

skip_name_resolve=1

innodb_file_per_table=1

# Disabling symbolic-links is recommended to prevent assorted security risks

symbolic-links=0

 

log-error=/var/log/mysqld.log

pid-file=/var/run/mysqld/mysqld.pid

192.168.52.253

vim /etc/my.cnf

[mysqld]

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

character-set-server=utf8

 

server-id=3

log-bin=master-log

relay-log=relay-log

relay_log_purge=0

read_only=1

skip_name_resolve=1

innodb_file_per_table=1

# Disabling symbolic-links is recommended to prevent assorted security risks

symbolic-links=0

 

log-error=/var/log/mysqld.log

pid-file=/var/run/mysqld/mysqld.pid

 

第二：新建用戶repl_user設置密碼123456

      重置root密碼：123456

 

#更改密碼
mysql -u root
#更改密碼策略
set global validate_password_policy=0;
set global validate_password_length=4;
SET PASSWORD = PASSWORD('123456');
GRANT ALL PRIVILEGES ON *.* TO 'root'@'192.168.52.%' IDENTIFIED BY '123456' WITH GRANT OPTION;
flush privileges;
主從複製受權：

第三：在 3 個 mysql 節點作受權配置(主從複製受權)

Master:

grant replication slave,replication client on *.* to 'repluser'@'192.168.52.%' identified by '123456' ;

flush privileges;

#受權MHA管理用戶-mhaadmin

grant all on *.* to 'mhaadmin'@'192.168.52.%' identified by 'mhapass' ;

flush privileges;

mysql> show master status;

+-------------------+----------+--------------+------------------+-------------------+

| File              | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |

+-------------------+----------+--------------+------------------+-------------------+

| master-log.000005 |      154 |              |                  |                   |

+-------------------+----------+--------------+------------------+-------------------+

1 row in set (0.00 sec)

slave（兩個從庫）

 #配置主從複製起點

 change master to master_host='192.168.52.251',master_user='repluser',master_password='123456',master_log_file='master-log.000005',master_log_pos=154;

 start slave;

 show slave status\G

            Slave_IO_Running: Yes

            Slave_SQL_Running: Yes

mysql> set global read_only=1; #查看slave IO和slave sql是否都正常

#查看主從複製狀況

 show grants for 'repluser'@'192.168.52.%';

mysql> flush privileges;   #刷新權限

Query OK, 0 rows affected (0.00 sec)

刪除多餘用戶

mysql> drop user root@'localhost';

mysql> select user,host from mysql.user;

1.5 ssh授信

配置三臺機器的ssh互信（三臺都要操做）

     ssh-keygen -t rsa

     ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.52.250

     ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.52.251

     ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.52.252

     ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.52.253

 

#測試是否成功

    ssh 192.168.52.251 date

1.6 安裝MHA軟件

安裝MHA軟件（在三個節點上都裝mha的node軟件）

    #先安裝依賴

     wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

     rpm -ivh epel-release-latest-7.noarch.rpm

     yum install perl-DBD-MySQL perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager　-y

    下載軟件

     wget https://qiniu.wsfnk.com/mha4mysql-node-0.58-0.el7.centos.noarch.rpm

     rpm -ivh mha4mysql-node-0.58-0.el7.centos.noarch.rpm

僅在manager節點上安裝mha管理軟件(192.168.52.250)

     wget https://qiniu.wsfnk.com/mha4mysql-manager-0.58-0.el7.centos.noarch.rpm

     yum install perl-Parallel-ForkManager -y

    rpm -ivh mha4mysql-manager-0.58-0.el7.centos.noarch.rpm

    yum install mailx -y   #該軟件是用來發送郵件的

-rwxr-xr-x 1 root root  5172 Jan  7 14:09 masterha_secondary_check

-rwxr-xr-x 1 root root  1739 Jan  7 14:09 masterha_stop

-rwxr-xr-x 1 root root  8337 Jan  7 14:14 purge_relay_logs

-rwxr-xr-x 1 root root  7525 Jan  7 14:14 save_binary_logs

1.7 配置MHA軟件

192.168.52.250機器操做：

[root@db250 bin]# cd /usr/bin/

[root@db250 bin]# find ./ -name apply_diff_relay_logs

./apply_diff_relay_logs

[root@db250 bin]# cp /usr/bin/save_binary_logs /usr/local/bin/

[root@db250 bin]# cp /usr/bin/purge_relay_logs /usr/local/bin/

[root@db250 bin]# cp /usr/bin/filter_mysqlbinlog /usr/local/bin/

[root@db250 bin]# cp /usr/bin/apply_diff_relay_logs /usr/local/bin/

[root@db250 bin]# ln -s /usr/local/mysql/bin/mysql  /usr/bin/mysql

[root@db250 bin]# ln -s /usr/local/mysql/bin/mysqlbinlog  /usr/bin/mysqlbinlog

[root@db250 bin]# cd /usr/local/bin/

[root@db250 bin]# ll

total 88

-rwxr-xr-x 1 root root 17639 Jan  7 14:14 apply_diff_relay_logs

-rwxr-xr-x 1 root root  4807 Jan  7 14:14 filter_mysqlbinlog

-rwxr-xr-x 1 root root  1995 Jan  7 14:09 masterha_check_repl

-rwxr-xr-x 1 root root  1779 Jan  7 14:09 masterha_check_ssh

-rwxr-xr-x 1 root root  1865 Jan  7 14:09 masterha_check_status

-rwxr-xr-x 1 root root  3201 Jan  7 14:09 masterha_conf_host

-rwxr-xr-x 1 root root  2517 Jan  7 14:09 masterha_manager

-rwxr-xr-x 1 root root  2165 Jan  7 14:09 masterha_master_monitor

-rwxr-xr-x 1 root root  2373 Jan  7 14:09 masterha_master_switch

 

MHA配置文件以下

mkdir -p /etc/masterha
[root@db250 app1]# cat  /etc/masterha/app1.cnf

[server default]

manager_log=/var/log/masterha/app1/manager.log

manager_workdir=/var/log/masterha/app1.log

master_binlog_dir=/var/lib/mysql

master_ip_failover_script=/usr/local/bin/master_ip_failover

master_ip_online_change_script=/usr/local/bin/master_ip_online_change

password=123456

ping_interval=1

remote_workdir=/tmp

repl_password=123456

repl_user=repluser

secondary_check_script=/usr/local/bin/masterha_secondary_check -s db251 -s db252 --user=root --master_host=db252 --master_ip=192.168.52.252 --master_port=3306

shutdown_script=""

ssh_port=22

ssh_user=root

 

[server1]

hostname=192.168.52.251

candidate_master=1

port=3306

 

[server2]

candidate_master=1

check_repl_delay=0

hostname=192.168.52.252

port=3306

 

[server3]

hostname=192.168.52.253

port=3306

2、設置relay log的清除方式（在每一個slave節點上）：

三個節點服務器本地hosts解析

[root@db250 app1]# vim /etc/hosts

192.168.52.250 db250

192.168.52.251 db251

192.168.52.252 db252

192.168.52.253 db253

 

在slave master01 192.168.52.252操做：

[root@db251 ~]# mysql -uroot -p123456 -e "set global relay_log_purge=0"

mysql: [Warning] Using a password on the command line interface can be insecure.

 

在slave master02 192.168.52.253操做：

[root@ db252 ~]# mysql -uroot -p123456 -e "set global relay_log_purge=0"

mysql: [Warning] Using a password on the command line interface can be insecure.

 

注意：

MHA在發生切換的過程當中，從庫的恢復過程當中依賴於relay log的相關信息，因此這裏要將relay log的自動清除設置爲OFF，採用手動清除relay log的方式。在默認狀況下，從服務器上的中繼日誌會在SQL線程執行完畢後被自動刪除。可是在MHA環境中，這些中繼日誌在恢復其餘從服務器時可能會被用到，所以須要禁用中繼日誌的自動刪除功能。按期清除中繼日誌須要考慮到複製延時的問題。在ext3的文件系統下，刪除大的文件須要必定的時間，會致使嚴重的複製延時。爲了不復制延時，須要暫時爲中繼日誌建立硬連接，由於在linux系統中經過硬連接刪除大文件速度會很快。（在mysql數據庫中，刪除大表時，一般也採用創建硬連接的方式）

 

2.2設置按期清理relay腳本（兩臺slave服務器）：

[root@ db252~]# cat /data/scripts/purge_relay_log.sh

#!/bin/bash

user=root

passwd=123456

port=3306

log_dir='/data/masterha/log'

work_dir='/data'

purge='/usr/local/bin/purge_relay_logs'

 

if [ ! -d $log_dir ]

then

   mkdir $log_dir -p

fi

 

$purge --user=$user --password=$passwd --disable_relay_log_purge --port=$port --workdir=$work_dir >> $log_dir/purge_relay_logs.log 2>

 

腳本介紹：

--user mysql                      //用戶名

--password mysql                  //密碼

--port                            //端口號

--workdir                         //指定建立relay log的硬連接的位置，默認是/var/tmp，因爲系統不一樣分區建立硬連接文件會失敗，故須要執行硬連接具體位置，成功執行腳本後，硬連接的中繼日誌文件被刪除

--disable_relay_log_purge         //默認狀況下，若是relay_log_purge=1，腳本會什麼都不清理，自動退出，經過設定這個參數，當relay_log_purge=1的狀況下會將relay_log_purge設置爲0。清理relay log以後，最後將參數設置爲OFF。

 

purge_relay_logs腳本刪除中繼日誌不會阻塞SQL線程。下面咱們手動執行看看什麼狀況:

[root@ db252 ~]# purge_relay_logs --user=root --password=123456 --port=3306 -disable_relay_log_purge --workdir=/data/

2018-07-01 11:53:16: purge_relay_logs script started.

 Found relay_log.info: /data/mysql/relay-log.info

 Opening /data/mysql/logs/relay-log/relay-bin.000001 ..

 Opening /data/mysql/logs/relay-log/relay-bin.000002 ..

 Executing SET GLOBAL relay_log_purge=1; FLUSH LOGS; sleeping a few seconds so that SQL thread can delete older relay log files (if it keeps up); SET GLOBAL relay_log_purge=0; .. ok.

2018-07-01 11:53:20: All relay log purging operations succeeded.

 

主從failover腳本

[root@db250 app1]#  cat /usr/local/bin/master_ip_failover

#!/usr/bin/env perl

 

#Copyright (C) 2011 DeNA Co.,Ltd.

 

#This program is free software; you can redistribute it and/or modify

#t under the terms of the GNU General Public License as published by

#the Free Software Foundation; either version 2 of the License, or

#(at your option) any later version.

 

#This program is distributed in the hope that it will be useful,

#but WITHOUT ANY WARRANTY; without even the implied warranty of

#MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

#GNU General Public License for more details.

#You should have received a copy of the GNU General Public License

#along with this program; if not, write to the Free Software

#Foundation, Inc.,

#51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA

 

##Note: This is a sample script and is not complete. Modify the script based on your environment.

######################################################

use strict;

use warnings FATAL => 'all';

 

use Getopt::Long;

use MHA::DBHelper;

 

my (

  $command,        $ssh_user,         $orig_master_host,

  $orig_master_ip, $orig_master_port, $new_master_host,

  $new_master_ip,  $new_master_port,  $new_master_user,

  $new_master_password

);

my $vip = '192.168.52.199';

my $ssh_start_vip = "systemctl start keepalived";

my $ssh_stop_vip = "systemctl stop keepalived ";

 

GetOptions(

  'command=s'             => \$command,

  'ssh_user=s'            => \$ssh_user,

  'orig_master_host=s'    => \$orig_master_host,

  'orig_master_ip=s'      => \$orig_master_ip,

  'orig_master_port=i'    => \$orig_master_port,

  'new_master_host=s'     => \$new_master_host,

  'new_master_ip=s'       => \$new_master_ip,

  'new_master_port=i'     => \$new_master_port,

  'new_master_user=s'     => \$new_master_user,

  'new_master_password=s' => \$new_master_password,

);

 

exit &main();

 

sub main {

 

    print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";

 

  if ( $command eq "stop" || $command eq "stopssh" ) {

 

    #$orig_master_host, $orig_master_ip, $orig_master_port are passed.

    #If you manage master ip address at global catalog database,

    #invalidate orig_master_ip here.

    my $exit_code = 1;

    eval {

      print "Disabling the VIP on old master: $orig_master_host \n";

      &stop_vip();

      #updating global catalog, etc

      $exit_code = 0;

    };

    if ($@) {

      warn "Got Error: $@\n";

      exit $exit_code;

    }

    exit $exit_code;

  }

  elsif ( $command eq "start" ) {

 

    #all arguments are passed.

    #If you manage master ip address at global catalog database,

    #activate new_master_ip here.

    #You can also grant write access (create user, set read_only=0, etc) here.

    my $exit_code = 10;

    eval {

      print "Enabling the VIP - $vip on the new master - $new_master_host \n";

      &start_vip();

      $exit_code = 0;

        };

    if ($@) {

      warn $@;

 

      #If you want to continue failover, exit 10.

      exit $exit_code;

    }

    exit $exit_code;

  }

  elsif ( $command eq "status" ) {

     print "Checking the Status of the script.. OK \n";

    #do nothing

    exit 0;

  }

  else {

    &usage();

    exit 1;

  }

}

 

sub start_vip() {

    `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;

}

 

#A simple system call that disable the VIP on the old_master

sub stop_vip() {

    `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;

    }

 

sub usage {

  print

"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";

}

 

chmod +x /usr/local/bin/master_ip_failover

1.8 MHA相關測試

測試SSH免密碼登陸

masterha_check_ssh --conf=/etc/masterha/app1.cnf

 

測試MHA數據庫同步

masterha_check_repl --conf=/etc/masterha/app1.cnf

chmod +x /usr/local/bin/masterha_check_repl
啓動MHA監控服務

 

查看MHA Manager監控是否正常：

masterha_check_status --conf=/etc/masterha/app1.cnf
 

關閉MHA Manage監控

masterha_stop --conf=/etc/masterha/app1.cnf
1.9 啓動mha數據監控

masterha_check_status --conf=/etc/masterha/app1.cnf

mkdir -p  /var/log/masterha/app1/
 

nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 &
ps -ef|grep perl
1.10 配置vip實現MHA架構中主庫故障自動切換

192.168.52.251和192.168.52.252

Yum install keepalived -y

 

192.168.52.251

[root@db251 ~]# vim /etc/keepalived/keepalived.conf

global_defs {

   notification_email {

   305xxx7536@qq.com

   }

   notification_email_from Alexandre.Cassen@firewall.loc

   smtp_server 192.168.52.251

   smtp_connect_timeout 30

   router_id LVS_01

}

 

vrrp_instance VI_1 {

    #state MASTER

    state BACKUP

    interface ens33

    virtual_router_id 51

    priority 100

    advert_int 1

    nopreempt

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

    192.168.2.199/24

    }

}

 

systemctl start keepalived

 

192.168.52.252

[root@db252 ~]# vim /etc/keepalived/keepalived.conf

 

global_defs {

   notification_email {

   305xxx7536@qq.com

   }

   notification_email_from Alexandre.Cassen@firewall.loc

   smtp_server 192.168.52.252

   smtp_connect_timeout 30

   router_id LVS_01

}

 

vrrp_instance VI_1 {

    #state MASTER

    state BACKUP

    interface ens33

    virtual_router_id 51

    priority 90

    advert_int 1

    nopreempt

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

    192.168.52.199/24

    }

}

 

systemctl start keepalived

#####特別注意！！！！！
上面兩臺服務器的keepalived都設置爲了BACKUP模式，在keepalived中2種模式，分別是master->backup模式和backup->backup模式。這兩種模式有很大區別。在master->backup模式下，一旦主庫宕機，虛擬ip會自動漂移到從庫，當主庫修復後，keepalived啓動後，還會把虛擬ip搶佔過來，即便設置了非搶佔模式（nopreempt）搶佔ip的動做也會發生。在backup->backup模式下，當主庫宕機後虛擬ip會自動漂移到從庫上，當原主庫恢復和keepalived服務啓動後，並不會搶佔新主的虛擬ip，即便是優先級高於從庫的優先級別，也不會發生搶佔。爲了減小ip漂移次數，一般是把修復好的主庫當作新的備庫。
++++到此處MHA架構中keepalived服務安裝配置完成++++

1.11檢查故障切換後MHA集羣相關服務的變化
配置文件/etc/masterha/app1.cnf變化

一、在db250 192.168.52.250管理節點查看一下配置文件/etc/masterha/app1.cnf能夠發現[server1]的內容已經被自動去掉了：

二、masterha_manager 服務自動退出

三、源master192.168.52.251機器上keepalived服務被停掉了

 

#######重要提示！！！ 
當db251 192.168.52.251 機器上的mysql掛掉後，db252 192.168.52.252機器提高爲master時，192.168.52.251 機器上的keepalived會停掉，而 192.168.52.252機器的keepalived會開啓，VIP票到199機器上。
此時須要重啓192.168.52.251上的mysql,通常都是要恢復它做爲252新主的從庫，此時192.168.52.251機器上的keepalived千萬不要開啓，由於開啓keepalived，會搶佔252機器上的VIP,致使程序鏈接數據庫出現混亂。同時192.168.52.251機器和192.168.52.252機器上的keepalived服務不要設置爲開機自啓動

1.12 mha高可用從新加入監控

一、出問題的master:show master status;Last_IO_Error: Fatal error: The slave I/O thread stops because master and slave have equal MySQL server UUIDs; these UUIDs must be different for replication to work.二、解決方法：rm -rf /var/lib/mysql/auto.cnfsystemctl restart mysqld三、數據庫從庫同步主庫 show master status； #配置主從複製起點change master to master_host='192.168.52.251',master_user='repluser',master_password='123456',master_log_file='master-log.000004',master_log_pos=154; #注意masterIP地址 start slave; show slave status\G四、切換完畢後缺乏secondary_check_script=/usr/local/bin/masterha_secondary_check -s db251 -s db252 --user=root --master_host=db252 --master_ip=192.168.52.252 --master_port=3306添加：[server1]hostname=192.168.52.252candidate_master=1port=3306五、測試測試sshmasterha_check_ssh --conf=/etc/masterha/app1.cnf測試MHA數據庫同步，並自動啓動masterha_check_repl --conf=/etc/masterha/app1.cnf查看MHA Manager監控是否正常：masterha_check_status --conf=/etc/masterha/app1.cnf六、日誌查看tailf /var/log/masterha/app1/manager.log七、開啓MHA Manager監控nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 &八、查詢manger程序ps -ef|grep perl--------------------- 做者：guoshaoliang789 來源：CSDN 原文：https://blog.csdn.net/guoshaoliang789/article/details/86086181 版權聲明：本文爲博主原創文章，轉載請附上博文連接！