MySQL高可用架構之MHA

該文章轉自: https://www.cloudbility.com/club/7104.htmlhtml

 

目錄

  •  1、當前高可用方案
    • 一、Heartbeat+DRBD
    • 二、MySQL Cluster
    • 三、全局事務ID
    • 四、PXC
    • 五、MHA的優點
      • 1)故障切換快
      • 2)master故障不會致使數據不一致
      • 3)無需修改當前的MySQL設置
      • 4)無需增長大量的服務器
      • 5)無性能降低
      • 6)適用於任何存儲引擎
  • 2、MHA簡介:
    • 一、MHA結構
      • 1)MHA Manager
        • 1.Manager工具包主要工具
      • 2)MHA Node
        • 1.Node工具包
      • 3)注意
    • 二、MAH工做原理
  • 3、部署MHA
    • 一、環境準備
    • 二、安裝epel源
    • 三、環境初始化
      • 1)修改每臺主機名
      • 2)主機名解析
      • 3)ssh無密碼登陸
  • 4、規劃mysql
    • 1)安裝mysql
    • 2)配置master、slave01和slave02之間的主從複製
    • 3)在master、slave01上建立主從同步的帳號。
    • 4)在master上執行命令,查看master狀態信息
    • 5)在slave01和slave02上執行主從同步
  • 5、規劃mha
    • 1)建立mha管理用的複製帳號
    • 2)在3臺主機上(master、slave01和slave02)上分別安裝mha4mysql-node包
    • 3)在manager上安裝mha4mysql-manager和mha4mysql-node包
    • 4)修改manager端mha的配置文件
    • 5)檢查ssh是否暢通
    • 6)masterha_check_repl工具檢查mysql主從複製是否成功
  • 6、mha實驗模擬
    • 1)在每次作mha實驗的時候,咱們都最好先執行以下命令作檢測
    • 2)在manager端啓動mha服務並時刻監控日誌文件的輸出變化
    • 3)測試master宕機後會自動切換
    • 4)恢復master服務
    • 5)再次啓動MHA的manager服務,並中止slave01
    • 6)恢復slave01服務
    • 7)重啓MHA的manager服務
  • 7、經過vip實現mysql的高可用
    • 1)修改/usr/local/mha/mha.cnf
    • 2)修改腳本/usr/local/mha/scripts/master_ip_failover
    • 3)模擬故障進行切換
  • 8、MHA平常維護命令
    • 一、查看ssh登錄是否成功
    • 二、查看複製是否創建好
    • 三、啓動mha
    • 四、檢查啓動的狀態
    • 五、中止mha
    • 六、failover後下次重啓
  • 9、FAQ(常見問題解答)
    • 一、可能報錯1
    • 二、可能報錯2
    • 三、可能報錯3
    • 四、可能報錯4
    • 五、可能報錯5
    • 六、小知識
    • 1、當前高可用方案

      一、Heartbeat+DRBD

      開銷:須要額外添加處於被動狀態的master server(並不處理應用流量) 性能:爲了實現DRBD複製環境的高可用,innodb-flush-log-at-trx-commit和sync-binlog必須設置爲1,這樣會致使寫性能降低。node

      一致性:在master上必要的binlog時間可能會丟失,這樣slave就沒法進行復制,致使產生數據一致性問題。mysql

      二、MySQL Cluster

      MySQL Cluster真正實現了高可用,可是使用的是NDB存儲引擎,而且SQL節點有單點故障問題。linux

      半同步複製(5.5+) 半同步複製大大減小了「binlog events只存在故障master上」的問題。sql

      在提交時,保證至少一個slave(並非全部的)接收到binlog,所以一些slave可能沒有接收到binlog。shell

      三、全局事務ID

      在二進制文件中添加全局事務ID(global transaction id)須要更改binlog格式,在5.1/5.5版本中不支持。數據庫

      在應用方面有不少方法能夠直線全局事務ID,可是仍避免不了複雜度、性能、數據丟失或者一致性的問題。vim

      四、PXC

      PXC實現了服務高可用,數據同步時是併發複製。可是僅支持InnoDB引擎,全部的表都要有主鍵。鎖衝突、死鎖問題相對較多等等問題。api

      五、MHA的優點

      1)故障切換快

      在主從複製集羣中,只要從庫在複製上沒有延遲,MHA一般能夠在數秒內實現故障切換。9-10秒內檢查到master故障,能夠選擇在7-10秒關閉master以免出現裂腦,幾秒鐘內,將差別中繼日誌(relay log)應用到新的master上,所以總的宕機時間一般爲10-30秒。恢復新的master後,MHA並行的恢復其他的slave。即便在有數萬臺slave,也不會影響master的恢復時間。緩存

      2)master故障不會致使數據不一致

      當目前的master出現故障是,MHA自動識別slave之間中繼日誌(relay log)的不一樣,並應用到全部的slave中。這樣全部的salve可以保持同步,只要全部的slave處於存活狀態。和Semi-Synchronous Replication一塊兒使用,(幾乎)能夠保證沒有數據丟失。

      3)無需修改當前的MySQL設置

      MHA的設計的重要原則之一就是儘量地簡單易用。MHA工做在傳統的MySQL版本5.0和以後版本的主從複製環境中。和其它高可用解決方法比,MHA並不須要改變MySQL的部署環境。MHA適用於異步和半同步的主從複製。

      啓動/中止/升級/降級/安裝/卸載MHA不須要改變(包擴啓動/中止)MySQL複製。當須要升級MHA到新的版本,不須要中止MySQL,僅僅替換到新版本的MHA,而後重啓MHA Manager就行了。

      MHA運行在MySQL 5.0開始的原生版本上。一些其它的MySQL高可用解決方案須要特定的版本(好比MySQL集羣、帶全局事務ID的MySQL等等),但並不只僅爲了master的高可用才遷移應用的。在大多數狀況下,已經部署了比較舊MySQL應用,而且不想僅僅爲了實現Master的高可用,花太多的時間遷移到不一樣的存儲引擎或更新的前沿發行版。MHA工做的包括5.0/5.1/5.5的原生版本的MySQL上,因此並不須要遷移。

      4)無需增長大量的服務器

      MHA由MHA Manager和MHA Node組成。

      MHA Node運行在須要故障切換/恢復的MySQL服務器上,所以並不須要額外增長服務器。

      MHA Manager運行在特定的服務器上,所以須要增長一臺(實現高可用須要2臺),可是MHA Manager能夠監控大量(甚至上百臺)單獨的master,所以,並不須要增長大量的服務器。即便在一臺slave上運行MHA Manager也是能夠的。綜上,實現MHA並沒用額外增長大量的服務。

      5)無性能降低

      MHA適用與異步或半同步的MySQL複製。監控master時,MHA僅僅是每隔幾秒(默認是3秒)發送一個ping包,並不發送重查詢。能夠獲得像原生MySQL複製同樣快的性能。

      6)適用於任何存儲引擎

      MHA能夠運行在只要MySQL複製運行的存儲引擎上,並不只限制於InnoDB,即便在不易遷移的傳統的MyISAM引擎環境,同樣可使用MHA。

    • 2、MHA簡介:

      MHA(Master High Availability),是比較成熟的MySQL高可用方案。MHA可以在30秒內實現故障切換,並能在故障切換中,最大可能的保證數據一致性。目前淘寶也正在開發類似產品TMHA,目前已支持一主一從。 image

      一、MHA結構

      該軟件由兩部分組成:MHA Manager(管理節點)和MHA Node(數據節點)。

      1)MHA Manager

      能夠單獨部署在一臺獨立的機器上管理多個master-slave集羣,也能夠部署在一臺slave節點上。MHA Manager主要運行一些工具,好比masterha_manager工具實現自動監控MySQL Master和實現master故障切換,其它工具實現手動實現master故障切換、在線master轉移、鏈接檢查等等。

      1.Manager工具包主要工具

      masterha_check_ssh 檢查MHA的SSH配置情況 masterha_check_repl 檢查MySQL複製情況 masterha_manger 啓動MHA masterha_check_status 檢測當前MHA運行狀態 masterha_master_monitor 檢測master是否宕機 masterha_master_switch 控制故障轉移(自動或者手動) masterha_conf_host 添加或刪除配置的server信息 

      2)MHA Node

      MHA Node 運行在每臺MySQL服務器上MHA Manager會定時探測集羣中的master節點,當master出現故障時,它能夠自動將最新數據的slave提高爲新的master,而後將全部其餘的slave從新指向新的master。整個故障轉移過程對應用程序徹底透明。

      部署在全部運行MySQL的服務器上,不管是master仍是slave。主要做用有三個。

      Ⅰ、保存二進制日誌 若是可以訪問故障master,會拷貝master的二進制日誌

      II、應用差別中繼日誌 從擁有最新數據的slave上生成差別中繼日誌,而後應用差別日誌。

      III、清除中繼日誌 在不中止SQL線程的狀況下刪除中繼日誌

      1.Node工具包

      這些工具一般由MHA Manager的腳本觸發,無需人爲操做主要包括如下幾個工具:

      save_binary_logs                保存和複製master的二進制日誌 apply_diff_relay_logs 識別差別的中繼日誌事件並將其差別的事件應用於其餘的slave filter_mysqlbinlog 去除沒必要要的ROLLBACK事件(MHA已再也不使用這個工具) purge_relay_logs 清除中繼日誌(不會阻塞SQL線程) 

      3)注意

      爲了儘量的減小主庫硬件損壞宕機形成的數據丟失,所以在配置MHA的同時建議配置成MySQL 5.5的半同步複製。關於半同步複製原理各位本身進行查閱。(不是必須)

      二、MAH工做原理

      1.從宕機崩潰的Master保存二進制日誌事件(binlog event);

      2.識別含有最新更新的Slave;

      3.應用差別的中繼日誌(relay log)到其餘Slave;

      4.應用從Master保存的二進制日誌事件;

      5.提高一個Slave爲新的Master;

      6.使其餘的Slave鏈接新的Master進行復制;

    • 3、部署MHA

      一、環境準備

      [root@server01 ~]# cat /etc/redhat-release CentOS release 6.8 (Final) [root@server01 ~]# uname -r 2.6.32-642.el6.x86_64 

      二、安裝epel源

      全部節點

      #備份 mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup #下載epel源 wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo #生成緩存 yum makecache 

      三、環境初始化

      1)修改每臺主機名

      172.16.1.241 master 172.16.1.242 slave01 172.16.1.243 slave02 172.16.1.244 manager 

      其中master對外提供寫服務,備選master(實際的slave,主機名slave01)提供讀服務,slave也提供相關的讀服務,一旦master宕機,將會把備選master提高爲新的master,slave指向新的master。

      2)主機名解析

      #每臺服務器執行修改主機名解析

      echo '''
      172.16.1.241 master 172.16.1.242 slave01 172.16.1.243 slave02 172.16.1.244 manager''' >>/etc/hosts 

      3)ssh無密碼登陸

      使用key登陸,工做中經常使用,服務器之間無需密碼驗證的。關於配置使用key登陸,一點須要注意:不能禁止 password 登錄,不然會出現錯誤

      注意:因此所有機器都要相互作密鑰登陸。服務器間,無密碼ssh登陸 #主機:master執行命令

      [root@master ~]# ssh-keygen -t rsa [root@master ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@manager [root@master ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave01 [root@master ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave02 

      #主機:slave01執行命令

      [root@slave01 ~]# ssh-keygen -t rsa [root@slave01 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@manager [root@slave01 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@master [root@slave01 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave02 

      #主機: slave02執行命令

      [root@slave02 ~]# ssh-keygen -t rsa [root@slave02 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@manager [root@slave02 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@master [root@slave02 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave01 

      #主機:manager執行命令

      [root@manager ~]# ssh-keygen -t rsa [root@manager ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@master [root@manager ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave01 [root@manager ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave02
    • 4、規劃mysql

      1)安裝mysql

      #master配置文件/etc/my.cnf 核心配置以下:

      basedir = /application/mysql
      datadir = /application/mysql/data
      port = 3306
      server_id = 241
      socket = /tmp/mysql.sock
      log-bin=mysql-bin
      log-slave-updates
      expire_logs_days = 10

      #slave01配置文件/etc/my.cnf 核心配置以下:

      basedir = /application/mysql
      datadir = /application/mysql/data
      port = 3306
      server_id = 242
      socket = /tmp/mysql.sock
      log-bin=mysql-bin
      log-slave-updates
      expire_logs_days = 10

      #slave02配置文件/etc/my.cnf 核心配置以下:

      basedir = /application/mysql
      datadir = /application/mysql/data
      port = 3306
      server_id = 243
      socket = /tmp/mysql.sock
      log-bin=mysql-bin
      log-slave-updates
      expire_logs_days = 10
      read_only = 1

      2)配置master、slave01和slave02之間的主從複製

      注意:binlog-do-db 和 replicate-ignore-db 設置必須相同。 MHA 在啓動時候會檢測過濾規則,若是過濾規則不一樣,MHA 不啓動監控和故障轉移。

      在MySQL5.6 的Replication配置中,master端一樣要開啓兩個重要的選項,server-id和log-bin,而且選項server-id在全局架構中而且惟一,不能被其它主機使用,這裏採用主機ip地址的最後一位充當server-id的值;slave端要開啓relay-log;

      #主機: master執行命令

      [root@master ~]# egrep "log-bin|server_id" /etc/my.cnf server_id = 241 log-bin=mysql-bin 

      #主機: slave01執行命令

      [root@slave01 ~]# egrep "log-bin|server_id" /etc/my.cnf server_id = 242 log-bin=mysql-bin 

      #主機: slave02執行命令

      [root@slave02 ~]# egrep "log-bin|server_id" /etc/my.cnf server_id = 243 log-bin=mysql-bin 

      3)在master、slave01上建立主從同步的帳號。

      slave01是備用master,這個也須要創建受權用戶。

      #master [root@master ~]# mysql -e "grant replication slave on *.* to 'backup'@'172.16.1.%' identified by 'backup';flush privileges; #slave01 [root@slave01 ~]# mysql -e "grant replication slave on *.* to 'backup'@'172.16.1.%' identified by 'backup';flush privileges;" 

      4)在master上執行命令,查看master狀態信息

      [root@master ~]# mysql -e 'show master status;' +------------------+----------+--------------+------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | +------------------+----------+--------------+------------------+ | mysql-bin.000007 | 107 | | | +------------------+----------+--------------+------------------+ 

      5)在slave01和slave02上執行主從同步

      #slave01配置主從

      [root@slave01 ~]# mysql mysql> change master to master_host='172.16.1.241',master_user='backup',master_password='backup',master_port=3306,master_log_file='mysql-bin.000007',master_log_pos=107; Query OK, 0 rows affected (0.12 sec) mysql> start slave; Query OK, 0 rows affected (0.00 sec) mysql> show slave status\G *************************** 1. row ***************************  Slave_IO_State: Waiting for master to send event  Master_Host: 172.16.1.241  Master_User: backup  Master_Port: 3306  Connect_Retry: 60  Master_Log_File: mysql-bin.000007  Read_Master_Log_Pos: 107  Relay_Log_File: slave01-relay-bin.000002  Relay_Log_Pos: 253  Relay_Master_Log_File: mysql-bin.000007  Slave_IO_Running: Yes  Slave_SQL_Running: Yes  Replicate_Do_DB:  Replicate_Ignore_DB:  Replicate_Do_Table:  Replicate_Ignore_Table:  Replicate_Wild_Do_Table:  Replicate_Wild_Ignore_Table:  Last_Errno: 0  Last_Error:  Skip_Counter: 0  Exec_Master_Log_Pos: 107  Relay_Log_Space: 411  Until_Condition: None  Until_Log_File:  Until_Log_Pos: 0  Master_SSL_Allowed: No  Master_SSL_CA_File:  Master_SSL_CA_Path:  Master_SSL_Cert:  Master_SSL_Cipher:  Master_SSL_Key:  Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No  Last_IO_Errno: 0  Last_IO_Error:  Last_SQL_Errno: 0  Last_SQL_Error:  Replicate_Ignore_Server_Ids:  Master_Server_Id: 241 1 row in set (0.00 sec) 

      #slave02配置主從

      [root@slave02 ~]# mysql mysql> change master to master_host='172.16.1.241',master_user='backup',master_password='backup',master_port=3306,master_log_file='mysql-bin.000007',master_log_pos=107; Query OK, 0 rows affected (0.12 sec) mysql> start slave; Query OK, 0 rows affected (0.00 sec) mysql> show slave status\G *************************** 1. row ***************************  Slave_IO_State: Waiting for master to send event  Master_Host: 172.16.1.241  Master_User: backup  Master_Port: 3306  Connect_Retry: 60  Master_Log_File: mysql-bin.000007  Read_Master_Log_Pos: 107  Relay_Log_File: slave01-relay-bin.000002  Relay_Log_Pos: 253  Relay_Master_Log_File: mysql-bin.000007  Slave_IO_Running: Yes  Slave_SQL_Running: Yes  Replicate_Do_DB:  Replicate_Ignore_DB:  Replicate_Do_Table:  Replicate_Ignore_Table:  Replicate_Wild_Do_Table:  Replicate_Wild_Ignore_Table:  Last_Errno: 0  Last_Error:  Skip_Counter: 0  Exec_Master_Log_Pos: 107  Relay_Log_Space: 411  Until_Condition: None  Until_Log_File:  Until_Log_Pos: 0  Master_SSL_Allowed: No  Master_SSL_CA_File:  Master_SSL_CA_Path:  Master_SSL_Cert:  Master_SSL_Cipher:  Master_SSL_Key:  Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No  Last_IO_Errno: 0  Last_IO_Error:  Last_SQL_Errno: 0  Last_SQL_Error:  Replicate_Ignore_Server_Ids:  Master_Server_Id: 241 1 row in set (0.00 sec) 

      #至此主從已經配置完成!

  • 5、規劃mha

    1)建立mha管理用的複製帳號

    每臺數據庫(master、slave0一、slave02)上都要建立帳號,在這裏以其中master爲例.。

    [root@master ~]# mysql -e "grant all privileges on *.* to 'mha_rep'@'172.16.1.%' identified by '123456';flush privileges;" [root@master ~]# mysql mysql> select host,user from mysql.user; 

    2)在3臺主機上(master、slave01和slave02)上分別安裝mha4mysql-node包

    安裝完成後會在/usr/local/bin目錄下生成如下腳本文件:

    -r-xr-xr-x 1 root root 15498 4 2 16:04 apply_diff_relay_logs # 識別差別的中繼日誌事件並將其差別的事件應用於其餘的slave -r-xr-xr-x 1 root root  4807 4 2 16:04 filter_mysqlbinlog # 去除沒必要要的ROLLBACK事件(MHA已再也不使用這個工具) -r-xr-xr-x 1 root root  7401 4 2 16:04 purge_relay_logs # 清除中繼日誌(不會阻塞SQL線程) -r-xr-xr-x 1 root root  7263 4 2 16:04 save_binary_logs # 保存和複製master的二進制日誌 

    這裏以master爲例,其它主機同理。

    [root@master ~]# yum install perl-DBD-MySQL -y [root@master ~]# rpm -ivh https://downloads.mariadb.com/files/MHA/mha4mysql-node-0.54-0.el6.noarch.rpm 

    3)在manager上安裝mha4mysql-manager和mha4mysql-node包

    MHA Manager中主要包括了幾個管理員的命令行工具,例如master_manger,master_master_switch等。MHA Manger也依賴於perl模塊,具體以下:

    安裝完成後會在/usr/local/bin目錄下面生成如下腳本文件

    -r-xr-xr-x 1 root root 15498 4 2 15:59 apply_diff_relay_logs # 識別差別的中繼日誌事件並將其差別的事件應用於其餘的slave -r-xr-xr-x 1 root root  4807 4 2 15:59 filter_mysqlbinlog # 去除沒必要要的ROLLBACK事件(MHA已再也不使用這個工具) -r-xr-xr-x 1 root root  1995 4 2 16:21 masterha_check_repl # 檢查MySQL複製情況 -r-xr-xr-x 1 root root  1779 4 2 16:21 masterha_check_ssh # 檢查MHA的SSH配置情況 -r-xr-xr-x 1 root root  1865 4 2 16:21 masterha_check_status # 檢測當前MHA運行狀態 -r-xr-xr-x 1 root root  3201 4 2 16:21 masterha_conf_host # 添加或刪除配置的server信息 -r-xr-xr-x 1 root root  2517 4 2 16:21 masterha_manager # 啓動MHA -r-xr-xr-x 1 root root  2165 4 2 16:21 masterha_master_monitor # 檢測master是否宕機 -r-xr-xr-x 1 root root  2373 4 2 16:21 masterha_master_switch # 控制故障轉移(自動或者手動) -r-xr-xr-x 1 root root  3749 4 2 16:21 masterha_secondary_check # -r-xr-xr-x 1 root root  1739 4 2 16:21 masterha_stop # -r-xr-xr-x 1 root root  7401 4 2 15:59 purge_relay_logs # 清除中繼日誌(不會阻塞SQL線程) -r-xr-xr-x 1 root root  7263 4 2 15:59 save_binary_logs # 保存和複製master的二進制日誌 

    複製相關腳本到/usr/local/bin目錄(軟件包解壓縮後就有了,不是必須,由於這些腳本不完整,須要本身修改,這是軟件開發着留給咱們本身發揮的,若是開啓下面的任何一個腳本對應的參數,而對應這裏的腳本又沒有修改,則會報錯,本身被坑的很慘)

    [root@manager ~]# cd mha4mysql-manager-0.56/samples/scripts/ # 這是咱們下載解壓軟件的目錄 [root@manager scripts]# ll 總用量 32 -rwxr-xr-x 1 root root 3443 18 2012 master_ip_failover #自動切換時vip管理的腳本,不是必須,若是咱們使用keepalived的,咱們能夠本身編寫腳本完成對vip的管理,好比監控mysql,若是mysql異常,咱們中止keepalived就行,這樣vip就會自動漂移 -rwxr-xr-x 1 root root 9186 18 2012 master_ip_online_change #在線切換時vip的管理,不是必須,一樣能夠能夠自行編寫簡單的shell完成 -rwxr-xr-x 1 root root 11867 18 2012 power_manager #故障發生後關閉主機的腳本,不是必須 -rwxr-xr-x 1 root root 1360 18 2012 send_report #因故障切換後發送報警的腳本,不是必須,可自行編寫簡單的shell完成。 [root@manager ~]# cp * /usr/local/bin/ 

    #在manager上安裝mha4mysql-manager和mha4mysql-node包

    [root@manager ~]# yum install perl cpan perl-DBD-MySQL perl-DBD-MySQL perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager perl-Net-Telnet -y [root@manager ~]# rpm -ivh https://downloads.mariadb.com/files/MHA/mha4mysql-node-0.54-0.el6.noarch.rpm [root@manager ~]# wget https://downloads.mariadb.com/files/MHA/mha4mysql-manager-0.56.tar.gz [root@manager ~]# tar zvxf mha4mysql-manager-0.56.tar.gz [root@manager ~]# cd mha4mysql-manager-0.56 [root@manager ~]# perl Makefile.PL [root@manager mha4mysql-manager-0.56]# make && make install [root@manager mha4mysql-manager-0.56]# mkdir -p /usr/local/mha/scripts [root@manager mha4mysql-manager-0.56]# cp samples/conf/app1.cnf /usr/local/mha/mha.cnf [root@manager mha4mysql-manager-0.56]# cp samples/scripts/* /usr/local/mha/scripts/ 

    4)修改manager端mha的配置文件

    記得去註釋

    [root@manager mha4mysql-manager-0.56]# vim /usr/local/mha/mha.cnf [server default] user=mha_rep #MHA管理mysql的用戶名 password=123456 #MHA管理mysql的密碼 manager_workdir=/usr/local/mha #MHA的工做目錄 manager_log=/usr/local/mha/manager.log #MHA的日誌路徑 ssh_user=root #免祕鑰登錄的用戶名 repl_user=backup #主從複製帳號,用來在主從之間同步數據 repl_password=backup ping_interval=1 #ping間隔時間,用來檢查master是否正常 [server1] hostname=172.16.1.241 master_binlog_dir=/application/mysql/data/ candidate_master=1 #master宕機後,優先啓用這臺做爲master [server2] hostname=172.16.1.242 master_binlog_dir=/application/mysql/data/ candidate_master=1 [server3] hostname=172.16.1.243 master_binlog_dir=/application/mysql/data/ no_master=1 

    5)檢查ssh是否暢通

    注意:全部主機之間必須作SSH免密鑰登陸。不然報錯。研究了兩天。(經過查看MHA的功能實現過程發現)

    [root@manager ~]# masterha_check_ssh --conf=/usr/local/mha/mha.cnf Mon Apr 3 21:42:33 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Mon Apr 3 21:42:33 2017 - [info] Reading application default configurations from /usr/local/mha/mha.cnf.. Mon Apr 3 21:42:33 2017 - [info] Reading server configurations from /usr/local/mha/mha.cnf.. Mon Apr 3 21:42:33 2017 - [info] Starting SSH connection tests.. Mon Apr 3 21:42:33 2017 - [debug] Mon Apr 3 21:42:33 2017 - [debug] Connecting via SSH from root@172.16.1.241(172.16.1.241:22) to root@172.16.1.242(172.16.1.242:22).. Mon Apr 3 21:42:33 2017 - [debug] ok. Mon Apr 3 21:42:33 2017 - [debug] Connecting via SSH from root@172.16.1.241(172.16.1.241:22) to root@172.16.1.243(172.16.1.243:22).. Mon Apr 3 21:42:33 2017 - [debug] ok. Mon Apr 3 21:42:34 2017 - [debug] Mon Apr 3 21:42:34 2017 - [debug] Connecting via SSH from root@172.16.1.243(172.16.1.243:22) to root@172.16.1.241(172.16.1.241:22).. Mon Apr 3 21:42:34 2017 - [debug] ok. Mon Apr 3 21:42:34 2017 - [debug] Connecting via SSH from root@172.16.1.243(172.16.1.243:22) to root@172.16.1.242(172.16.1.242:22).. Mon Apr 3 21:42:34 2017 - [debug] ok. Mon Apr 3 21:42:34 2017 - [debug] Mon Apr 3 21:42:33 2017 - [debug] Connecting via SSH from root@172.16.1.242(172.16.1.242:22) to root@172.16.1.241(172.16.1.241:22).. Mon Apr 3 21:42:33 2017 - [debug] ok. Mon Apr 3 21:42:33 2017 - [debug] Connecting via SSH from root@172.16.1.242(172.16.1.242:22) to root@172.16.1.243(172.16.1.243:22).. Mon Apr 3 21:42:34 2017 - [debug] ok. Mon Apr 3 21:42:34 2017 - [info] All SSH connection tests passed successfully. 

    #若是獲得以上結果,代表主機之間ssh互信是暢通的

    6)masterha_check_repl工具檢查mysql主從複製是否成功

    注意:slave01 slave02和master確保已經作好主從複製。不然出錯。(研究22個小時)不懂perl 挺麻煩的。

    [root@manager ~]#  masterha_check_repl --conf=/usr/local/mha/mha.cnf Mon Apr 3 21:44:13 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Mon Apr 3 21:44:13 2017 - [info] Reading application default configurations from /usr/local/mha/mha.cnf.. Mon Apr 3 21:44:13 2017 - [info] Reading server configurations from /usr/local/mha/mha.cnf.. Mon Apr 3 21:44:13 2017 - [info] MHA::MasterMonitor version 0.56. Mon Apr 3 21:44:14 2017 - [info] Dead Servers: Mon Apr 3 21:44:14 2017 - [info] Alive Servers: Mon Apr 3 21:44:14 2017 - [info] 172.16.1.241(172.16.1.241:3306) Mon Apr 3 21:44:14 2017 - [info] 172.16.1.242(172.16.1.242:3306) Mon Apr 3 21:44:14 2017 - [info] 172.16.1.243(172.16.1.243:3306) Mon Apr 3 21:44:14 2017 - [info] Alive Slaves: Mon Apr 3 21:44:14 2017 - [info] 172.16.1.242(172.16.1.242:3306) Version=5.5.32-log (oldest major version between slaves) log-bin:enabled Mon Apr 3 21:44:14 2017 - [info] Replicating from 172.16.1.241(172.16.1.241:3306) Mon Apr 3 21:44:14 2017 - [info] Primary candidate for the new Master (candidate_master is set) Mon Apr 3 21:44:14 2017 - [info] 172.16.1.243(172.16.1.243:3306) Version=5.5.32-log (oldest major version between slaves) log-bin:enabled Mon Apr 3 21:44:14 2017 - [info] Replicating from 172.16.1.241(172.16.1.241:3306) Mon Apr 3 21:44:14 2017 - [info] Not candidate for the new Master (no_master is set) Mon Apr 3 21:44:14 2017 - [info] Current Alive Master: 172.16.1.241(172.16.1.241:3306) Mon Apr 3 21:44:14 2017 - [info] Checking slave configurations.. Mon Apr 3 21:44:14 2017 - [info] read_only=1 is not set on slave 172.16.1.242(172.16.1.242:3306). Mon Apr 3 21:44:14 2017 - [warning] relay_log_purge=0 is not set on slave 172.16.1.242(172.16.1.242:3306). Mon Apr 3 21:44:14 2017 - [warning] relay_log_purge=0 is not set on slave 172.16.1.243(172.16.1.243:3306). Mon Apr 3 21:44:14 2017 - [info] Checking replication filtering settings.. Mon Apr 3 21:44:14 2017 - [info] binlog_do_db= , binlog_ignore_db= Mon Apr 3 21:44:14 2017 - [info] Replication filtering check ok. Mon Apr 3 21:44:14 2017 - [info] Starting SSH connection tests.. Mon Apr 3 21:44:16 2017 - [info] All SSH connection tests passed successfully. Mon Apr 3 21:44:16 2017 - [info] Checking MHA Node version.. Mon Apr 3 21:44:16 2017 - [info] Version check ok. Mon Apr 3 21:44:16 2017 - [info] Checking SSH publickey authentication settings on the current master.. Mon Apr 3 21:44:16 2017 - [info] HealthCheck: SSH to 172.16.1.241 is reachable. Mon Apr 3 21:44:17 2017 - [info] Master MHA Node version is 0.54. Mon Apr 3 21:44:17 2017 - [info] Checking recovery script configurations on the current master.. Mon Apr 3 21:44:17 2017 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/application/mysql/data/ --output_file=/var/tmp/save_binary_logs_test --manager_version=0.56 --start_file=mysql-bin.000007 Mon Apr 3 21:44:17 2017 - [info] Connecting to root@172.16.1.241(172.16.1.241).. Creating /var/tmp if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /application/mysql/data/, up to mysql-bin.000007 Mon Apr 3 21:44:17 2017 - [info] Master setting check done. Mon Apr 3 21:44:17 2017 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers.. Mon Apr 3 21:44:17 2017 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mha_rep' --slave_host=172.16.1.242 --slave_ip=172.16.1.242 --slave_port=3306 --workdir=/var/tmp --target_version=5.5.32-log --manager_version=0.56 --relay_log_info=/application/mysql/data/relay-log.info --relay_dir=/application/mysql/data/ --slave_pass=xxx Mon Apr 3 21:44:17 2017 - [info] Connecting to root@172.16.1.242(172.16.1.242:22).. Checking slave recovery environment settings.. Opening /application/mysql/data/relay-log.info ... ok. Relay log found at /application/mysql/data, up to slave01-relay-bin.000002 Temporary relay log file is /application/mysql/data/slave01-relay-bin.000002 Testing mysql connection and privileges.. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Mon Apr 3 21:44:17 2017 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mha_rep' --slave_host=172.16.1.243 --slave_ip=172.16.1.243 --slave_port=3306 --workdir=/var/tmp --target_version=5.5.32-log --manager_version=0.56 --relay_log_info=/application/mysql/data/relay-log.info --relay_dir=/application/mysql/data/ --slave_pass=xxx Mon Apr 3 21:44:17 2017 - [info] Connecting to root@172.16.1.243(172.16.1.243:22).. Checking slave recovery environment settings.. Opening /application/mysql/data/relay-log.info ... ok. Relay log found at /application/mysql/data, up to slave02-relay-bin.000002 Temporary relay log file is /application/mysql/data/slave02-relay-bin.000002 Testing mysql connection and privileges.. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Mon Apr 3 21:44:18 2017 - [info] Slaves settings check done. Mon Apr 3 21:44:18 2017 - [info] 172.16.1.241 (current master) +--172.16.1.242 +--172.16.1.243 Mon Apr 3 21:44:18 2017 - [info] Checking replication health on 172.16.1.242.. Mon Apr 3 21:44:18 2017 - [info] ok. Mon Apr 3 21:44:18 2017 - [info] Checking replication health on 172.16.1.243.. Mon Apr 3 21:44:18 2017 - [info] ok. Mon Apr 3 21:44:18 2017 - [warning] master_ip_failover_script is not defined. Mon Apr 3 21:44:18 2017 - [warning] shutdown_script is not defined. Mon Apr 3 21:44:18 2017 - [info] Got exit code 0 (Not master dead). MySQL Replication Health is OK.
  • 6、mha實驗模擬

    1)在每次作mha實驗的時候,咱們都最好先執行以下命令作檢測

    [root@manager ~]# masterha_check_ssh --conf=/usr/local/mha/mha.cnf [root@manager ~]# masterha_check_repl --conf=/usr/local/mha/mha.cnf 

    #肯定兩條命令的返回結果都是無異常的,而後啓動mha服務

    2)在manager端啓動mha服務並時刻監控日誌文件的輸出變化

    [root@manager ~]# nohup masterha_manager --conf=/usr/local/mha/mha.cnf > /tmp/mha_manager.log 2>&1 & [root@manager ~]# ps -ef |grep masterha |grep -v 'grep' root 2840 2470 2 10:53 pts/0 00:00:00 perl /usr/local/bin/masterha_manager --conf=/usr/local/mha/mha.cnf 

    3)測試master宕機後會自動切換

    #測試前查看slave01,slave02的主從同步狀況

    #slave01

    [root@slave01 ~]# mysql -e 'show slave status\G' |egrep 'Slave_IO_Running:|Slave_SQL_Running:'  Slave_IO_Running: Yes  Slave_SQL_Running: Yes 

    #slave02

    [root@slave02 ~]# mysql -e 'show slave status\G' |egrep 'Slave_IO_Running:|Slave_SQL_Running:'  Slave_IO_Running: Yes  Slave_SQL_Running: Yes 

    #中止master的mysql服務

    [root@master ~]# service mysqld stop Shutting down MySQL (Percona Server)..... SUCCESS! 

    #manager上查看manager節點日誌

    [root@manager ~]# cat /usr/local/mha/manager.log
    
    ----- Failover Report -----
     mha: MySQL Master failover 172.16.1.241 to 172.16.1.242 succeeded Master 172.16.1.241 is down! Check MHA Manager logs at manager:/usr/local/mha/manager.log for details. Started automated(non-interactive) failover. The latest slave 172.16.1.242(172.16.1.242:3306) has all relay logs for recovery. Selected 172.16.1.242 as a new master. 172.16.1.242: OK: Applying all logs succeeded. 172.16.1.243: This host has the latest relay log events. Generating relay diff files from the latest slave succeeded. 172.16.1.243: OK: Applying all logs succeeded. Slave started, replicating from 172.16.1.242. 172.16.1.242: Resetting slave info succeeded. Master failover to 172.16.1.242(172.16.1.242:3306) completed successfully. 

    從上面的輸出能夠看出整個MHA的切換過程,共包括如下的步驟:

    1. 配置文件檢查階段,這個階段會檢查整個集羣配置文件配置。
    2. 宕機的master處理,這個階段包括虛擬ip摘除操做,主機關機操做(待研究)。
    3. 複製dead maste和最新slave相差的relay log,並保存到MHA Manger具體的目錄下。
    4. 識別含有最新更新的slave。
    5. 應用從binlog服務器保存的二進制日誌事件(binlog events)。
    6. 提高一個slave爲新的master進行復制。
    7. 使其餘的slave鏈接新的master進行復制。

    6)驗證new master(172.16.1.242)

    #咱們查看slave02的主從同步信息

    [root@slave02 ~]# mysql -e 'show slave status\G' |egrep 'Master_Host|Slave_IO_Running:|Slave_SQL_Running:'  Master_Host: 172.16.1.242 # 表示已經轉移新的ip  Slave_IO_Running: Yes # 表示主從OK  Slave_SQL_Running: Yes 

    4)恢復master服務

    #manage刪除故障轉移文件

    [root@manager ~]# cat /usr/local/mha/mha.failover.complete [root@manager ~]# rm -rf /usr/local/mha/mha.failover.complete 

    #master重啓mysql服務

    [root@master ~]# service mysqld start Starting MySQL... SUCCESS! 

    #在manager的日誌文件中找到主從同步的sql語句

    [root@manager ~]# grep MASTER_HOST /usr/local/mha/manager.log
    Mon Apr  3 21:50:59 2017 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='172.16.1.242', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000016', MASTER_LOG_POS=107, MASTER_USER='backup', MASTER_PASSWORD='xxx'; 

    #在master上啓動主從同步,密碼爲backup

    master_log_file和master_log_pos參數須要和上面manager的日誌文件中同步的語句參數裏的值相同。

    mysql> change master to master_host='172.16.1.242',master_user='backup',master_password='backup',master_port=3306,master_log_file='mysql-bin.000016',master_log_pos=107; Query OK, 0 rows affected (1.02 sec) mysql> start slave; Query OK, 0 rows affected (0.00 sec) 

    #在master和slave02上執行,檢查主從同步是否都正常,這裏以master爲例,slave02同理

    [root@master ~]# mysql -e 'show slave status\G' |egrep 'Master_Host|Slave_IO_Running:|Slave_SQL_Running:'  Master_Host: 172.16.1.242  Slave_IO_Running: Yes  Slave_SQL_Running: Yes 

    5)再次啓動MHA的manager服務,並中止slave01

    [root@manager ~]# nohup masterha_manager --conf=/usr/local/mha/mha.cnf > /tmp/mha_manager.log 2>&1 & 

    #關閉slave01的mysql服務

    [root@slave01 ~]# service mysqld stop
    Shutting down MySQL... SUCCESS [root@slave01 ~]#tail -f /usr/local/mha/manager.log ----- Failover Report -----  mha: MySQL Master failover 172.16.1.242 to 172.16.1.241 succeeded Master 172.16.1.242 is down! Check MHA Manager logs at manager:/usr/local/mha/manager.log for details. Started automated(non-interactive) failover. The latest slave 172.16.1.241(172.16.1.241:3306) has all relay logs for recovery. Selected 172.16.1.241 as a new master. 172.16.1.241: OK: Applying all logs succeeded. 172.16.1.243: This host has the latest relay log events. Generating relay diff files from the latest slave succeeded. 172.16.1.243: OK: Applying all logs succeeded. Slave started, replicating from 172.16.1.241. 172.16.1.241: Resetting slave info succeeded. Master failover to 172.16.1.241(172.16.1.241:3306) completed successfully. 

    出現故障的快速恢復步驟

    [root@slave01 ~]# service mysqld stop Shutting down MySQL... SUCCESS [root@manager mha]# tail -f /usr/local/mha/manager.log ----- Failover Report ----- mha: MySQL Master failover 172.16.1.242 Master 172.16.1.242 is down! Check MHA Manager logs at manager:/usr/local/mha/manager.log for details. Started automated(non-interactive) failover. The latest slave 172.16.1.241(172.16.1.241:3306) has all relay logs for recovery. Got Error so couldn't continue failover from here. #出現沒法切換回去,後來通過排查是manager /usr/local/mha/mha.cnf [server1] (比較低級的錯誤,排查好久。不過主要是想跟你們分享出現問題如何恢復到以前的狀態。) hostname=172.16.1.241 master_binlog_dir=/application/mysql/data/ candidate_master=1r #這裏多加了一個r#修改完畢 hostname=172.16.1.241 master_binlog_dir=/application/mysql/data/ candidate_master=1 實現文件手動恢復到以前的狀態。 #manager [root@manager ~]# rm -rf /usr/local/mha/mha.failover.complete [root@manager ~]# rm -rf /usr/local/mha/mha.failover.error [root@manager ~]# nohup masterha_manager --conf=/usr/local/mha/mha.cnf > /tmp/mha_manager.log 2>&1 & #master [root@master ~]# mysql mysql> stop slave; mysql> reset slave; mysql> show master status\G *************************** 1. row *************************** File: mysql-bin.000013 Position: 107 Binlog_Do_DB: Binlog_Ignore_DB: 1 row in set (0.00 sec) #slave01 [root@slave01 ~]# mysql mysql> stop slave; mysql> change master to master_host='172.16.1.241',master_user='backup',master_password='backup',master_port=3306,master_log_file='mysql-bin.000013',master_log_pos=107; mysql> start slave; # slave01和slave02恢復以前的狀態。 [root@slave01 ~]# mysql -e 'show slave status\G' |egrep 'Master_Host|Slave_IO_Running:|Slave_SQL_Running:' Master_Host: 172.16.1.241 Slave_IO_Running: Yes Slave_SQL_Running: Yes [root@slave02 ~]# mysql -e 'show slave status\G' |egrep 'Master_Host|Slave_IO_Running:|Slave_SQL_Running:' Master_Host: 172.16.1.241 Slave_IO_Running: Yes Slave_SQL_Running: Yes 

    #manager上查看manager節點日誌

    [root@manager ~]# cat /usr/local/mha/manager.log ----- Failover Report ----- mha: MySQL Master failover 172.16.1.242 Master 172.16.1.242 is down! Check MHA Manager logs at manager:/usr/local/mha/manager.log for details. Started automated(non-interactive) failover. The latest slave 172.16.1.241(172.16.1.241:3306) has all relay logs for recovery. Got Error so couldn't continue failover from here. 

    6)恢復slave01服務

    #刪除故障轉移文件

    [root@manager ~]# rm -rf /usr/local/mha/mha.failover.complete 

    #重啓mysql服務

    [root@slave01 ~]# service mysqld start Starting MySQL.. SUCCESS! 

    #在manager的日子文件中找到主從同步的sql語句

    [root@manager ~]#  grep MASTER_HOST /usr/local/mha/manager.log
    Tue Apr  4 02:47:33 2017 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='172.16.1.241', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000015', MASTER_LOG_POS=107, MASTER_USER='backup', MASTER_PASSWORD='xxx'; 

    #在slave01上啓動主從同步,密碼爲backup 記得修改MASTER_PASSWORD='xxx' 爲 MASTER_PASSWORD='bakcup'

    [root@slave01 ~]# mysql
    
    mysql> stop slave
    
    mysql> CHANGE MASTER TO MASTER_HOST='172.16.1.241', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000015', MASTER_LOG_POS=107, MASTER_USER='backup', MASTER_PASSWORD='backup'; Query OK, 0 rows affected (0.39 sec) mysql> start slave; Query OK, 0 rows affected (0.00 sec) 

    #在slave01和slave02上執行,檢查主從同步是否都正常,

    #slave01 [root@slave01 ~]# mysql -e 'show slave status\G' |egrep 'Master_Host|Slave_IO_Running:|Slave_SQL_Running:'  Master_Host: 172.16.1.241  Slave_IO_Running: Yes  Slave_SQL_Running: Yes #slave02 [root@slave02 ~]# mysql -e 'show slave status\G' |egrep 'Master_Host|Slave_IO_Running:|Slave_SQL_Running:'  Master_Host: 172.16.1.241  Slave_IO_Running: Yes  Slave_SQL_Running: Yes 

    7)重啓MHA的manager服務

    [root@manager ~]# nohup masterha_manager --conf=/usr/local/mha/mha.cnf > /tmp/mha_manager.log 2>&1 & [1] 30389 

    7、經過vip實現mysql的高可用

    1)修改/usr/local/mha/mha.cnf

    [server default] user=mha_rep password=123456 manager_workdir=/usr/local/mha manager_log=/usr/local/mha/manager.log ssh_user=root master_ip_failover_script=/usr/local/mha/scripts/master_ip_failover #添加管理vip的腳本 repl_user=backup repl_password=backup ping_interval=1  [server1] hostname=172.16.1.241 master_binlog_dir=/application/mysql/data/ candidate_master=1 port=3306  [server2] hostname=172.16.1.242 master_binlog_dir=/application/mysql/data/ candidate_master=1 port=3306  [server3] hostname=172.16.1.243 master_binlog_dir=/application/mysql/data/ port=3306 no_master=1 

    2)修改腳本/usr/local/mha/scripts/master_ip_failover

    #!/usr/bin/env perl use strict; use warnings FATAL => 'all'; use Getopt::Long; my ( $command, $ssh_user, $orig_master_host, $orig_master_ip, $orig_master_port, $new_master_host, $new_master_ip, $new_master_port ); my $vip = '172.16.1.240'; #vip地址 my $key = '1'; my $ssh_start_vip = "/sbin/ifconfig eth1:$key $vip"; #綁定在指定的網卡上面 my $ssh_stop_vip = "/sbin/ifconfig eth1:$key down"; #個人機器有兩塊網卡eth1是172網段的全部我把vip綁定在eth1上,個人eth0網段是10.0.0.%。 GetOptions( 'command=s' => \$command, 'ssh_user=s' => \$ssh_user, 'orig_master_host=s' => \$orig_master_host, 'orig_master_ip=s' => \$orig_master_ip, 'orig_master_port=i' => \$orig_master_port, 'new_master_host=s' => \$new_master_host, 'new_master_ip=s' => \$new_master_ip, 'new_master_port=i' => \$new_master_port, ); exit &main(); sub main { print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n"; if ( $command eq "stop" || $command eq "stopssh" ) { my $exit_code = 1; eval { print "Disabling the VIP on old master: $orig_master_host \n"; &stop_vip(); $exit_code = 0; }; if ($@) { warn "Got Error: $@\n"; exit $exit_code; } exit $exit_code; } elsif ( $command eq "start" ) { my $exit_code = 10; eval { print "Enabling the VIP - $vip on the new master - $new_master_host \n"; &start_vip(); $exit_code = 0; }; if ($@) { warn $@; exit $exit_code; } exit $exit_code; } elsif ( $command eq "status" ) { print "Checking the Status of the script.. OK \n"; exit 0; } else { &usage(); exit 1; } } sub start_vip() { `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`; } # A simple system call that disable the VIP on the old_master sub stop_vip() { `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`; } sub usage { print "Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n"; } 

    3)模擬故障進行切換

    #中止master的mysql服務

    [root@master ~]# service mysqld stop Shutting down MySQL... SUCCESS! 

    #查看slave02的同步信息

    [root@slave02 ~]# mysql -e 'show slave status\G' |egrep 'Master_Host|Slave_IO_Running:|Slave_SQL_Running:'  Master_Host: 172.16.1.242  Slave_IO_Running: Yes  Slave_SQL_Running: Yes 

    #查看slave01的IP信息

    [root@slave01 ~]# ifconfig eth0 Link encap:Ethernet HWaddr 00:1C:42:58:08:EF inet addr:10.0.0.242 Bcast:10.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::21c:42ff:fe58:8ef/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:6925 errors:0 dropped:0 overruns:0 frame:0 TX packets:2869 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:679548 (663.6 KiB) TX bytes:420365 (410.5 KiB) eth0:1 Link encap:Ethernet HWaddr 00:1C:42:58:08:EF inet addr:172.16.1.240 Bcast:172.16.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 eth1 Link encap:Ethernet HWaddr 00:1C:42:F4:DF:3E inet addr:172.16.1.242 Bcast:172.16.1.255 Mask:255.255.255.0 inet6 addr: fe80::21c:42ff:fef4:df3e/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:10272 errors:0 dropped:0 overruns:0 frame:0 TX packets:7875 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1575148 (1.5 MiB) TX bytes:1644494 (1.5 MiB) eth1:1 Link encap:Ethernet HWaddr 00:1C:42:F4:DF:3E inet addr:172.16.1.240 Bcast:172.16.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 # 這能夠看到咱們添加的VIP已經自動添加了 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:640 errors:0 dropped:0 overruns:0 frame:0 TX packets:640 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:51251 (50.0 KiB) TX bytes:51251 (50.0 KiB) 

    四、恢復master的mysql服務同開始恢復方法同樣。

  • 8、MHA平常維護命令

    一、查看ssh登錄是否成功

    masterha_check_ssh --conf=/usr/local/mha/mha.cnf 

    二、查看複製是否創建好

    masterha_check_repl --conf=/usr/local/mha/mha.cnf 

    三、啓動mha

    nohup masterha_manager --conf=/usr/local/mha/mha.cnf > /tmp/mha_manager.log 2>&1 & 

    四、檢查啓動的狀態

    masterha_check_status --conf=/usr/local/mha/mha.cnf 

    五、中止mha

    masterha_stop masterha_check_status --conf=/usr/local/mha/mha.cnf 

    六、failover後下次重啓

    #每次failover切換後會在管理目錄生成文件app1.failover.complete ,下次在切換的時候會發現有這個文件致使切換不成功,須要手動清理掉。

    rm -rf /usr/local/mha/mha.failover.complete 

    9、FAQ(常見問題解答)

    一、可能報錯1

    [root@server02 mha4mysql-node-0.53]# perl Makefile.PL Can't locate ExtUtils/MakeMaker.pm in @INC (@INC contains: inc /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at inc/Module/Install/Can.pm line 6. BEGIN failed--compilation aborted at inc/Module/Install/Can.pm line 6. Compilation failed in require at inc/Module/Install.pm line 307. Can't locate ExtUtils/MakeMaker.pm in @INC (@INC contains: inc /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at inc/Module/Install/Makefile.pm line 4. BEGIN failed--compilation aborted at inc/Module/Install/Makefile.pm line 4. Compilation failed in require at inc/Module/Install.pm line 307. Can't locate ExtUtils/MM_Unix.pm in @INC (@INC contains: inc /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at inc/Module/Install/Metadata.pm line 316. 

    解決辦法:

    yum install cpan -y yum install perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker -y cpan ExtUtils::Install 

    若是不想用cpan安裝,那就使用下面這條命令

    yum install perl-ExtUtils-Embed -y 

    二、可能報錯2

    [root@server02 mha4mysql-node-0.53]# perl Makefile.PL Can't locate ExtUtils/MakeMaker.pm in @INC (@INC contains: /usr/local/lib64/perl BEGIN failed--compilation aborted at Makefile.PL line 3. 

    解決辦法:

    yum install perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker 

    三、可能報錯3

    [root@server01 ~]# masterha_check_ssh --conf=/etc/masterha/app1.cnf

    報錯:

    Sun Apr  2 18:58:10 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Apr 2 18:58:10 2017 - [info] Reading application default configurations from /etc/masterha/app1.cnf.. Sun Apr 2 18:58:10 2017 - [info] Reading server configurations from /etc/masterha/app1.cnf.. Sun Apr 2 18:58:10 2017 - [info] Starting SSH connection tests.. Sun Apr 2 18:58:11 2017 - [error][/usr/local/share/perl5/MHA/SSHCheck.pm, ln63] Sun Apr 2 18:58:10 2017 - [debug] Connecting via SSH from root@172.16.1.50(172.16.1.50:22) to root@172.16.1.60(172.16.1.60:22).. Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). Sun Apr 2 18:58:10 2017 - [error][/usr/local/share/perl5/MHA/SSHCheck.pm, ln107] SSH connection from root@172.16.1.50(172.16.1.50:22) to root@172.16.1.60(172.16.1.60:22) failed! Sun Apr 2 18:58:11 2017 - [error][/usr/local/share/perl5/MHA/SSHCheck.pm, ln63] Sun Apr 2 18:58:10 2017 - [debug] Connecting via SSH from root@172.16.1.60(172.16.1.60:22) to root@172.16.1.50(172.16.1.50:22).. Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). Sun Apr 2 18:58:10 2017 - [error][/usr/local/share/perl5/MHA/SSHCheck.pm, ln107] SSH connection from root@172.16.1.60(172.16.1.60:22) to root@172.16.1.50(172.16.1.50:22) failed! Sun Apr 2 18:58:11 2017 - [error][/usr/local/share/perl5/MHA/SSHCheck.pm, ln63] Sun Apr 2 18:58:11 2017 - [debug] Connecting via SSH from root@172.16.1.70(172.16.1.70:22) to root@172.16.1.50(172.16.1.50:22).. Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). Sun Apr 2 18:58:11 2017 - [error][/usr/local/share/perl5/MHA/SSHCheck.pm, ln107] SSH connection from root@172.16.1.70(172.16.1.70:22) to root@172.16.1.50(172.16.1.50:22) failed! SSH Configuration Check Failed! at /usr/local/bin/masterha_check_ssh line 44 

    緣由分析,程序須要從manage管理ssh鏈接,因此會從mysql-test3 ssh到 mysql-test 再ssh到 mysql-test2,問題出在第二次鏈接,須要輸入key的密碼,致使測試失敗。因此所有機器都要相互作密鑰登陸。

    四、可能報錯4

    mysql> change master to master_host='172.16.1.241',master_user='backup',master_password='backup',master_port=3306,master_log_file='mysql-bin.000007',master_log_pos=333;

    報錯:

    ERROR 1201 (HY000): Could not initialize master info structure; more error messages can be found in the MySQL error log

    解決方法

    mysql> reset slave;
    Query OK, 0 rows affected (0.00 sec) mysql> change master to master_host='172.16.1.241',master_user='backup',master_password='backup',master_port=3306,master_log_file='mysql-bin.000007',master_log_pos=333; Query OK, 0 rows affected (0.15 sec) mysql> start slave; Query OK, 0 rows affected (0.01 sec) mysql> show slave status\G 

    五、可能報錯5

    Can't locate Log/Dispatch.pm in @INC(報錯)耗時22個小時才解決

    [root@manager ~]# masterha_check_ssh --conf=/usr/local/mha/mha.cnf

    Can't locate Log/Dispatch.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/local/share/perl5/MHA/MasterMonitor.pm line 28. BEGIN failed--compilation aborted at /usr/local/share/perl5/MHA/MasterMonitor.pm line 28. Compilation failed in require at /usr/local/bin/masterha_manager line 26. BEGIN failed--compilation aborted at /usr/local/bin/masterha_manager line 26. 
    $ sudo cpan cpan[1]> install CPAN cpan[2]> install Module::Build cpan[3]> quit $ sudo cpan cpan[1]> install Log::Dispatch cpan[2]> install Log::Dispatch::FileRotate cpan[3]> quit 

    六、小知識

    [root@server02 ~]# mysqldump -uroot -p123456 --master-data=2 --single-transaction -R --triggers -A --event > /server/backup/all_bak_$(date +%F).sql

    -u 數據庫登陸用戶名 -p 數據庫登陸密碼 --event 因爲mysql在全量導出時不導出event事件表,故須要在全量導出時忽略事件表,不加此參數會出現告警。 Warning: Skipping the data of table mysql.event. Specify the --events option explicitly --master-data=2表明備份時刻記錄master的Binlog位置和Position(位置點) --single-transaction意思是獲取一致性快照,-R意思是備份存儲過程和函數 --triggres的意思是備份觸發器 -A表明備份全部的庫 更多信息請自行mysqldump --help查看。或http://note.youdao.com/noteshare?id=60607599966788d19a4d46d4ccd2ce9d 

    查看複製狀態(能夠看見覆製成功):

    [root@server03 ~]# mysql -uroot -p123456 -e 'show slave status\G' | egrep 'Slave_IO|Slave_SQL|Until_Log_Pos'  Slave_IO_State: Waiting for master to send event  Slave_IO_Running: Yes # IO線程  Slave_SQL_Running: Yes # SQL線程  Until_Log_Pos: 0 # 主從之間的延遲 

    修改app1.cnf配置文件,修改後的文件內容以下: 配置參數註釋請看:MHA app1.cnf 配置文件註釋

    (2)設置relay log的清除方式(在每一個slave節點上):

    server03
    [root@server03 ~]# mysql -uroot -p123456 -e 'set global relay_log_purge=0' server04 [root@server04 ~]# mysql -uroot -p123456 -e 'set global relay_log_purge=0' 

    注意:

    MHA在發生切換的過程當中,從庫的恢復過程當中依賴於relay log的相關信息,因此這裏要將relay log的自動清除設置爲OFF,採用手動清除relay log的方式。

    在默認狀況下,從服務器上的中繼日誌會在SQL線程執行完畢後被自動刪除。可是在MHA環境中,這些中繼日誌在恢復其餘從服務器時可能會被用到,所以須要禁用中繼日誌的自動刪除功能。

    按期清除中繼日誌須要考慮到複製延時的問題。在ext3的文件系統下,刪除大的文件須要必定的時間,會致使嚴重的複製延時。爲了不復制延時,須要暫時爲中繼日誌建立硬連接,由於在linux系統中經過硬連接刪除大文件速度會很快。(在mysql數據庫中,刪除大表時,一般也採用創建硬連接的方式)

    MHA節點中包含了pure_relay_logs命令工具,它能夠爲中繼日誌建立硬連接,執行SET GLOBAL relay_log_purge=1,等待幾秒鐘以便SQL線程切換到新的中繼日誌,再執行SET GLOBAL relay_log_purge=0。

    pure_relay_logs腳本參數以下所示:

    --user mysql 用戶名 --password mysql 密碼 --port 端口號 --workdir 指定建立relaylog的硬連接的位置,默認是/var/tmp,因爲系統不一樣分區建立硬連接文件會失敗,故須要執行硬連接具體位置,成功執行腳本後,硬連接的中繼日誌文件被刪除 --disable_relay_log_purge 默認狀況下,若是relay_log_purge=1,腳本會什麼都不清理,自動退出,經過設定這個參數,
相關文章
相關標籤/搜索