1、實驗拓撲前端
2、拓撲說明node
分別在兩個節點上實現部署MySQL,數據庫文件存儲在後端的NFS主機上,經過掛載至相應的節點上使用,在兩個節點分別安裝corosync和pacemaker用於實現MySQL的高可用,經過crmsh程序對pacemaker進行配置,當其中一個節點出現問題時用於前端訪問的VIP地址將被移到另外一個節點上,並掛載後端的NFS數據庫存儲文件,而後啓動MySQL數據庫程序,以實如今兩個節點上實現MySQL高可用。mysql
3、架構佈置linux
服務器:CentOS 6.6 x86_64;web
數據庫IP地址即VIP:172.16.9.100;sql
兩個節點分別是:node-0二、node-03;相應的IP地址分別爲:172.16.9.82,172.16.9.83;數據庫
NFS服務器:IP地址172.16.9.8四、主機名爲:node-04; 後端
網關服務器:提供時間服務器,網關地址爲:172.16.0.1安全
MySQL版本:mariadb-5.5.43-linux-x86_64.tar.gzbash
corosync版本:corosync-1.4.7-1.el6.x86_64
pacemaker版本:pacemaker-1.1.12-4.el6.x86_64
crmsh版本:crmsh-2.1-1.6.x86_64.rpm
crmsh依賴包:pssh-2.3.1-2.el6.x86_64.rpm
4、準備工做
在構建高可用集羣服務器時須要作四個準備工做,分別是:
①節點間時間必須同步:使用ntp協議實現;
②節點間須要經過主機名互相通訊,必須解析主機至IP地址;
(a)建議名稱解析功能使用hosts文件來實現;
(b)通訊中使用的名字與節點名字必須保持一致:「uname -n」命令,或「hostname」展現出的名字保持一致;
③考慮仲裁設備是否會用到;
④創建各節點之間的root用戶可以基於密鑰認證;
1)配置節點時間同步
配置時間同步使用ntpdate命令,創建一個定時任務,實現週期性的時間同步
[root@node-02 ~]# ntpdate 172.16.0.1 3Jun 08:56:53 ntpdate[1655]: step time server 172.16.0.1 offset 22520.390088 sec [root@node-02 ~]# crontab -l */3 * * * * /usr/sbin/ntpdate 172.16.0.1&>/dev/null [root@node-03 ~]# /usr/sbin/ntpdate172.16.0.1 3Jun 08:57:50 ntpdate[2094]: step time server 172.16.0.1 offset 23311.837688 sec [root@node-03 ~]# crontab -l */3 * * * * /usr/sbin/ntpdate 172.16.0.1&>/dev/null
2)節點間基於主機名互相通訊,在/etc/hosts文件中進行配置
[root@node-02 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 172.16.0.1 server.magelinux.com server 172.16.9.82 node-02 node2 172.16.9.83 node-03 node3 [root@node-03 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 172.16.0.1 server.magelinux.com server 172.16.9.82 node-02 node2 172.16.9.83 node-03 node3
3)節點之間基於root用戶的密鑰認證
[root@node-02 ~]# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key(/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in/root/.ssh/id_rsa. Your public key has been saved in/root/.ssh/id_rsa.pub. The key fingerprint is: 93:c1:f9:42:63:cd:2c:98:b3:a9:ef:7e:02:24:db:f2root@node-02 The key's randomart p_w_picpath is: +--[ RSA 2048]----+ | | | + = | | + O + | | .. * * | | = o S . | | oo. o | | o.. | | E.. . | | o+o | +-----------------+ [root@node-02 ~]# ssh-copy-id -i.ssh/id_rsa.pub node3 Warning: Permanently added 'node3' (RSA) tothe list of known hosts. root@node3's password: Now try logging into the machine, with"ssh 'node3'", and check in: .ssh/authorized_keys to make sure we haven't added extra keysthat you weren't expecting. [root@node-03 ~]# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key(/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in/root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: b7:b1:3b:ca:78:9c:bc:72:0e:a0:18:a6:8d:ac:d0:99root@node-03 The key's randomart p_w_picpath is: +--[ RSA 2048]----+ | | | | | | | | |.. . S o | |+* + . . + | |=.E .o .o | |o .+* .. | |. .==o.. | +-----------------+ [root@node-03 ~]# [root@node-03 ~]# ssh-copy-id -i .ssh/id_rsa.pub node2 root@node2's password: Now try logging into the machine, with"ssh 'node2'", and check in: .ssh/authorized_keys to make sure we haven't added extra keysthat you weren't expecting.
測試節點間是否名密鑰登錄
[root@node-02 ~]# ssh node3 Last login: Wed Jun 3 02:28:06 2015 from 172.16.9.9 [root@node-03 ~]# exit logout Connection to node3 closed. [root@node-02 ~]# [root@node-03 ~]# ssh node2 Last login: Wed Jun 3 02:41:17 2015 from 172.16.9.9 [root@node-02 ~]# exit logout Connection to node2 closed. [root@node-03 ~]#
5、配置共享存儲NFS
5.1 配置NFS
[root@node-04 ~]# mkdir /mydata/data -p [root@node-04 ~]# cat /etc/exports /web/htdoc 172.16.0.0/16(rw) /mydata/data 172.16.0.0/16(rw,no_root_squash)
#提示:建議在安裝完數據庫後所no_root_squash選擇取消了,使用選擇太危險。
5.2 設置共享文件/mydata/data文件的屬主和屬組
[root@node-04 ~]# userdel -r mysql [root@node-04 ~]# useradd -r -u 336 mysql [root@node-04 ~]# id mysql uid=336(mysql) gid=336(mysql)groups=336(mysql) [root@node-04 ~]# chown -R mysql.mysql/mydata/data/
5.3啓動NFS服務
[root@node-04 ~]# service rpcbind start [root@node-04 ~]# service nfs start Starting NFS services: [ OK ] Starting NFS quotas: [ OK ] Starting NFS mountd: [ OK ] Starting NFS daemon: [ OK ] Starting RPC idmapd: [ OK ]
5.4 查看nfs共享存儲文件
[root@node-04 ~]# showmount -e 172.16.9.84 Export list for 172.16.9.84: /mydata/data 172.16.0.0/16
6、安裝佈置MySQL數據庫
MySQL使用的是MairaDB的數據庫,只有在兩個節點中的其中一個節點初始化數據庫就行,由於兩個節點都是共享提供的一個數據庫文件,如安裝初始化數據庫在node2上進行操做,在node3節點上不用初始化數據庫其它操都是同樣的,這裏就不給出操做過程。
6.1 建立MariaDB運行的用戶
[root@node-02 ~]# useradd -r -u 336 mysql [root@node-02 ~]# id mysql uid=336(mysql) gid=336(mysql)groups=336(mysql)
6.2 掛載NFS共享數據庫目錄
[root@node-02 ~]# mkdir /data [root@node-02 ~]# mount -t nfs172.16.9.84:/mydata/data /data [root@node-02 ~]# mount |tail -1 172.16.9.84:/mydata/data on /data type nfs(rw,vers=4,addr=172.16.9.84,clientaddr=172.16.9.82)
6.3 解壓MariaDB程序包到/usr/local目錄下
[root@node-2 tools]# tar xfmariadb-5.5.43-linux-x86_64.tar.gz -C /usr/local/
6.4 建立軟連接
[root@node-2 tools]# cd /usr/local/ [root@node-2 local]# ln -smariadb-5.5.43-linux-x86_64/ mysql
6.5 初始化數據庫
[root@node-2 local]# cd mysql [root@node-2 mysql]# chown -R root.mysql ./* [root@node-2 mysql]# scripts/mysql_install_db--datadir=/data --user=mysql
6.6 提供MySQL的主配置文件
[root@node-2 mysql]# mkdir /etc/mysql [root@node-2 mysql]# cpsupport-files/my-large.cnf /etc/mysql/my.cnf
6.7 編輯/etc/mysql/my.cnf配置文件
在/etc/mysql/my.cnf配置文件中在[mysqld]標籤中添加數據庫存放目錄。 datadir = /data innodb_file_per_table= on skip_name_resolve = on
6.8 爲MySQL提供服務腳本
[root@node-2 mysql]# cpsupport-files/mysql.server /etc/rc.d/init.d/mysqld [root@node-2 mysql]# chmod +x/etc/rc.d/init.d/mysqld [root@node-2 mysql]# chkconfig --add mysqld [root@node-2 mysql]# chkconfig mysqld off
6.9 啓動MariaDB服務進行測試
[root@node-02 mysql]# service mysqld start Starting MySQL.... [ OK ] [root@node-02 mysql]# bin/mysql Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 2 Server version: 5.5.43-MariaDB-log MariaDBServer Copyright (c) 2000, 2015, Oracle, MariaDBCorporation Ab and others. Type 'help;' or '\h' for help. Type '\c' toclear the current input statement. MariaDB [(none)]> create database node2; Query OK, 1 row affected (0.03 sec) MariaDB [(none)]> flush privileges; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | node2 | | performance_schema | | test | +--------------------+ 5 rows in set (0.01 sec) MariaDB [(none)]> exit Bye
6.10 中止MySQL服務,卸載NFS共享存儲/data
[root@node-02 mysql]# service mysqld stop Shutting down MySQL. [ OK ] [root@node-02 mysql]# umount /data
7、安裝配置HA程序
7.1 安裝corosync和pacemaker程序
在配置好yum源以後直接安裝corosync和pacemaker程序,分別兩個節點上執行yum 命令安裝
yum install corosync pacemaker -y
7.2 corosync默認配置文件解釋
corosync程序安裝後配置文件所在目錄爲/etc/corosync目錄下,啓動程序/etc/init.d/corosync.
[root@node-02 ~]# cd /etc/corosync/ [root@node-02 corosync]# ls corosync.conf.example corosync.conf.example.udpu service.d uidgid.d [root@node-02 corosync]# cpcorosync.conf.example corosync.conf [root@node-02 corosync]# egrep -v "#|^$" corosync.conf compatibility: whitetank #是否兼容whitetank totem { #用於定義底層信息層是如何通訊的相關屬性 version:2 #定義版本號 secauth:off #是否啓用安全認證功能,啓用後要使用corosync-keygen命令生成密鑰 threads:0 #工做時所使用的線程數,「0」表示不基於線程模型,而是進程模型 interface{ #定義多個接口之間,基於哪一個地址,哪一個多播地址,監聽什麼端口完成多播通訊; ringnumber:0 #環數,有點類型於TTL值對方是否回傳 bindnetaddr:192.168.1.0 #多播地址監聽的IP網絡地址 mcastaddr:239.255.1.1 #多播地址 mcastport:5405 #多播地址監聽的端口 ttl:1 #指明TTL值 } } logging { #定義日誌相關屬性 fileline:off # to_stderr:no #是否把日誌輸出爲標準輸出即屏幕 to_logfile:yes #開啓記錄在日誌文件中 logfile:/var/log/cluster/corosync.log to_syslog:yes #是否發往系統的日誌文件中 debug:off timestamp:on #是否在日誌文件中開啓時間戳功能,建議不開啓 logger_subsys{ #日誌文件是否記錄子系統 subsys:AMF debug:off } }
7.3 配置pacemaker
pacemaker與corosync結合運行pacemaker的運行方式有兩種,一種是做爲corosync的插件運行,另外一種是以獨立的守護進程運行,以CentOS 6中建議以插件的方式運行,不過這樣日誌中可能會用警告,能夠忽略的。在corosync.conf文件後面添加以下內容:
service { ver: 0 name: pacemaker use_mgmtd:yes } aisexec { user: root grout: root }
7.4爲corosync提供密鑰文件,它須要在/dev/random中讀取1024個隨機數
[root@node-02 corosync]# corosync-keygen Corosync Cluster Engine Authentication keygenerator. Gathering 1024 bits for key from/dev/random. Press keys on your keyboard to generateentropy. Press keys on your keyboard to generateentropy (bits = 176).
#此時已經卡住了,說沒有這麼多個隨機數,能夠在打開一個終端,不斷的敲擊鍵盤,不過這麼有一點的久,你能夠在ftp下載一個大的文件,這樣會產生大量的IO。
7.5 corosync+pacemaker最終配置文件
[root@node-02 corosync]# egrep -v "#|^$" corosync.conf compatibility: whitetank totem { version:2 secauth:on threads:0 interface{ ringnumber:0 bindnetaddr:172.16.0.0 mcastaddr:239.255.9.9 mcastport:5405 ttl:1 } } logging { fileline:off to_stderr:no to_logfile:yes logfile:/var/log/cluster/corosync.log to_syslog:no debug:off timestamp:on logger_subsys{ subsys:AMF debug:off } } service { ver: 0 name: pacemaker use_mgmtd:yes } aisexec { user: root grout: root }
7.6 將配置文件和密鑰文件同步至node3節點
[root@node-02 corosync]# scp authkeycorosync.conf node3:/etc/corosync/ authkey 100% 128 0.1KB/s 00:00 corosync.conf 100% 2794 2.7KB/s 00:00
7.7 啓動corosync服務
[root@node-02 corosync]# service corosyncstart;ssh node3 'service corosync start' Starting Corosync Cluster Engine(corosync): [ OK ] Starting Corosync Cluster Engine(corosync): [ OK ]
7.8安裝crmsh
把準備好的程序直接使用yum進行安裝,這樣能夠解決依賴關係,在生產環境中只須要選擇一臺節點上進行安裝,在這裏咱們在兩個節點上都進行安裝,以方便測試。
[root@node-02 ~]# yum installcrmsh-2.1-1.6.x86_64.rpm pssh-2.3.1-2.el6.x86_64.rpm -y
8、配置高可用MySQL服務
8.1 初始化配置
[root@node-02 ~]# crm #切換至crm命令提示符 crm(live)# configure #切換至配置模式 crm(live)configure# propertystonith-enabled=false #禁用stonith設備,由於咱們這裏沒有stonith設備全部要禁用 crm(live)configure# propertyno-quorum-policy=ignore #忽略集羣中當節點數小於等於quorum,節點數將沒法運行,默認是stop crm(live)configure# verify #檢驗語法 crm(live)configure# commit #提交併保存服務當即生效
8.2 配置VIP資源
crm(live)configure# primitive mysqlipocf:heartbeat:IPaddr \ params ip=172.16.9.100 nic=eth0cidr_netmask=16 \ op monitor interval=10s timeout=20s crm(live)configure# verify #primitive :配置主資源即基本資源 #mysqlip :資源名,爲VIP的地址 # ocf:heartbeat:IPaddr :表示爲ocf風格的heartbeat中的IPaddr,用於設置IP地址 #parmas :參數,即ocf:heartbeat:IPaddr選項中的要進行配置的值 #ip=172.16.9.100 :設置IP地址爲172.6.9.100 #nic :把VIP設置在哪塊網卡上,可省 #cidr_netmask=16 :使用cidr風格的子網掩碼格式 #op :表示此資源帶的選項 #monitor :爲監控操做 # interval :每隔多少時間監控一次 # timeout :每次監控超時時間
8.3 配置nfs掛載資源
crm(live)configure# primitive mysqlnfsocf:heartbeat:Filesystem \ paramsdevice="172.16.9.84:/mydata/data" directory="/data"fstype=nfs \ op monitor interval=20s timeout=40s opstart timeout=60s op stop timeout=60s crm(live)configure# verify # ocf:heartbeat:Filesystem : 示爲ocf風格的heartbeat中的文件系統 # device="172.16.9.84:/mydata/data" :設備路徑 # directory="/data" :掛載點 # fstype=nfs :文件系統類型
8.4 配置mysql服務資源
crm(live)configure# primitive mysqlserverlsb:mysqld op monitor interval=20s timeout=40s crm(live)configure# verify
8.5 定義資源之間的啓動順序
只有在先啓動VIP地址才能掛載NFS文件系統,掛載成功後才能啓動MySQL服務,這裏經過group的方式來進行定義資源之間的啓動順序。
crm(live)configure# group mysqlservicemysqlip mysqlnfs mysqlserver crm(live)configure# verify crm(live)configure# commit
9、測試資源
9.1查看集羣資源的運行狀態
[root@node-02 ~]# crm status Last updated: Wed Jun 3 11:18:21 2015 Last change: Wed Jun 3 11:15:35 2015 Stack: classic openais (with plugin) Current DC: node-02 - partition with quorum #當前DC,擁不擁法定票數 Version: 1.1.11-97629de 2 Nodes configured, 2 expected votes #有幾個節點,有幾票 3 Resources configured #當前配置的資源數 Online: [ node-02 node-03 ] #在線的節點 Resource Group: mysqlservice mysqlip (ocf::heartbeat:IPaddr): Started node-02 mysqlnfs (ocf::heartbeat:Filesystem): Started node-02 mysqlserver (lsb:mysqld): Started node-02
9.2 測試MySQL服務
[root@node-02 ~]# /usr/local/mysql/bin/mysql Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 2 Server version: 5.5.43-MariaDB-log MariaDBServer Copyright (c) 2000, 2015, Oracle, MariaDBCorporation Ab and others. Type 'help;' or '\h' for help. Type '\c' toclear the current input statement. MariaDB [(none)]> create databasetestnode2; #建立測試數據庫testnode2 Query OK, 1 row affected (0.02 sec) MariaDB [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | node2 | | performance_schema | | test | | testnode2 | +--------------------+ 6 rows in set (0.13 sec)
9.2 將節點node2設置爲備用節點
把節點node2設置爲備用節點後資源將會轉移至node3節點,而後將node2節點上線,此時資源並不會轉移至node2節點上,由於沒有配置資源對節點的傾向性或資源之間在一塊兒的參數。
[root@node-02 ~]# crm node standby [root@node-02 ~]# crm status Last updated: Wed Jun 3 11:25:28 2015 Last change: Wed Jun 3 11:25:21 2015 Stack: classic openais (with plugin) Current DC: node-02 - partition with quorum Version: 1.1.11-97629de 2 Nodes configured, 2 expected votes 3 Resources configured Node node-02: standby #節點node2已是standby的狀態 Online: [ node-03 ] Resource Group: mysqlservice #資源都已經轉移至node3節點 mysqlip (ocf::heartbeat:IPaddr): Started node-03 mysqlnfs (ocf::heartbeat:Filesystem): Started node-03 mysqlserver (lsb:mysqld): Started node-03
將node2上線,經過狀態發現資源並無轉移回至node3節點
[root@node-02 ~]# crm node online [root@node-02 ~]# crm status Last updated: Wed Jun 3 11:27:13 2015 Last change: Wed Jun 3 11:27:10 2015 Stack: classic openais (with plugin) Current DC: node-02 - partition with quorum Version: 1.1.11-97629de 2 Nodes configured, 2 expected votes 3 Resources configured Online: [ node-02 node-03 ] Resource Group: mysqlservice mysqlip (ocf::heartbeat:IPaddr): Started node-03 mysqlnfs (ocf::heartbeat:Filesystem): Started node-03 mysqlserver (lsb:mysqld): Started node-03
9.3 在節點node3上進行測試MySQL服務
[root@node-03 ~]#/usr/local/mysql/bin/mysql Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 2 Server version: 5.5.43-MariaDB-log MariaDBServer Copyright (c) 2000, 2015, Oracle, MariaDBCorporation Ab and others. Type 'help;' or '\h' for help. Type '\c' toclear the current input statement. MariaDB [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | node2 | | performance_schema | | test | | testnode2 | +--------------------+ 6 rows in set (0.04 sec) MariaDB [(none)]> create databasetestnode3; Query OK, 1 row affected (0.07 sec) MariaDB [(none)]> flush privileges; Query OK, 0 rows affected (0.01 sec) MariaDB [(none)]>
經過上面的測試MySQL已經在節點node二、node3上能實現高可用,在節點node2上建立的數據庫文件在掛載至節點node3上並無丟失。
9.4 模擬數據庫不當心關閉服務
這裏經過直接把mysqld進程直接killall,檢查HA會不會再次啓動MySQL服務
[root@node-03 ~]# ss -tanp|grep":3306" #查看MySQL進程 LISTEN 0 50 *:3306 *:* users:(("mysqld",5781,15)) [root@node-03 ~]# killall mysqld #killall全部的MySQL進程 [root@node-03 ~]# killall mysqld mysqld: no process killed [root@node-03 ~]# ss -tanp|grep":3306" [root@node-03 ~]# ss -tanp|grep":3306" [root@node-03 ~]# ss -tanp|grep ":3306" [root@node-03 ~]# crm status #查看集羣的狀態,MySQL資源已經中止 Last updated: Wed Jun 3 11:41:45 2015 Last change: Wed Jun 3 11:41:24 2015 Stack: classic openais (with plugin) Current DC: node-02 - partition with quorum Version: 1.1.11-97629de 2 Nodes configured, 2 expected votes 3 Resources configured Online: [ node-02 node-03 ] Resource Group: mysqlservice mysqlip (ocf::heartbeat:IPaddr): Started node-03 mysqlnfs (ocf::heartbeat:Filesystem): Started node-03 mysqlserver (lsb:mysqld): Stopped [root@node-03 ~]# ss -tanp|grep":3306" [root@node-03 ~]# ss -tanp|grep":3306" [root@node-03 ~]# ss -tanp|grep":3306" [root@node-03 ~]# ss -tanp|grep":3306" [root@node-03 ~]# ss -tanp|grep":3306" #先行幾秒後MySQL服務再次自動的被集羣啓動 LISTEN 0 50 *:3306 *:* users:(("mysqld",11361,15)) [root@node-03 ~]# crm status Last updated: Wed Jun 3 11:37:36 2015 Last change: Wed Jun 3 11:27:10 2015 Stack: classic openais (with plugin) Current DC: node-02 - partition with quorum Version: 1.1.11-97629de 2 Nodes configured, 2 expected votes 3 Resources configured Online: [ node-02 node-03 ] Resource Group: mysqlservice mysqlip (ocf::heartbeat:IPaddr): Started node-03 mysqlnfs (ocf::heartbeat:Filesystem): Started node-03 mysqlserver (lsb:mysqld): Started node-03 [root@node-03 ~]# mysql Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.5.43-MariaDB-log MariaDBServer Copyright (c) 2000, 2013, Oracle and/or itsaffiliates. All rights reserved. Oracle is a registered trademark of OracleCorporation and/or its affiliates. Other names may be trademarksof their respective owners. Type 'help;' or '\h' for help. Type '\c' toclear the current input statement. mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | node2 | | performance_schema | | test | | testnode2 | | testnode3 | +--------------------+ 7 rows in set (0.02 sec)
小結:
一個簡單的MySQL高可用就配置完畢,NFS服務器將是集羣節點中的一個單點故障所在,NFS所能實現的併發訪問量也是有現的,整個架構並不能實現完整的高可用,還有不少地方須要改進。
歡迎各位觀客爲小烏提出寶貴的意見,小烏等待你。。。。。
奮鬥的年紀,毫不能怠慢本身!