DRBD+Heartbeat+Mysql高可用讀寫分離架構

聲明：本案例僅爲評估測試版本

注意：全部服務器之間必須作好時間同步

架構拓撲

IP信息：

Heartbeat安裝部署

1.安裝heartbeat（主備節點同時安裝）

[root@master1 ~]# yum install heartbeat* -y
[root@master2 ~]# yum install heartbeat* -y

heartbeat的yum安裝完的默認路徑爲 /etc/ha.d/

[root@master1 ~]# cd /etc/ha.d/
[root@master1 ha.d]# ls -al
total 28
drwxr-xr-x   4 root root 4096 May  3 19:08 .
drwxr-xr-x. 67 root root 4096 May  3 19:08 ..
-rwxr-xr-x   1 root root  745 Dec  3  2013 harc
drwxr-xr-x   2 root root 4096 May  3 19:08 rc.d
-rw-r--r--   1 root root  692 Dec  3  2013 README.config
drwxr-xr-x   2 root root 4096 May  3 19:08 resource.d
-rw-r--r--   1 root root 2082 Mar 24 21:01 shellfuncs

通常安裝任何軟件注意閱讀README文件

從readme文件咱們能夠獲取到三個重要文件：

• ha.cf：主配置文件
• haresources：資源配置文件
• authkeys：認證信息文件
  此三類文件在主備服務器上保持徹底一致

[root@master1 ha.d]# cat README.config 
You need three configuration files to make heartbeat happy,
and they all go in this directory.

They are:
    ha.cf       Main configuration file
    haresources Resource configuration file
    authkeys    Authentication information

These first two may be readable by everyone, but the authkeys file
must not be.

The good news is that sample versions of these files may be found in
the documentation directory (providing you installed the documentation).

If you installed heartbeat using rpm packages then
this command will show you where they are on your system:
        rpm -q heartbeat -d

If you installed heartbeat using Debian packages then
the documentation should be located in /usr/share/doc/heartbeat

須要將此文件從 /usr/share/doc/heartbeat 下copy到 /etc/ha.d/ 目錄下

[root@master1 ~]# cd /usr/share/doc/heartbeat-3.0.4/
[root@master1 heartbeat-3.0.4]# cp authkeys ha.cf haresources /etc/ha.d/

2.配置ha.cf

ha.cf文件詳解請參考：http://blog.csdn.net/lgstudyvc/article/details/48341185

[root@master1 ha.d]# egrep -v "^$|#" ha.cf 
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 30
warntime 10
initdead 120
mcast eth0:2 225.0.0.1 694 1 0
auto_failback on
node    master1
node    master2

3.配置authkeys

[root@master1 ha.d]# cat authkeys 
#
#   Authentication file.  Must be mode 600
#
#
#   Must have exactly one auth directive at the front.
#   auth    send authentication using this method-id
#
#   Then, list the method and key that go with that method-id
#
#   Available methods: crc sha1, md5.  Crc doesn't need/want a key.
#
#   You normally only have one authentication method-id listed in this file
#
#   Put more than one to make a smooth transition when changing auth
#   methods and/or keys.
#
#
#   sha1 is believed to be the "best", md5 next best.
#
#   crc adds no security, except from packet corruption.
#       Use only on physically secure networks.
#
auth 2
#1 crc
2 sha1 HI!
#3 md5 Hello!
[root@master1 ha.d]# chmod 600 authkeys #必須修改文件權限爲600

4.配置haresources

master1 IPaddr::10.0.0.130/24/eth0:3 drbddisk::data Filesystem::/dev/drbd1::/data::ext3 mysqld
#添加主節點爲master設置vip爲10.0.0.130/24,網卡爲eth0:3
#drbddisk掛在data目錄
#fliesystem將drbd1目錄掛在到/data目錄下，而且啓動mysql服務

DRBD安裝部署

1.兩臺主庫數據分區信息

2.對/dev/sdb進行分區

[root@master1 ~]# fdisk /dev/sdb 
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x0e4d1c28.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-130, default 1): 
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-130, default 130): +500M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (66-130, default 66): 
Using default value 66
Last cylinder, +cylinders or +size{K,M,G} (66-130, default 130): 
Using default value 130

Command (m for help): p

Disk /dev/sdb: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0e4d1c28

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1          65      522081   83  Linux
/dev/sdb2              66         130      522112+  83  Linux

3.爲新分區建立文件系統

注意：sdb2做爲drbd的狀態信息分區不須要格式化建立文件系統，因此只對sdb1進行格式化建立文件系統便可

[root@master1 ~]# mkfs.ext3 /dev/sdb1 
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
130560 inodes, 522080 blocks
26104 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67633152
64 block groups
8192 blocks per group, 8192 fragments per group
2040 inodes per group
Superblock backups stored on blocks: 
    8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409

Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 33 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@master1 ~]# tune2fs -c -1 /dev/sdb1 #只容許一個設備進行掛載
tune2fs 1.41.12 (17-May-2010)
Setting maximal mount count to -1

4.掛在/dev/sdb1分區到/data目錄下，給後續mysql服務器存放data數據

[root@master1 ~]# mount /dev/sdb
sdb   sdb1  sdb2  
[root@master1 ~]# mount /dev/sdb1 /data/
[root@master1 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_master1-lv_root
                      6.5G  2.2G  4.1G  35% /
tmpfs                 491M     0  491M   0% /dev/shm
/dev/sda1             477M   28M  425M   7% /boot
/dev/sdb1             494M   11M  458M   3% /data

5.安裝drbd內核模塊

[root@master1 ~]# rpm -ivh http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm
[root@master1 ~]# yum install drbd84* kmod-drbd84

6.確認drbd模塊已經加載到內核

[root@master1 ~]# modprobe drbd
[root@master1 ~]# lsmod | grep drbd
drbd                  374888  0 
libcrc32c               1246  1 drbd

7.查看drbd的配置文件

查看配置文件可知，須要單首創建響應的.res資源文件，且有相關的實例文件

[root@master1 etc]# cat drbd.conf 
# You can find an example in  /usr/share/doc/drbd.../drbd.conf.example

include "drbd.d/global_common.conf";
include "drbd.d/*.res";

[root@master1 drbd.d]# cat /etc/drbd.d/mysql.res 
global {
# minor-count 64;
# dialog-refresh 5; # 5 seconds
# disable-ip-verification;
usage-count no;
}
common {
protocol C;
disk {
on-io-error   detach;
#size 454G;
no-disk-flushes;
no-md-flushes;
}
net {
sndbuf-size 512k;
# timeout       60;    #  6 seconds  (unit = 0.1 seconds)
# connect-int   10;    # 10 seconds  (unit = 1 second)
# ping-int      10;    # 10 seconds  (unit = 1 second)
# ping-timeout   5;    # 500 ms (unit = 0.1 seconds)
max-buffers     8000;
unplug-watermark   1024;
max-epoch-size  8000;
# ko-count 4;
# allow-two-primaries;
cram-hmac-alg "sha1";
shared-secret "hdhwXes23sYEhart8t";
after-sb-0pri disconnect;
after-sb-1pri disconnect;
after-sb-2pri disconnect;
rr-conflict disconnect;
# data-integrity-alg "md5";
# no-tcp-cork;
}
syncer {
rate 120M;
al-extents 517;
}
}
resource data {
on master1 {
device     /dev/drbd1;
disk       /dev/sdb1;
address    10.0.2.110:7788;
meta-disk  /dev/sdb2 [0];
}
on master2 {
device     /dev/drbd1;
disk       /dev/sdb1;
address    10.0.0.2.120:7788;
meta-disk  /dev/sdb2 [0];
}
}

8.初始化meta分區

[root@master1 ~]# drbdadm create-md data
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.

9.初始化設備同步(覆蓋備節點，保持數據一致)

[root@master1 ~]# drbdadm -- --overwrite-data-of-peer primary data

10.啓動drbd

[root@master1 ~]# drbdadm up all
[root@master1 ~]# chkconfig drbd off

11.掛在drbd分區到data目錄下

[root@master1 ~]# drbdadm primary all
[root@master1 ~]# mount /dev/drbd1 /data
說明：/data目錄爲數據庫的數據目錄

安裝MySQL數據庫（兩臺主服務器上同時安裝）

1.將mysql數據的/data文件夾創建在/dev/sdb1分區的data目錄下

#到MySQL官網下載相應的mysql安裝包，能夠選擇rpm，yum方式，此處選擇二進制方式，方便定製安裝路徑
[root@master1 ~]# tar zxvf mysql-5.6.32-linux-glibc2.5-x86_64.tar.gz
#直接將解壓後的mysql文件copy到安裝目錄便可
[root@master1 ~]# mv mysql-5.6.32-linux-glibc2.5-x86_64 /usr/local/mysql-5.6.32
#建立軟鏈接不然後續利用二進制腳本啓動過程當中會保存，由於mysql.server腳本中引用的路徑爲/usr/local/mysql
[root@master1 ~]# ln -s /usr/local/mysql-5.6.32/ /usr/local/mysql 
#使用mysql自帶的mysql_install_db腳本進行初始化安裝
[root@master1 scripts]# useradd -s /sbin/nologin -M mysql
[root@master1 scripts]# ./mysql_install_db -user=mysql --basedir=/usr/local/mysql-5.6.32/ --datadir=/data/

2.修改my.cnf配置文件

[root@master1 mysql-5.6.32]# vim my.cnf 
[root@master1 mysql-5.6.32]# cat my.cnf 
# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.6/en/server-configuration-defaults.html

[mysqld]

# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M

# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin

# These are commonly set, remove the # and set as required.
basedir = /usr/local/mysql-5.6.32/
datadir = /data
port = 3306
server_id = 110
socket = /tmp/mysql.sock

# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M 

sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES

3.複製二進制mysql啓動文件到/etc/init.d/下，方便heartbeat調用

[root@master1 support-files]# cp mysql.server /etc/init.d/mysqld

4.啓動msyqld時出現報錯請刪除系統自帶的/etc/my.cnf配置文件

[root@master1 ~]# /etc/init.d/mysqld start
Starting MySQL... ERROR! The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
[root@master1 ~]# rm -rf /etc/my.cnf
[root@master1 ~]# /etc/init.d/mysqld start
Starting MySQL. SUCCESS!

5.設置mysql數據庫密碼

#複製mysql安裝文件下的bin文件到系統環境路徑下
[root@master1 bin]# mysqladmin -uroot password 
New password: 
Confirm new password:
[root@master1 ~]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.32 MySQL Community Server (GPL)

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+---------------------+
| Database            |
+---------------------+
| information_schema  |
| #mysql50#lost+found |
| mysql               |
| performance_schema  |
| test                |
+---------------------+
5 rows in set (0.01 sec)

Heartbeat+Mysql+DRBD故障測試：

1.主備節點IP信息：

主節點IP：

[root@master1 /]# ifconfig 
eth0      Link encap:Ethernet  HWaddr 00:0C:29:8A:B3:73  
          inet addr:10.0.0.110  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe8a:b373/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4055 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3888 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:494029 (482.4 KiB)  TX bytes:1890864 (1.8 MiB)

eth0:1    Link encap:Ethernet  HWaddr 00:0C:29:8A:B3:73  
          inet addr:10.0.1.110  Bcast:10.0.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

eth0:2    Link encap:Ethernet  HWaddr 00:0C:29:8A:B3:73  
          inet addr:10.0.2.110  Bcast:10.0.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

eth0:3    Link encap:Ethernet  HWaddr 00:0C:29:8A:B3:73  
          inet addr:10.0.0.130  Bcast:10.0.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

備節點IP：

[root@master2 /]# ifconfig 
eth0      Link encap:Ethernet  HWaddr 00:0C:29:85:D5:3D  
          inet addr:10.0.0.120  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe85:d53d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:41412 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16511 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:47131293 (44.9 MiB)  TX bytes:3064356 (2.9 MiB)

eth0:1    Link encap:Ethernet  HWaddr 00:0C:29:85:D5:3D  
          inet addr:10.0.1.120  Bcast:10.0.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

eth0:2    Link encap:Ethernet  HWaddr 00:0C:29:85:D5:3D  
          inet addr:10.0.2.120  Bcast:10.0.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

2.主備節點drbd信息

主節點：

[root@master1 /]# cat /proc/drbd 
version: 8.4.9-1 (api:1/proto:86-101)
GIT-hash: 9976da086367a2476503ef7f6b13d4567327a280 build by mockbuild@Build64R6, 2016-12-13 18:38:15

 1: cs:Connected ro:Primary/Secondary ds:UpToDate/Diskless C r-----
    ns:1326 nr:26 dw:594 dr:28007 al:16 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:522084

備節點：

[root@master2 /]# cat /proc/drbd 
version: 8.4.9-1 (api:1/proto:86-101)
GIT-hash: 9976da086367a2476503ef7f6b13d4567327a280 build by mockbuild@Build64R6, 2016-12-13 18:38:15

 1: cs:Connected ro:Secondary/Primary ds:Diskless/UpToDate C r-----
    ns:26 nr:1326 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

3.主備節點data目錄信息

主節點：

[root@master2 data]# ll
total 110616
-rw-rw---- 1 mysql mysql       56 May  3 12:59 auto.cnf
-rw-rw---- 1 mysql mysql 12582912 May  3 14:08 ibdata1
-rw-rw---- 1 mysql mysql 50331648 May  3 14:08 ib_logfile0
-rw-rw---- 1 mysql mysql 50331648 May  3 12:58 ib_logfile1
-rw-r----- 1 mysql root      5849 May  3 14:08 master2.err
drwx------ 2 mysql mysql     4096 May  3 12:58 mysql
drwx------ 2 mysql mysql     4096 May  3 12:58 performance_schema
drwx------ 2 mysql mysql     4096 May  3 12:58 test

備節點：

[root@master1 data]# ll
total 111081
-rw-rw---- 1 mysql mysql       56 May  3 12:47 auto.cnf
-rw-rw---- 1 mysql mysql 12582912 May  3 14:18 ibdata1
-rw-rw---- 1 mysql mysql 50331648 May  3 14:18 ib_logfile0
-rw-rw---- 1 mysql mysql 50331648 May  3  2017 ib_logfile1
drwx------ 2 root  root     12288 May  3  2017 lost+found
-rw-r----- 1 mysql root     24692 May  3 14:18 master1.err
-rw-rw---- 1 mysql mysql        5 May  3 14:18 master1.pid
-rw-r----- 1   500 root      4907 May  3 14:08 master2.err
drwx------ 2 mysql mysql     4096 May  3  2017 mysql
drwx------ 2 mysql mysql     4096 May  3 12:41 performance_schema
drwx------ 2 mysql mysql     1024 May  3  2017 test