MariaDB做爲Mysql的一個分支，在開源項目中已經普遍使用，例如大熱的openstack，因此，爲了保證服務的高可用性，同時提升系統的負載能力，集羣部署是必不可少的。
MariaDB Galera Cluster 介紹node
Galera Cluster是由第三方公司Codership所研發的一套免費開源的集羣高可用方案，實現了數據零丟失，官網地址爲http://galeracluster.com/。其在MySQLInnoDB存儲引擎基礎上打了wrep(虛擬全同步複製)，Percona/MariaDB已捆綁在各自的發行版本中。mysql
MariaDB Galera Cluster是MariaDB同步多主機集羣。它僅支持XtraDB/InnoDB存儲引擎（雖然有對MyISAM實驗支持，具體看wsrep_replicate_myisam系統變量）。linux
MariaDB Galera Cluster主要功能：sql
l 同步複製數據庫
l 真正的multi-master，即全部節點能夠同時讀寫數據庫bootstrap
l 自動的節點成員控制，失效節點自動被清除centos
l 新節點加入數據自動複製緩存
l 真正的並行複製，行級安全
l 用戶能夠直接鏈接集羣，使用感覺上與MySQL徹底一致bash
優點：
l 由於是多主，因此不存在Slavelag(延遲)
l 不存在丟失事務的狀況
l 同時具備讀和寫的擴展能力
l 更小的客戶端延遲
l 節點間數據是同步的，而Master/Slave模式是異步的，不一樣slave上的binlog多是不一樣的
缺點：
l 加入新節點時開銷大，須要複製完整的數據
l 不能有效地解決寫擴展的問題，全部的寫操做都發生在全部的節點
l 有多少個節點，就有多少份重複的數據
l 因爲事務提交須要跨節點通訊，即涉及分佈式事務操做，所以寫入會比主從複製慢不少，節點越多，寫入越慢，死鎖和回滾也會更加頻繁
l 對網絡要求比較高，若是網絡出現波動不穩定，則可能會形成兩個節點失聯，Galera Cluster集羣會發生腦裂，服務將不可用
還有一些地方存在侷限：
l 僅支持InnoDB/XtraDB存儲引擎，任何寫入其餘引擎的表，包括mysql.*表都不會被複制。可是DDL語句能夠複製，可是insert into mysql.user(MyISAM存儲引擎)之類的插入數據不會被複制
l Delete操做不支持沒有主鍵的表，由於沒有主鍵的表在不一樣的節點上的順序不一樣，若是執行select … limit …將出現不一樣的結果集
l LOCK/UNLOCK TABLES/FLUSH TABLES WITH READ LOCKS不支持單表所鎖，以及鎖函數GET_LOCK()、RELEASE_LOCK()，但FLUSH TABLES WITH READ LOCK支持全局表鎖
l General Query Log日誌不能保存在表中，若是開始查詢日誌，則只能保存到文件中
l 不能有大事務寫入，不能操做wsrep_max_ws_rows=131072(行)，且寫入集不能超過wsrep_max_ws_size=1073741824(1GB)，不然客戶端直接報錯
l 因爲集羣是樂觀鎖併發控制，所以，在commit階段會有事務衝突發生。若是兩個事務在集羣中的不一樣節點上對同一行寫入並提交，則失敗的節點將回滾，客戶端返回死鎖報錯
l XA分佈式事務不支持Codership Galera Cluster，在提交時可能會回滾
l 整個集羣的寫入吞吐量取決於最弱的節點限制，集羣要使用同一的配置
技術：
Galera集羣的複製功能是基於認證的複製，其流程以下：
當客戶端發出一個commit的指令，在事務被提交以前，全部對數據庫的更改都會被write-set收集起來，而且將write-set 記錄的內容發送給其餘節點。
write-set 將在每一個節點上使用搜索到的主鍵進行確認性認證測試，測試結果決定着節點是否應用write-set更改數據。若是認證測試失敗，節點將丟棄 write-set ；若是認證測試成功，則事務提交，工做原理以下圖：
關於新節點的加入，流程以下：
新加入的節點叫作Joiner，給Joiner提供複製的節點叫Donor。在該過程當中首先會檢查本地grastate.dat文件的seqno事務號是否在遠端donor節點galera.cache文件裏，若是存在，那麼進行Incremental State Transfer(IST)增量同步複製，將剩餘的事務發送過去；若是不存在那麼進行State Snapshot Transfer(SST)全量同步複製。SST有三種全量拷貝方式：mysqldump、rsync和xtrabackup。SST的方法能夠經過wsrep_sst_method這個參數來設置。
備註：
SST是指從donor到joiner的數據全量拷貝，它一般使用在一個新的節點加入時，爲了與集羣同步，新的節點不得不去一個已經在集羣中的節點上拷貝數據，在PXC(Percona Xtradb Cluster)中，有三種SST的方法，mysqldump，rsync，Xtrabackup。
建議使用XtraBackup，另外對XtraBackup補充說明：
在XtraBackup 2.1.x版本里，使用innobackupex備份時，備份流程以下：
1. 備份InnoDB表數據
2. 執行全局表讀鎖FLUSH TABLES WITH READ LOCKS
3. 拷貝.frm和MyISAM表數據
4. 獲得當前的binlog文件名和position點
5. 完成redo log事務日誌的後臺複製
6. 解鎖UNLOCK TABLES
由上面能夠看出若是備份好幾張MyISAM存儲的大表時，將會進行鎖表。
2、環境準備
環境說明：安裝MariaDB集羣至少須要3臺服務器（若是隻有兩臺的話須要特殊配置，請參照官方文檔）
一、硬件規劃
二、系統版本
root@mariadb-node1 ~]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
三、關閉防火牆
[root@mariadb-node1 ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
[root@mariadb-node1 ~]# setenforce 0
setenforce: SELinux is disabled
四、selinux
[root@mariadb-node1 ~]# systemctl stop firewalld.service
[root@mariadb-node1 ~]# systemctl disable firewalld.service
五、配置域名解析
[root@mariadb-node1 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.120 mariadb-node1
192.168.1.121 mariadb-node2
192.168.1.122 mariadb-node3
六、加大文件描述符
vi /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
|
vi /etc/sysctl.conf
fs.file-max=655350
net.ipv4.ip_local_port_range = 1025 65000
net.ipv4.tcp_tw_recycle = 1
|
最後執行：
# sysctl -p
安裝Percona XtraBackup熱備份工具
下載地址：
tar -zxvf percona-xtrabackup-2.4.6-Linux-x86_64.tar.gz
cd percona-xtrabackup-2.4.6-Linux-x86_64/bin/
cp -a * /usr/bin/
|
建立XtraBackup備份時用的用戶名和密碼：
MariaDB [(none)]> grant all on *.* to 'galera'@'localhost' identified by '123456';
|
五、配置mariadb源
備註：從MariaDB 10.1.20 版本開始，Galera Cluster就已經包含在MariaDB包裏面了，不須要單獨部署MariaDB-Galera-server 和galera 包。
使用YUM方式部署MariaDB Galera Cluster。
#三臺機器同時配置mariadb源
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
#這裏用的是openstack-newton的源，裏面包含了mariadb
yum install centos-release-openstack-newton -y
#查看openstack-newton源是否存在
[root@mariadb-node1 ~]# cd /etc/yum.repos.d/
[root@mariadb-node1 yum.repos.d]# ll
total 52
-rw-r--r--. 1 root root 2573 Nov 21 2014 CentOS-Base.repo #阿里雲國內yum源
-rw-r--r--. 1 root root 1664 Dec 9 2015 CentOS-Base.repo.backup
-rw-r--r-- 1 root root 1056 Sep 6 2016 CentOS-Ceph-Jewel.repo
-rw-r--r--. 1 root root 1309 Dec 9 2015 CentOS-CR.repo
-rw-r--r--. 1 root root 649 Dec 9 2015 CentOS-Debuginfo.repo
-rw-r--r--. 1 root root 290 Dec 9 2015 CentOS-fasttrack.repo
-rw-r--r--. 1 root root 630 Dec 9 2015 CentOS-Media.repo
-rw-r--r-- 1 root root 1113 Jun 23 2017 CentOS-OpenStack-newton.repo
-rw-r--r-- 1 root root 509 Sep 12 22:11 CentOS-QEMU-EV.repo
-rw-r--r--. 1 root root 1331 Dec 9 2015 CentOS-Sources.repo
-rw-r--r--. 1 root root 1952 Dec 9 2015 CentOS-Vault.repo
-rw-r--r--. 1 root root 951 Oct 3 01:44 epel.repo #阿里雲國內epel源
-rw-r--r--. 1 root root 1050 Oct 3 01:44 epel-testing.repo
|
#更新緩存
|
1
2
|
yum clean all
yum makecache
|
3、安裝 MariaDB Galera Cluster （#備註：三臺機器同時操做，並把yum源改爲國內阿里源）
yum install mariadb mariadb-galera-server mariadb-galera-common galera rsync -y
|
#配置mariadb
下面咱們開始配置MariaDB Galera Cluster，分別修改MariaDB Galera集羣的每一個節點上的/etc/my.cnf.d/server.cnf文件，具體每一個節點的內容以下：
一、192.168.1.120節點的/etc/my.cnf.d/server.cnf文件內容：
[root@mariadb-node1 ~]# cat /etc/my.cnf.d/server.cnf
[server]
[mysqld]
server_id=129
datadir=/app/galera
user=mysql
skip-external-locking
skip-name-resolve
character-set-server=utf8
[galera]
wsrep_causal_reads=ON
wsrep_provider_options="gcache.size=4G"
wsrep_certify_nonPK=ON
query_cache_size=0
wsrep_on=ON
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_name=MariaDB-Galera-Cluster
wsrep_cluster_address="gcomm://192.168.1.120,192.168.1.121,192.168.1.122"
wsrep_node_name=mariadb-a04
wsrep_node_address=192.168.1.120
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
wsrep_slave_threads=8
innodb_flush_log_at_trx_commit=0
innodb_buffer_pool_size=2G
wsrep_sst_method=rsync
[embedded]
[mariadb]
[mariadb-10.1]
# 上面配置使用的是rsync方式同步數據，若是要使用xtrabackup方式（建議使用），須要設置：
wsrep_sst_auth=galera:123456
wsrep_sst_method=xtrabackup-v2 #默認是rsync全量拷貝，可是須要在donor節點上執行全局讀鎖(flushtables with read lock)，建議採用xtrabackup熱備份方式，只有在備份.frm表結構文件纔會鎖表
二、192.168.1.121節點的/etc/my.cnf.d/server.cnf文件內容：
[root@mariadb-node2 ~]# vi /etc/my.cnf.d/server.cnf
[server]
[mysqld]
server_id=129
datadir=/app/galera
user=mysql
skip-external-locking
skip-name-resolve
character-set-server=utf8
[galera]
wsrep_causal_reads=ON
wsrep_provider_options="gcache.size=4G"
wsrep_certify_nonPK=ON
query_cache_size=0
wsrep_on=ON
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_name=MariaDB-Galera-Cluster
wsrep_cluster_address="gcomm://192.168.1.120,192.168.1.121,192.168.1.122"
wsrep_node_name=mariadb-a04
wsrep_node_address=192.168.1.121
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
wsrep_slave_threads=8
innodb_flush_log_at_trx_commit=0
innodb_buffer_pool_size=2G
wsrep_sst_method=rsync
[embedded]
[mariadb]
[mariadb-10.1]
|
三、192.168.1.122節點的/etc/my.cnf.d/server.cnf文件內容：
[root@mariadb-node3 ~]# vi /etc/my.cnf.d/server.cnf
[server]
[mysqld]
server_id=130
datadir=/app/galera
user=mysql
skip-external-locking
skip-name-resolve
character-set-server=utf8
[galera]
wsrep_causal_reads=ON
wsrep_provider_options="gcache.size=4G"
wsrep_certify_nonPK=ON
query_cache_size=0
wsrep_on=ON
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_name=MariaDB-Galera-Cluster
wsrep_cluster_address="gcomm://192.168.1.120,192.168.1.121,192.168.1.122"
wsrep_node_name=mariadb-a05
wsrep_node_address=192.168.1.122
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
wsrep_slave_threads=8
innodb_flush_log_at_trx_commit=0
innodb_buffer_pool_size=2G
wsrep_sst_method=rsync
[embedded]
[mariadb]
[mariadb-10.1]
|
四、MariaDB一個節點初始化安裝（192.168.1.120）：
mysql_install_db --defaults-file=/etc/my.cnf.d/server.cnf --user=mysql
|
五、在192.168.1.120節點上經過bootstrap啓動（第一次啓動必定要使用--wsrep-new-cluster，再次啓動就不須要）
mysqld_safe --defaults-file=/etc/my.cnf.d/server.cnf --user=mysql --wsrep-new-cluster &
|
六、初始化Mariadb，設置root密碼與安全設置 (在192.168.1.120節點上面操做)
[root@mariadb-node1 ~]# mysql_install_db --defaults-file=/etc/my.cnf.d/server.cnf --user=mysql
Installing MariaDB/MySQL system tables in '/app/galera' ...
2017-12-24 3:42:27 139950307854528 [Note] /usr/libexec/mysqld (mysqld 10.1.20-MariaDB) starting as process 22527 ...
2017-12-24 3:42:27 139950307854528 [Note] WSREP: Read nil XID from storage engines, skipping position init
2017-12-24 3:42:27 139950307854528 [Note] WSREP: wsrep_load(): loading provider library 'none'
2017-12-24 3:42:28 139950307854528 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2017-12-24 3:42:28 139950307854528 [Note] InnoDB: The InnoDB memory heap is disabled
2017-12-24 3:42:28 139950307854528 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-12-24 3:42:28 139950307854528 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-12-24 3:42:28 139950307854528 [Note] InnoDB: Compressed tables use zlib 1.2.7
2017-12-24 3:42:28 139950307854528 [Note] InnoDB: Using Linux native AIO
2017-12-24 3:42:28 139950307854528 [Note] InnoDB: Using SSE crc32 instructions
2017-12-24 3:42:28 139950307854528 [Note] InnoDB: Initializing buffer pool, size = 2.0G
2017-12-24 3:42:29 139950307854528 [Note] InnoDB: Completed initialization of buffer pool
2017-12-24 3:42:30 139950307854528 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
2017-12-24 3:42:30 139950307854528 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
2017-12-24 3:42:30 139950307854528 [Note] InnoDB: Database physically writes the file full: wait...
2017-12-24 3:42:30 139950307854528 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
2017-12-24 3:42:30 139950307854528 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
2017-12-24 3:42:30 139950307854528 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2017-12-24 3:42:30 139950307854528 [Warning] InnoDB: New log files created, LSN=45883
2017-12-24 3:42:30 139950307854528 [Note] InnoDB: Doublewrite buffer not found: creating new
2017-12-24 3:42:30 139950307854528 [Note] InnoDB: Doublewrite buffer created
2017-12-24 3:42:30 139950307854528 [Note] InnoDB: 128 rollback segment(s) are active.
2017-12-24 3:42:30 139950307854528 [Warning] InnoDB: Creating foreign key constraint system tables.
2017-12-24 3:42:30 139950307854528 [Note] InnoDB: Foreign key constraint system tables created
2017-12-24 3:42:30 139950307854528 [Note] InnoDB: Creating tablespace and datafile system tables.
2017-12-24 3:42:30 139950307854528 [Note] InnoDB: Tablespace and datafile system tables created.
2017-12-24 3:42:30 139950307854528 [Note] InnoDB: Creating zip_dict and zip_dict_cols system tables.
2017-12-24 3:42:30 139950307854528 [Note] InnoDB: zip_dict and zip_dict_cols system tables created.
2017-12-24 3:42:30 139950307854528 [Note] InnoDB: Waiting for purge to start
2017-12-24 3:42:30 139950307854528 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.34-79.1 started; log sequence number 0
2017-12-24 3:42:30 139947298572032 [Note] InnoDB: Dumping buffer pool(s) not yet started
OK
Filling help tables...
2017-12-24 3:42:34 139725903902912 [Note] /usr/libexec/mysqld (mysqld 10.1.20-MariaDB) starting as process 22557 ...
2017-12-24 3:42:34 139725903902912 [Note] WSREP: Read nil XID from storage engines, skipping position init
2017-12-24 3:42:34 139725903902912 [Note] WSREP: wsrep_load(): loading provider library 'none'
2017-12-24 3:42:34 139725903902912 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2017-12-24 3:42:34 139725903902912 [Note] InnoDB: The InnoDB memory heap is disabled
2017-12-24 3:42:34 139725903902912 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-12-24 3:42:34 139725903902912 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-12-24 3:42:34 139725903902912 [Note] InnoDB: Compressed tables use zlib 1.2.7
2017-12-24 3:42:34 139725903902912 [Note] InnoDB: Using Linux native AIO
2017-12-24 3:42:34 139725903902912 [Note] InnoDB: Using SSE crc32 instructions
2017-12-24 3:42:34 139725903902912 [Note] InnoDB: Initializing buffer pool, size = 2.0G
2017-12-24 3:42:34 139725903902912 [Note] InnoDB: Completed initialization of buffer pool
2017-12-24 3:42:34 139725903902912 [Note] InnoDB: Highest supported file format is Barracuda.
2017-12-24 3:42:34 139725903902912 [Note] InnoDB: 128 rollback segment(s) are active.
2017-12-24 3:42:34 139725903902912 [Note] InnoDB: Waiting for purge to start
2017-12-24 3:42:34 139725903902912 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.34-79.1 started; log sequence number 1622818
2017-12-24 3:42:34 139722923599616 [Note] InnoDB: Dumping buffer pool(s) not yet started
OK
Creating OpenGIS required SP-s...
2017-12-24 3:42:38 139838952904896 [Note] /usr/libexec/mysqld (mysqld 10.1.20-MariaDB) starting as process 22587 ...
2017-12-24 3:42:38 139838952904896 [Note] WSREP: Read nil XID from storage engines, skipping position init
2017-12-24 3:42:38 139838952904896 [Note] WSREP: wsrep_load(): loading provider library 'none'
2017-12-24 3:42:38 139838952904896 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2017-12-24 3:42:38 139838952904896 [Note] InnoDB: The InnoDB memory heap is disabled
2017-12-24 3:42:38 139838952904896 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-12-24 3:42:38 139838952904896 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-12-24 3:42:38 139838952904896 [Note] InnoDB: Compressed tables use zlib 1.2.7
2017-12-24 3:42:38 139838952904896 [Note] InnoDB: Using Linux native AIO
2017-12-24 3:42:38 139838952904896 [Note] InnoDB: Using SSE crc32 instructions
2017-12-24 3:42:38 139838952904896 [Note] InnoDB: Initializing buffer pool, size = 2.0G
2017-12-24 3:42:39 139838952904896 [Note] InnoDB: Completed initialization of buffer pool
2017-12-24 3:42:39 139838952904896 [Note] InnoDB: Highest supported file format is Barracuda.
2017-12-24 3:42:39 139838952904896 [Note] InnoDB: 128 rollback segment(s) are active.
2017-12-24 3:42:39 139838952904896 [Note] InnoDB: Waiting for purge to start
2017-12-24 3:42:39 139838952904896 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.34-79.1 started; log sequence number 1622828
2017-12-24 3:42:39 139835976857344 [Note] InnoDB: Dumping buffer pool(s) not yet started
OK
To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system
PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
To do so, start the server, then issue the following commands:
'/usr/bin/mysqladmin' -u root password 'new-password'
'/usr/bin/mysqladmin' -u root -h 192.168.1.120 password 'new-password'
Alternatively you can run:
'/usr/bin/mysql_secure_installation'
which will also give you the option of removing the test
databases and anonymous user created by default. This is
strongly recommended for production servers.
See the MariaDB Knowledgebase at http://mariadb.com/kb or the
MySQL manual for more instructions.
You can start the MariaDB daemon with:
cd '/usr' ; /usr/bin/mysqld_safe --datadir='/app/galera'
You can test the MariaDB daemon with mysql-test-run.pl
cd '/usr/mysql-test' ; perl mysql-test-run.pl
Please report any problems at http://mariadb.org/jira
The latest information about MariaDB is available at http://mariadb.org/.
You can find additional information about the MySQL part at:
http://dev.mysql.com
Consider joining MariaDB's strong and vibrant community:
https://mariadb.org/get-involved/
|
#192.168.1.120 節點啓動MariaDB
[root@mariadb-node1 ~]# mysqld_safe --defaults-file=/etc/my.cnf.d/server.cnf --user=mysql --wsrep-new-cluster &
[1] 22614
[root@mariadb-node1 ~]# 171224 03:43:26 mysqld_safe Logging to '/app/galera/mariadb-node1.err'.
171224 03:43:26 mysqld_safe Starting mysqld daemon with databases from /app/galera
171224 03:43:26 mysqld_safe WSREP: Running position recovery with --log_error='/app/galera/wsrep_recovery.vISRp0' --pid-file='/app/galera/mariadb-node1-recover.pid'
171224 03:43:29 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
[root@mariadb-node1 ~]#
|
#檢查mysqld是否啓動
[root@mariadb-node1 ~]# netstat -lntup
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1080/sshd
tcp 0 0 0.0.0.0:4567 0.0.0.0:* LISTEN 22806/mysqld
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1179/master
tcp6 0 0 :::3306 :::* LISTEN 22806/mysqld
tcp6 0 0 :::22 :::* LISTEN 1080/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1179/master
|
七、其它兩個節點, 初始化MariaDB（備註：192.168.1.121,192.168.1.122）
mysql_install_db --defaults-file=/etc/my.cnf.d/server.cnf --user=mysql
|
八、啓動MariaDB（備註：192.168.1.121,192.168.1.122）
|
1
|
mysqld_safe --defaults-file=/etc/my.cnf.d/server.cnf --user=mysql &
|
4、驗證操做（備註：三個節點，相同操做）
一、查看集羣節點
[root@mariadb-node1 ~]# mysql -uroot -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 12
Server version: 10.1.20-MariaDB MariaDB Server
Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
+--------------------+
3 rows in set (0.00 sec)
#查看幾個集羣節點
MariaDB [(none)]> SHOW STATUS LIKE 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 3 |
+--------------------+-------+
1 row in set (0.00 sec)
#查看集羣節點
MariaDB [(none)]> show global status like 'ws%';
+------------------------------+----------------------------------------------------------+
| Variable_name | Value |
+------------------------------+----------------------------------------------------------+
| wsrep_apply_oooe | 0.000000 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 0.000000 |
| wsrep_causal_reads | 8 |
| wsrep_cert_deps_distance | 0.000000 |
| wsrep_cert_index_size | 0 |
| wsrep_cert_interval | 0.000000 |
| wsrep_cluster_conf_id | 2 |
| wsrep_cluster_size | 3 |
| wsrep_cluster_state_uuid | 8d1b5d98-e819-11e7-96c9-9a239fd041bf |
| wsrep_cluster_status | Primary |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 0.000000 |
| wsrep_connected | ON |
| wsrep_desync_count | 0 |
| wsrep_evs_delayed | |
| wsrep_evs_evict_list | |
| wsrep_evs_repl_latency | 0/0/0/0/0 |
| wsrep_evs_state | OPERATIONAL |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_flow_control_paused_ns | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_flow_control_sent | 0 |
| wsrep_gcomm_uuid | 8d1a9b25-e819-11e7-8d07-62fb3ba88975 |
| wsrep_incoming_addresses | 192.168.1.122:3306,192.168.1.121:3306,192.168.1.120:3306 |
| wsrep_last_committed | 3 |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_cached_downto | 18446744073709551615 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_commits | 0 |
| wsrep_local_index | 2 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_avg | 0.375000 |
| wsrep_local_recv_queue_max | 3 |
| wsrep_local_recv_queue_min | 0 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_avg | 0.000000 |
| wsrep_local_send_queue_max | 1 |
| wsrep_local_send_queue_min | 0 |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_local_state_uuid | 8d1b5d98-e819-11e7-96c9-9a239fd041bf |
| wsrep_protocol_version | 7 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <info@codership.com> |
| wsrep_provider_version | 3.16(r5c765eb) |
| wsrep_ready | ON |
| wsrep_received | 8 |
| wsrep_received_bytes | 861 |
| wsrep_repl_data_bytes | 0 |
| wsrep_repl_keys | 0 |
| wsrep_repl_keys_bytes | 0 |
| wsrep_repl_other_bytes | 0 |
| wsrep_replicated | 0 |
| wsrep_replicated_bytes | 0 |
| wsrep_thread_count | 9 |
+------------------------------+----------------------------------------------------------+
58 rows in set (0.00 sec)
|
能夠看到集羣正常使用。
註釋：
wsrep_cluster_status爲Primary，表示節點爲主節點，正常讀寫。
wsrep_ready爲ON，表示集羣正常運行。
wsrep_cluster_size爲3，表示集羣有三個節點。
二、建立MyISAM表測試 (備註：192.168.1.120)
[root@mariadb-node1 ~]# mysql -uroot -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 13
Server version: 10.1.20-MariaDB MariaDB Server
Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.<br><br>#建立數據庫
MariaDB [(none)]> create database crm character set=utf8;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| crm |
| information_schema |
| mysql |
| performance_schema |
+--------------------+
4 rows in set (0.00 sec)
MariaDB [(none)]> use crm
Database changed
<br>#建立MyISAM表
MariaDB [crm]> create table myisam_tbl (id int,name text) ENGINE MyISAM;
Query OK, 0 rows affected (0.00 sec)
MariaDB [crm]> insert into myisam_tbl values(1,'jojo');
Query OK, 1 row affected (0.00 sec)
MariaDB [crm]> insert into myisam_tbl values(1,'nulige');
Query OK, 1 row affected (0.00 sec)
MariaDB [crm]> show tables;
+---------------+
| Tables_in_crm |
+---------------+
| myisam_tbl |
+---------------+
|
#查看錶內容 (備註：在192.168.1.120上面操做)
MariaDB [crm]> select * from myisam_tbl;
+------+--------+
| id | name |
+------+--------+
| 1 | jojo |
| 1 | nulige |
+------+--------+
2 rows in set (0.00 sec)
|
#其它節點查看數據庫（備註：192.168.1.121,192.168.1.122）
[root@mariadb-node2 ~]# mysql -uroot -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 14
Server version: 10.1.20-MariaDB MariaDB Server
Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| crm |
| information_schema |
| mysql |
| performance_schema |
+--------------------+
4 rows in set (0.00 sec)
MariaDB [(none)]> use crm;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [crm]> show tables;
+---------------+
| Tables_in_crm |
+---------------+
| myisam_tbl |
+---------------+
1 row in set (0.00 sec)
#查看錶內容，沒有同步過來
MariaDB [crm]> select * from myisam_tbl;
Empty set (0.00 sec)
|
備註：能夠看到MyISAM存儲的表，Galera不支持同步。它僅支持XtraDB/ InnoDB存儲引擎（雖然有對MyISAM實驗支持，具體看wsrep_replicate_myisam系統變量）。
三、驗證InnoDB存儲的表
[root@mariadb-node1 ~]# mysql -uroot -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 14
Server version: 10.1.20-MariaDB MariaDB Server
Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
#建立數據庫
MariaDB [crm]> create database kuaiwei character set=utf8;
Query OK, 1 row affected (0.01 sec)
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| crm |
| information_schema |
| kuaiwei |
| mysql |
| performance_schema |
+--------------------+
5 rows in set (0.00 sec)
MariaDB [(none)]> use kuaiwei
Database changed
#建立表
MariaDB [kuaiwei]> create table innodb_tbl(id int,name text) ENGINE InnoDB;
Query OK, 0 rows affected (0.01 sec)
MariaDB [kuaiwei]> insert into innodb_tbl values(1,'jojo');
Query OK, 1 row affected (0.00 sec)
MariaDB [kuaiwei]> insert into innodb_tbl values(1,'nulige');
Query OK, 1 row affected (0.01 sec)
MariaDB [kuaiwei]> show tables;
+-------------------+
| Tables_in_kuaiwei |
+-------------------+
| innodb_tbl |
+-------------------+
1 row in set (0.00 sec)
MariaDB [kuaiwei]> select * from innodb_tbl;
+------+--------+
| id | name |
+------+--------+
| 1 | jojo |
| 1 | nulige |
+------+--------+
2 rows in set (0.01 sec)
MariaDB [kuaiwei]> exit
Bye
|
#登陸其它節點查看數據庫，表中內容（備註：192.168.1.121，192.168.1.122）
[root@mariadb-node2 ~]# mysql -uroot -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 15
Server version: 10.1.20-MariaDB MariaDB Server
Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| crm |
| information_schema |
| kuaiwei |
| mysql |
| performance_schema |
+--------------------+
5 rows in set (0.00 sec)
MariaDB [(none)]> use kuaiwei;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [kuaiwei]> show tables;
+-------------------+
| Tables_in_kuaiwei |
+-------------------+
| innodb_tbl |
+-------------------+
1 row in set (0.00 sec)
MariaDB [kuaiwei]> select * from innodb_tbl;
+------+--------+
| id | name |
+------+--------+
| 1 | jojo |
| 1 | nulige |
+------+--------+
2 rows in set (0.00 sec)
MariaDB [kuaiwei]> exit
Bye
|
四、模擬故障：
停掉192.168.1.120 服務器上面的mariadb
[root@mariadb-node1 ~]# mysqladmin -uroot -p "shutdown"
Enter password: #輸入數據庫密碼
#檢查數據庫是否啓動
[root@mariadb-node1 ~]# netstat -lntup
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1080/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1179/master
tcp6 0 0 :::22 :::* LISTEN 1080/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1179/master
|
而後在其餘節點（192.168.1.121,192.168.1.122）上面執行：
[root@mariadb-node2 ~]# mysql -uroot -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 16
Server version: 10.1.20-MariaDB MariaDB Server
Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show global status like 'wsrep%';
+------------------------------+---------------------------------------+
| Variable_name | Value |
+------------------------------+---------------------------------------+
| wsrep_apply_oooe | 0.000000 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 1.000000 |
| wsrep_causal_reads | 24 |
| wsrep_cert_deps_distance | 1.166667 |
| wsrep_cert_index_size | 6 |
| wsrep_cert_interval | 0.000000 |
| wsrep_cluster_conf_id | 3 |
| wsrep_cluster_size | 2 |
| wsrep_cluster_state_uuid | 8d1b5d98-e819-11e7-96c9-9a239fd041bf |
| wsrep_cluster_status | Primary |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 1.000000 |
| wsrep_connected | ON |
| wsrep_desync_count | 0 |
| wsrep_evs_delayed | |
| wsrep_evs_evict_list | |
| wsrep_evs_repl_latency | 0/0/0/0/0 |
| wsrep_evs_state | OPERATIONAL |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_flow_control_paused_ns | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_flow_control_sent | 0 |
| wsrep_gcomm_uuid | 71aa0335-e81b-11e7-a8bc-dbee4f046e45 |
| wsrep_incoming_addresses | 192.168.1.122:3306,192.168.1.121:3306 |
| wsrep_last_committed | 9 |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_cached_downto | 4 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_commits | 0 |
| wsrep_local_index | 1 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_avg | 0.000000 |
| wsrep_local_recv_queue_max | 1 |
| wsrep_local_recv_queue_min | 0 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_avg | 0.000000 |
| wsrep_local_send_queue_max | 1 |
| wsrep_local_send_queue_min | 0 |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_local_state_uuid | 8d1b5d98-e819-11e7-96c9-9a239fd041bf |
| wsrep_protocol_version | 7 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <info@codership.com> |
| wsrep_provider_version | 3.16(r5c765eb) |
| wsrep_ready | ON |
| wsrep_received | 10 |
| wsrep_received_bytes | 2894 |
| wsrep_repl_data_bytes | 0 |
| wsrep_repl_keys | 0 |
| wsrep_repl_keys_bytes | 0 |
| wsrep_repl_other_bytes | 0 |
| wsrep_replicated | 0 |
| wsrep_replicated_bytes | 0 |
| wsrep_thread_count | 9 |
+------------------------------+---------------------------------------+
58 rows in set (0.00 sec)
MariaDB [(none)]> exit
Bye
|
此時集羣爲自動將192.168.1.120故障節點剔除掉，而且正常提供服務。
最後咱們恢復失敗的節點（192.168.1.120）：
[root@mariadb-node1 ~]# mysqld_safe --defaults-file=/etc/my.cnf.d/server.cnf --user=mysql &
|
再查看集羣環境：
[root@mariadb-node2 ~]# mysql -uroot -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 17
Server version: 10.1.20-MariaDB MariaDB Server
Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
#又恢復成了三個節點
MariaDB [(none)]> SHOW STATUS LIKE 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 3 |
+--------------------+-------+
1 row in set (0.00 sec)
MariaDB [(none)]> exit
Bye
|
五、模擬腦裂後的處理
下面模擬在網絡抖動發生丟包的狀況下，兩個節點失聯致使腦裂。首先，在192.168.1.121和192.168.1.122兩個節點上分別執行：
iptables -A INPUT -p tcp --sport 4567 -j DROP
iptables -A INPUT -p tcp --dport 4567 -j DROP
以上命令用來禁止wsrep全同步複製4567端口通訊。
而後咱們在192.168.1.120節點查看：
MariaDB [(none)]> show global statuslike 'ws%';
能夠看到下面的幾個值：
wsrep_cluster_size 1
wsrep_cluster_status non-Primary
wsrep_ready OFF
MariaDB [(none)]> use test_db;
ERROR 1047 (08S01): WSREP has not yetprepared node for application use
MariaDB [(none)]> select@@wsrep_node_name;
ERROR 1205 (HY000): Lock wait timeoutexceeded; try restarting transaction
|
如今已經出現腦裂的狀況，而且集羣沒法執行任何命令。
爲了解決這個問題，能夠執行：
set global wsrep_provider_options="pc.bootstrap=true";
經過這個命令來強制恢復出現腦裂的節點。
下面咱們來驗證一下：
MariaDB [(none)]> select @@wsrep_node_name;
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
MariaDB [(none)]> set global wsrep_provider_options="pc.bootstrap=true";
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> select @@wsrep_node_name;
+-------------------+
| @@wsrep_node_name |
+-------------------+
| mariadb-a03 |
+-------------------+
1 row in set (0.27 sec)
MariaDB [(none)]> use test_db;
Reading table information for completion oft able and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [test_db]> show tables;
+-------------------+
| Tables_in_test_db |
+-------------------+
| innodb_tbl |
| myisam_tbl |
+-------------------+
|
最後咱們將節點192.168.1.121和192.168.1.122恢復一下，只要清理一下iptables表便可(由於個人是測試環境，生產環境須要刪除上面的規則便可)：
iptables –F
各個節點驗證一下：
192.168.1.120：
MariaDB [test_db]> SHOW STATUS LIKE 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 3 |
+--------------------+-------+
1 row in set (0.00 sec)
192.168.1.121：
MariaDB [(none)]> select @@wsrep_node_name;
+-------------------+
| @@wsrep_node_name |
+-------------------+
| mariadb-node2 |
+-------------------+
|
六、避免髒讀
Galera Cluster不是真正意義上的全同步複製，存在延遲。咱們能夠在一個節點上面執行FLUSH TABLES WITH READ LOCK;全局讀鎖。
而後在其餘節點執行寫操做，觀察延遲狀況。
好比咱們在192.168.1.122節點執行全局讀鎖設置：
[root@mariadb-node3 ~]# mysql -uroot -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 13
Server version: 10.1.20-MariaDB MariaDB Server
Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| crm |
| information_schema |
| kuaiwei |
| mysql |
| performance_schema |
+--------------------+
5 rows in set (0.00 sec)
MariaDB [(none)]> use kuaiwei;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
#執行全局讀鎖設置
MariaDB [kuaiwei]> flush tables with read lock;
Query OK, 0 rows affected (0.00 sec)
MariaDB [kuaiwei]> select * from innodb_tbl;
+------+--------+
| id | name |
+------+--------+
| 1 | jojo |
| 1 | nulige |
+------+--------+
2 rows in set (0.00 sec)
|
而後在192.168.1.120節點插入操做
[root@mariadb-node1 ~]# mysql -uroot -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 11
Server version: 10.1.20-MariaDB MariaDB Server
Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| crm |
| information_schema |
| kuaiwei |
| mysql |
| performance_schema |
+--------------------+
5 rows in set (0.00 sec)
MariaDB [(none)]> use kuaiwei;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [kuaiwei]> select @@wsrep_node_name;
+-------------------+
| @@wsrep_node_name |
+-------------------+
| mariadb-a04 |
+-------------------+
1 row in set (0.00 sec)
MariaDB [kuaiwei]> insert into innodb_tbl values(2,'li men');
Query OK, 1 row affected (0.00 sec)
MariaDB [kuaiwei]> select * from innodb_tb1;
ERROR 1146 (42S02): Table 'kuaiwei.innodb_tb1' doesn't exist
MariaDB [kuaiwei]> insert into innodb_tbl values(2,'hbase');
Query OK, 1 row affected (0.00 sec)
MariaDB [kuaiwei]> select * from innodb_tbl;
+------+--------+
| id | name |
+------+--------+
| 1 | jojo |
| 1 | nulige |
| 2 | li men |
| 2 | hbase |
+------+--------+
4 rows in set (0.00 sec)
|
在節點192.168.1.122上測試查詢操做：
MariaDB [kuaiwei]> select * from innodb_tbl;
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
|
這裏之因此沒有讀取到髒數據，是由於我在MariaDB配置文件中設置了wsrep_causal_reads=ON;
咱們將wsrep_causal_reads修改成0或OFF來看一下效果：
MariaDB [kuaiwei]> set wsrep_causal_reads=0;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id: 16
Current database: kuaiwei
Query OK, 0 rows affected, 1 warning (13.43 sec)
MariaDB [kuaiwei]> select * from innodb_tbl;
+------+--------+
| id | name |
+------+--------+
| 1 | jojo |
| 1 | nulige |
| 2 | li men |
| 2 | hbase |
+------+--------+
4 rows in set (0.00 sec)
MariaDB [kuaiwei]> exit
Bye
|
經過上面的一系列測試，最後總結一下：
一、在生產環境下應該避免使用大事務，不建議在高併發寫入場景下使用Galera Cluster架構，會致使集羣限流，從而引發整個集羣hang住，出現生產故障。針對這種狀況能夠考慮主從，實現讀寫分離等手段。
二、對數據一致性要求較高，而且數據寫入不頻繁，數據庫容量也不大（50GB左右），網絡情況良好的狀況下，能夠考慮使用Galera方案。