MariaDB Galera Cluster環境搭建及高可用測試

1、服務器概況
Galera Cluster須要至少三個節點，在這次實驗過程當中，三個節點IP地址：
192.168.56.101
192.168.56.102
192.168.56.103
OS爲centos 7.2
服務器配置：
4G 內存，2核CPU，20G數據磁盤空間。

關閉防火牆:

service iptables stop
systemctl stop firewalld
chkconfig iptables off

關閉SELinux:

/etc/selinux/config中的SELINUX=disabled

修改主機名

vi /etc/hostname MariaDB01 MariaDB02 MariaDB03 systemctl restart systemd-hostnamed

三臺主機上加入/etc/hosts:

192.168.56.101 MariaDB01 192.168.56.102 MariaDB02 192.168.56.103 MariaDB03

 

2、MariaDB Galera Cluster軟件二進制安裝

  官網下載：https://downloads.mariadb.org/   10.0.29

  初始化節點1

su - mysql cd /app/mysql/ vi my.cnf

--my.cnf
[client]
port = 3306
socket = /tmp/mysql.sock
[mysqld]
port = 3306
socket = /tmp/mysql.sock
character-set-server=utf8
skip-name-resolve
basedir=/app/mysql
datadir=/data/mysqldata
back_log = 50
binlog_cache_size = 1M
max_heap_table_size = 64M
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
thread_cache_size = 8
thread_concurrency = 8
ft_min_word_len = 4
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
thread_stack = 240K
transaction_isolation = REPEATABLE-READ
tmp_table_size = 64M
log-bin=/data/binlog/mysql-bin
slow_query_log
long_query_time = 2
log_error=/data/logs/error.log
slow_query_log_file=/data/logs/slow.log
server-id = 2
key_buffer_size = 32M
bulk_insert_buffer_size = 64M
myisam_sort_buffer_size = 128M
myisam_max_sort_file_size = 10G
myisam_repair_threads = 1
myisam_recover
innodb_additional_mem_pool_size = 16M
innodb_buffer_pool_size = 1G
innodb_data_file_path = ibdata1:500M;ibdata2:16M:autoextend
innodb_file_per_table = 1
innodb_support_xa = on
sync_binlog = 0
table_definition_cache=65535
table_open_cache=4000
max_connections=3100
max_user_connections=3000
max_connect_errors=65536
max_allowed_packet=16M
innodb_strict_mode = 1
innodb_open_files = 4000
innodb_file_format = Barracuda
innodb_file_format_max = Barracuda
innodb_flush_method = O_DIRECT
expire_logs_days=3
innodb_data_home_dir = /data/mysqldata
innodb_write_io_threads = 8
innodb_read_io_threads = 8
innodb_thread_concurrency = 16
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 128M
innodb_log_file_size = 256M
innodb_log_files_in_group = 3
innodb_log_group_home_dir = /data/redolog
innodb_max_dirty_pages_pct = 90
innodb_flush_method=O_DIRECT
innodb_lock_wait_timeout = 120

[galera]
bind-address=0.0.0.0
wsrep_provider=none
wsrep_node_address='192.168.56.101'
wsrep_provider=/app/mariadb-galera-10.0.29-linux/lib/galera/libgalera_smm.so
wsrep_provider_options='gcache.size=1999M'
wsrep_cluster_name='galera_cluster'
wsrep_cluster_address ='gcomm://192.168.56.101,192.168.56.102,192.168.56.103'
wsrep_node_name='MariaDB01'
wsrep_slave_threads=4
wsrep_certify_nonPK=1
wsrep_max_ws_rows=131072
wsrep_max_ws_size=1073741824
wsrep_debug=0
wsrep_convert_LOCK_to_trx=0
wsrep_retry_autocommit=1
wsrep_auto_increment_control=1
wsrep_replicate_myisam=1
wsrep_drupal_282555_workaround=0
wsrep_causal_reads=0
wsrep_sst_method=rsync
wsrep_sst_auth= sst_user:dbpass

[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
[myisamchk]
key_buffer_size = 512M
sort_buffer_size = 512M
read_buffer = 8M
write_buffer = 8M
[mysqlhotcopy]
interactive-timeout
[mysqld_safe]
open-files-limit = 8192

　　

./scripts/mysql_install_db --defaults-file=my.cnf ./bin/mysqld_safe --defaults-file=my.cnf --wsrep-new-cluster &

      初始化節點二、3

./scripts/mysql_install_db --defaults-file=my.cnf ./bin/mysqld_safe --defaults-file=my.cnf &

     爲集羣建立賬號

mysql> DELETE FROM mysql.user WHERE user=''; mysql> GRANT ALL ON *.* TO 'root'@'%' IDENTIFIED BY 'dbpass'; mysql> GRANT USAGE ON *.* to sst_user@'%' IDENTIFIED BY 'dbpass'; mysql> GRANT ALL PRIVILEGES on *.* to sst_user@'%'; mysql> FLUSH PRIVILEGES;

 

 三、平常維護

       1. 集羣啓動
         啓動第一個節點：service mysql start --wsrep-new-cluster
         啓動其它節點：server mysql start
         關閉節點：server mysql stop
       2. 經過查看4567端口確認集羣是否啓動
         netstat -plantu | grep mysqld
       3. 查詢 galera插件是否已啓用

SHOW status LIKE 'wsrep_ready'; SHOW VARIABLES LIKE 'wsrep_cluster_address';

       4. 檢查集羣是否正常

          查看Galera集羣狀態: mysql> show status like 'wsrep%';

(1)SHOW GLOBAL STATUS LIKE 'wsrep_cluster_state_uuid'; 集羣中每一個節點的值必須一致。 (2)SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'; 顯示集羣的規模，此例應該顯示」3」。 (3)SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status'; 顯示」Primary」爲正常。 (4)SHOW GLOBAL STATUS LIKE 'wsrep_ready'; 顯示」ON」爲正常。 (5) SHOW GLOBAL STATUS LIKE 'wsrep_connected'; 顯示」ON」爲正常。 (6) SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment'; 顯示」 Joined」爲正常。

        cluster完整性檢測

        status 變量:
        wsrep_cluster_state_uuid: cluster中的全部node的該變量的值必須同樣, 不同表示node沒有鏈接到cluster。
        wsrep_cluster_conf_id: 次變量用來表示node是否在它相應的cluster中。cluster中全部node的該變量值應該同樣， 不同則表示nodes被分隔開了, node恢復的時候該變量也會恢復。
        wsrep_cluster_size: 表示cluster中有多少node節點， 等於預期的數量則表示全部node鏈接到了cluster。
        wsrep_cluster_status: 正常狀況下值爲Primary, 若是不爲Primay,則該node當前不能操做(歸咎於多成員關係的改變和quorum的缺失), 同時也可能知足split-brain的條件。
        若是cluster中沒有node 鏈接上(connected) PC(就是全部node屬於同一部分,可是node都是non-primary狀態),

 

        wsrep_sst_method的參數說明：

Method      Speed    Blocks Donor?    Available on Live Node    Type       Requires root access to database server? mysqldump slow yes yes logical  donor and joiner rsync fastest yes no physical none xtrabackup fast briefly no physical donor only

      rsync在數據同步（SST和IST)的時候，速度最快，可是會鎖住提供數據的節點，xtrabackup只會短暫的鎖住節點。

 

 4、高可用測試

     一、節點數據同步寫入

     二、故障節點修復後數據自行修復

附錄

MariaDB Galera Cluste安裝報錯處理

170525 23:19:48 [ERROR] WSREP: wsrep_load(): dlopen(): libssl.so.6: cannot open shared object file: No such file or directory
170525 23:19:48 [ERROR] WSREP: wsrep_load(/app/mariadb-galera-10.0.29-linux/lib/galera/libgalera_smm.so) failed: Invalid argument (22). Reverting to no provider.
cd /app/mysql/lib/galera/
ldd ./libgalera_smm.so
cd /usr/lib64
ln -s libcrypto.so.1.0.1e libcrypto.so.6
ln -s libssl.so.1.0.1e libssl.so.6

 

170526 0:04:07 [Warning] WSREP: Gap in state sequence. Need state transfer.
170526 0:04:07 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '192.168.56.103' --datadir '/data/mysqldata/' --defaults-file '/app/mariadb-galera-10.0.29-linux/my.cnf' --defaults-group-suffix '' --parent '2832' --binlog '/data/binlog/mysql-bin' '
which: no rsync in (/usr/sbin:/sbin:/app/mariadb-galera-10.0.29-linux//bin:/app/mysql/bin:/bin:/usr/bin:/usr/local/bin:/usr/sbin)
170526 0:04:07 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '192.168.56.103' --datadir '/data/mysqldata/' --defaults-file '/app/mariadb-galera-10.0.29-linux/my.cnf' --defaults-group-suffix '' --parent '2832' --binlog '/data/binlog/mysql-bin'
Read: ''rsync' not found in PATH'
170526 0:04:07 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '192.168.56.103' --datadir '/data/mysqldata/' --defaults-file '/app/mariadb-galera-10.0.29-linux/my.cnf' --defaults-group-suffix '' --parent '2832' --binlog '/data/binlog/mysql-bin' : 2 (No such file or directory)
170526 0:04:07 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
170526 0:04:07 [ERROR] Aborting
yum install rsync.x86_64

 

which: no lsof in (/usr/sbin:/sbin:/app/mariadb-galera-10.0.29-linux//bin:/app/mysql/bin:/bin:/usr/bin:/usr/local/bin:/usr/sbin:/app/xtrabackup/percona-xtrabackup/bin)
170526 10:13:28 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '192.168.56.103' --datadir '/data/mysqldata/' --defaults-file '/app/mariadb-galera-10.0.29-linux/my.cnf' --defaults-group-suffix '' --parent '2821' --binlog '/data/binlog/mysql-bin'
Read: ''lsof' not found in PATH'
170526 10:13:28 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '192.168.56.103' --datadir '/data/mysqldata/' --defaults-file '/app/mariadb-galera-10.0.29-linux/my.cnf' --defaults-group-suffix '' --parent '2821' --binlog '/data/binlog/mysql-bin' : 2 (No such file or directory)
170526 10:13:28 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
170526 10:13:28 [ERROR] Aborting
yum install lsof