MySQL PXC構建一個新節點只需IST傳輸的方法

時間 2019-11-06

標籤 mysql pxc 構建一個節點只需 ist 傳輸方法欄目 MySQL 简体版

原文原文鏈接

需求場景：原有的pxc環境數據量已經比較大，新買的服務器要加入此集羣中，如何讓其用IST的方式傳輸，而不是SST.html

PXC傳輸數據有兩種方式：node

IST: Incremental State Transfer 增量同步
SST：State Snapshot Transfer 全量同步
mysql

IST 發生的條件：This is done using caching mechanism on nodes （即只發生在以前集羣裏的節點數據還有在緩存中）sql

[root@zejin241 I6000]# cat grastate.dat
# GALERA saved state
version: 2.1
uuid: 6c86dc17-246f-11e6-9955-ae4d7e89eed2
seqno: 4
cert_index:

緩存

因此咱們的思路即經過備份一個集羣中節點的數據，恢復到新節點上，並僞造一個grastate.dat，來記錄備份數據的seqno,這樣當新節點啓動時就會自動用IST的方法來進行，而不是SST（注意gcache.size應該有足夠的容量來保證在備份及恢復這段時間內新產生的數據都被緩存到）服務器

galera.cache: This file is used as a main writeset store. It’s implemented as a permanent ring-buffer file
that is preallocated on disk when the node is initialized. File size can be controlled with the variable
gcache.size. If this value is bigger, more writesets are cached and chances are better that the rejoining node will get IST instead of SST. Filename can be changed with the gcache.name variable.session

關於如何搭建一個pxc環境請看：http://www.cnblogs.com/zejin2008/p/5475285.htmlapp

以下爲此次的操做步驟：socket

node240：192.168.1.240 --已經存在的集羣節點

node241：192.168.1.241 --將要加入的集羣節點

step1:啓動新的集羣中的第一個節點

/usr/local/pxc_56/bin/mysqld_safe --defaults-file=/home/mysql/pxc6000.cnf --ledir=/usr/local/pxc_56/bin/ --wsrep-new-cluster  &

配置文件以下：

[root@zejin240 I6000]# cat /home/mysql/pxc6000.cnf
[client]
port            = 6000
socket          = /home/mysql/I6000/mysql.sock
default-character-set=utf8

[mysqld]
basedir = /usr/local/pxc_56
datadir = /home/mysql/I6000

pid-file        = /home/mysql/I6000/mysql.pid
character-set-server=utf8
init_connect = 'SET NAMES utf8'
log-bin=/home/mysql/I6000/log_bin
server-id = 9816000

innodb_buffer_pool_size = 100M
innodb_data_file_path = ibdata1:10M:autoextend
innodb_data_home_dir = /home/mysql/I6000
innodb_file_per_table=1


skip-name-resolve


port            = 6000
socket          = /home/mysql/I6000/mysql.sock
user=mysql
log_error=/home/mysql/I6000/mysql_error.log

#pxc
user=mysql
log_error=error.log
binlog_format=ROW

#6000 for mysqld port, 6020 for sst port, 6031 for ist port, 6030 for cluster communication port
wsrep_cluster_address='gcomm://192.168.1.240:6030,192.168.1.241:6030,192.168.1.242:6030'
wsrep_provider=/usr/local/pxc_56/lib/libgalera_smm.so
wsrep_sst_receive_address=192.168.1.240:6020
wsrep_node_incoming_address=192.168.1.240:6000
wsrep_node_address=192.168.1.240:6000
wsrep_provider_options = "gmcast.listen_addr=tcp://192.168.1.240:6030;ist.recv_addr=192.168.1.240:6031;"

wsrep_slave_threads=2
wsrep_cluster_name=pxc_zejin
wsrep_sst_method=xtrabackup-v2
wsrep_node_name=node6000_240
innodb_autoinc_lock_mode=2
wsrep_sst_auth="sstuser:123"
log-slave-updates


[mysql]
no-auto-rehash
default-character-set=utf8

鏈接進去並創造更改一些數據：

mysql> use zejin;
mysql> select * from t1;
+----+-------+
| id | name  |
+----+-------+
|  1 | chen  |
|  2 | li    |
|  3 | zhang |
+----+-------+
3 rows in set (0.01 sec)

mysql> update mysql.user set password=password('123');
mysql> flush privileges;

mysql> show master status; +----------------+----------+--------------+------------------+-------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | +----------------+----------+--------------+------------------+-------------------+ | log_bin.000003 | 2772 | | | | +----------------+----------+--------------+------------------+-------------------+ 1 row in set (0.00 sec)

step2：用innobackupex備份node240數據：

[root@zejin240 mysql]# innobackupex --user=root --password=123 --port=6000 --host=127.0.0.1 --defaults-file=/home/mysql/pxc6000.cnf /home/mysql/

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013.  All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

Get the latest version of Percona XtraBackup, documentation, and help resources:
http://www.percona.com/xb/p

160528 09:39:07  innobackupex: Executing a version check against the server...
160528 09:39:07  innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_file=/home/mysql/pxc6000.cnf;mysql_read_default_group=xtrabackup;host=127.0.0.1;port=6000' as 'root'  (using password: YES).
160528 09:39:07  innobackupex: Connected to MySQL server
160528 09:39:07  innobackupex: Done.
160528 09:39:07  innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_file=/home/mysql/pxc6000.cnf;mysql_read_default_group=xtrabackup;host=127.0.0.1;port=6000' as 'root'  (using password: YES).
160528 09:39:07  innobackupex: Connected to MySQL server
160528 09:39:07  innobackupex: Starting the backup operation

IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".

innobackupex:  Using server version 5.6.26-74.0-25.12-log

innobackupex: Created backup directory /home/mysql/2016-05-28_09-39-07

160528 09:39:07  innobackupex: Starting ibbackup with command: xtrabackup  --defaults-file="/home/mysql/pxc6000.cnf"  --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/home/mysql/2016-05-28_09-39-07 --innodb_log_file_size="50331648" --tmpdir=/tmp --extra-lsndir='/tmp'
innobackupex: Waiting for ibbackup (pid=30280) to suspend
innobackupex: Suspend file '/home/mysql/2016-05-28_09-39-07/xtrabackup_suspended_2'

xtrabackup version 2.2.11 based on MySQL server 5.6.24 Linux (x86_64) (revision id: )
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /home/mysql/I6000
xtrabackup: open files limit requested 0, set to 1024
xtrabackup: using the following InnoDB configuration:
xtrabackup:   innodb_data_home_dir = /home/mysql/I6000
xtrabackup:   innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup:   innodb_log_group_home_dir = ./
xtrabackup:   innodb_log_files_in_group = 2
xtrabackup:   innodb_log_file_size = 50331648
>> log scanned up to (1637591)
xtrabackup: Generating a list of tablespaces
[01] Copying /home/mysql/I6000/ibdata1 to /home/mysql/2016-05-28_09-39-07/ibdata1
>> log scanned up to (1637591)
[01]        ...done
[01] Copying ./zejin/t1.ibd to /home/mysql/2016-05-28_09-39-07/zejin/t1.ibd
[01]        ...done
[01] Copying ./mysql/innodb_table_stats.ibd to /home/mysql/2016-05-28_09-39-07/mysql/innodb_table_stats.ibd
[01]        ...done
[01] Copying ./mysql/slave_master_info.ibd to /home/mysql/2016-05-28_09-39-07/mysql/slave_master_info.ibd
[01]        ...done
[01] Copying ./mysql/slave_relay_log_info.ibd to /home/mysql/2016-05-28_09-39-07/mysql/slave_relay_log_info.ibd
[01]        ...done
[01] Copying ./mysql/slave_worker_info.ibd to /home/mysql/2016-05-28_09-39-07/mysql/slave_worker_info.ibd
[01]        ...done
[01] Copying ./mysql/innodb_index_stats.ibd to /home/mysql/2016-05-28_09-39-07/mysql/innodb_index_stats.ibd
[01]        ...done
>> log scanned up to (1637591)
xtrabackup: Creating suspend file '/home/mysql/2016-05-28_09-39-07/xtrabackup_suspended_2' with pid '30280'

160528 09:39:10  innobackupex: Continuing after ibbackup has suspended
160528 09:39:10  innobackupex: Executing LOCK TABLES FOR BACKUP...
160528 09:39:10  innobackupex: Backup tables lock acquired

160528 09:39:10  innobackupex: Starting to backup non-InnoDB tables and files
innobackupex: in subdirectories of '/home/mysql/I6000/'
innobackupex: Backing up files '/home/mysql/I6000//performance_schema/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (53 files)
>> log scanned up to (1637598)
innobackupex: Backing up file '/home/mysql/I6000//zejin/db.opt'
innobackupex: Backing up file '/home/mysql/I6000//zejin/t1.frm'
innobackupex: Backing up files '/home/mysql/I6000//mysql/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (74 files)
160528 09:39:11  innobackupex: Finished backing up non-InnoDB tables and files

160528 09:39:11  innobackupex: Executing LOCK BINLOG FOR BACKUP...
160528 09:39:11  innobackupex: Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS...
160528 09:39:11  innobackupex: Waiting for log copying to finish

xtrabackup: The latest check point (for incremental): '1637598'
xtrabackup: Stopping log copying thread.
.>> log scanned up to (1637598)

xtrabackup: Creating suspend file '/home/mysql/2016-05-28_09-39-07/xtrabackup_log_copied' with pid '30280'
xtrabackup: Transaction log of lsn (1637591) to (1637598) was copied.
160528 09:39:12  innobackupex: Executing UNLOCK BINLOG
160528 09:39:12  innobackupex: Executing UNLOCK TABLES
160528 09:39:12  innobackupex: All tables unlocked

innobackupex: Backup created in directory '/home/mysql/2016-05-28_09-39-07'
innobackupex: MySQL binlog position: filename 'log_bin.000003', position 2772
160528 09:39:12  innobackupex: Connection to database server closed
160528 09:39:12  innobackupex: completed OK!

[root@zejin240 2016-05-28_09-39-07]# ll
total 75812
-rw-r--r--. 1 root root      358 May 28 09:39 backup-my.cnf
-rw-r-----. 1 root root 77594624 May 28 09:39 ibdata1
drwx------. 2 root root     4096 May 28 09:39 mysql
drwxr-xr-x. 2 root root     4096 May 28 09:39 performance_schema
-rw-r--r--. 1 root root       20 May 28 09:39 xtrabackup_binlog_info
-rw-r-----. 1 root root       89 May 28 09:39 xtrabackup_checkpoints
-rw-r--r--. 1 root root      635 May 28 09:39 xtrabackup_info
-rw-r-----. 1 root root     2560 May 28 09:39 xtrabackup_logfile
drwx------. 2 root root     4096 May 28 09:39 zejin

step3:此時咱們在node240上再加入一些數據：

此時咱們再在node240上插入一些數據：
mysql> use zejin;
mysql> insert into t1(name) values('lin');
Query OK, 1 row affected (0.06 sec)

mysql> insert into t1(name) values('liang');
Query OK, 1 row affected (0.00 sec)

mysql> select * from t1;
+----+-------+
| id | name  |
+----+-------+
|  1 | chen  |
|  2 | li    |
|  3 | zhang |
|  4 | lin   |
|  5 | liang |
+----+-------+
5 rows in set (0.00 sec)

step4：將備份的數據傳輸到node241上並恢復：

[root@zejin241 mysql]# innobackupex --apply-log 2016-05-28_09-39-07/ InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved. This software is published under the GNU GENERAL PUBLIC LICENSE Version 2, June 1991. Get the latest version of Percona XtraBackup, documentation, and help resources: http://www.percona.com/xb/p 160528 09:54:01 innobackupex: Starting the apply-log operation IMPORTANT: Please check that the apply-log run completes successfully. At the end of a successful apply-log run innobackupex prints "completed OK!". 160528 09:54:01 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/home/mysql/2016-05-28_09-39-07/backup-my.cnf" --defaults-group="mysqld" --prepare --target-dir=/home/mysql/2016-05-28_09-39-07 xtrabackup version 2.2.11 based on MySQL server 5.6.24 Linux (x86_64) (revision id: ) xtrabackup: cd to /home/mysql/2016-05-28_09-39-07 xtrabackup: This target seems to be not prepared yet. xtrabackup: xtrabackup_logfile detected: size=2097152, start_lsn=(1637591) xtrabackup: using the following InnoDB configuration for recovery: xtrabackup: innodb_data_home_dir = ./ xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend xtrabackup: innodb_log_group_home_dir = ./ xtrabackup: innodb_log_files_in_group = 1 xtrabackup: innodb_log_file_size = 2097152 xtrabackup: using the following InnoDB configuration for recovery: xtrabackup: innodb_data_home_dir = ./ xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend xtrabackup: innodb_log_group_home_dir = ./ xtrabackup: innodb_log_files_in_group = 1 xtrabackup: innodb_log_file_size = 2097152 xtrabackup: Starting InnoDB instance for recovery. xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter) InnoDB: Using atomics to ref count buffer pool pages InnoDB: The InnoDB memory heap is disabled InnoDB: Mutexes and rw_locks use GCC atomic builtins InnoDB: Memory barrier is not used InnoDB: Compressed tables use zlib 1.2.3 InnoDB: Using CPU crc32 instructions InnoDB: Initializing buffer pool, size = 100.0M InnoDB: Completed initialization of buffer pool InnoDB: Highest supported file format is Barracuda. InnoDB: Log scan progressed past the checkpoint lsn 1637591 InnoDB: Database was not shutdown normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 1637598 (0%) InnoDB: Last MySQL binlog file position 0 2772, file name log_bin.000003 InnoDB: 128 rollback segment(s) are active. InnoDB: Waiting for purge to start InnoDB: 5.6.24 started; log sequence number 1637598 [notice (again)] If you use binary log and don't use any hack of group commit,  the binary log position seems to be: InnoDB: Last MySQL binlog file position 0 2772, file name log_bin.000003 xtrabackup: Recovered WSREP position: 6c86dc17-246f-11e6-9955-ae4d7e89eed2:8 xtrabackup: starting shutdown with innodb_fast_shutdown = 1 InnoDB: FTS optimize thread exiting. InnoDB: Starting shutdown... InnoDB: Shutdown completed; log sequence number 1637608 160528 09:54:03 innobackupex: Restarting xtrabackup with command: xtrabackup --defaults-file="/home/mysql/2016-05-28_09-39-07/backup-my.cnf" --defaults-group="mysqld" --prepare --target-dir=/home/mysql/2016-05-28_09-39-07 for creating ib_logfile* xtrabackup version 2.2.11 based on MySQL server 5.6.24 Linux (x86_64) (revision id: ) xtrabackup: cd to /home/mysql/2016-05-28_09-39-07 xtrabackup: This target seems to be already prepared. xtrabackup: notice: xtrabackup_logfile was already used to '--prepare'. xtrabackup: using the following InnoDB configuration for recovery: xtrabackup: innodb_data_home_dir = ./ xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend xtrabackup: innodb_log_group_home_dir = ./ xtrabackup: innodb_log_files_in_group = 2 xtrabackup: innodb_log_file_size = 50331648 xtrabackup: using the following InnoDB configuration for recovery: xtrabackup: innodb_data_home_dir = ./ xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend xtrabackup: innodb_log_group_home_dir = ./ xtrabackup: innodb_log_files_in_group = 2 xtrabackup: innodb_log_file_size = 50331648 xtrabackup: Starting InnoDB instance for recovery. xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter) InnoDB: Using atomics to ref count buffer pool pages InnoDB: The InnoDB memory heap is disabled InnoDB: Mutexes and rw_locks use GCC atomic builtins InnoDB: Memory barrier is not used InnoDB: Compressed tables use zlib 1.2.3 InnoDB: Using CPU crc32 instructions InnoDB: Initializing buffer pool, size = 100.0M InnoDB: Completed initialization of buffer pool InnoDB: Setting log file ./ib_logfile101 size to 48 MB InnoDB: Setting log file ./ib_logfile1 size to 48 MB InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 InnoDB: New log files created, LSN=1637608 InnoDB: Highest supported file format is Barracuda. InnoDB: 128 rollback segment(s) are active. InnoDB: Waiting for purge to start InnoDB: 5.6.24 started; log sequence number 1637900 [notice (again)] If you use binary log and don't use any hack of group commit, the binary log position seems to be: InnoDB: Last MySQL binlog file position 0 2772, file name log_bin.000003 xtrabackup: starting shutdown with innodb_fast_shutdown = 1 InnoDB: FTS optimize thread exiting. InnoDB: Starting shutdown... InnoDB: Shutdown completed; log sequence number 1637910 160528 09:54:06 innobackupex: completed OK!

[root@zejin240 2016-05-28_09-39-07]# ll total 176168 -rw-r--r--. 1 root root 358 May 28 09:39 backup-my.cnf -rw-r-----. 1 root root 77594624 May 28 09:54 ibdata1 -rw-r--r--. 1 root root 50331648 May 28 09:54 ib_logfile0 -rw-r--r--. 1 root root 50331648 May 28 09:54 ib_logfile1 drwx------. 2 root root 4096 May 28 09:39 mysql drwxr-xr-x. 2 root root 4096 May 28 09:39 performance_schema -rw-r--r--. 1 root root 20 May 28 09:39 xtrabackup_binlog_info -rw-r--r--. 1 root root 20 May 28 09:54 xtrabackup_binlog_pos_innodb -rw-r-----. 1 root root 89 May 28 09:54 xtrabackup_checkpoints -rw-r--r--. 1 root root 38 May 28 09:54 xtrabackup_galera_info -rw-r--r--. 1 root root 635 May 28 09:39 xtrabackup_info -rw-r-----. 1 root root 2097152 May 28 09:54 xtrabackup_logfile drwx------. 2 root root 4096 May 28 09:39 zejin

step5:找出Xid的位置

有兩種方法

方法一：查看文件xtrabackup_galera_info

[root@zejin241 I6000]# cat xtrabackup_galera_info
6c86dc17-246f-11e6-9955-ae4d7e89eed2:8

方法二：經過binlog位置來查找：

[root@zejin241 I6000]# cat xtrabackup_binlog_info
log_bin.000003    2772

即備份時的備份點位置爲log_bin.000003 2772

咱們再去主庫查找此binlog對應的xid

[root@zejin240 I6000]# mysqlbinlog -vv log_bin.000003 | grep Xid
#160528  9:02:59 server id 9816000  end_log_pos 947 CRC32 0x37371b3e     Xid = 7
#160528 10:03:04 server id 9816000  end_log_pos 2969 CRC32 0xf7973829     Xid = 9
#160528 10:03:13 server id 9816000  end_log_pos 3168 CRC32 0x8db2aa41     Xid = 10

發現居然沒有此位置的xid，緣由在於咱們在最上面更新mysql.user表了，而此表是myisam表，不會有Xid的記錄，這種狀況時咱們根據上下位置把Xid設置爲8

step6:在node241上僞造一個grastate.dat文件（從node240上覆制過來，更改下seqno便可）：

# GALERA saved state
version: 2.1
uuid:    6c86dc17-246f-11e6-9955-ae4d7e89eed2
seqno:   8
cert_index:

step7:啓動新節點：

啓動node241:
[root@zejin241 mysql]# /usr/local/pxc_56/bin/mysqld_safe --defaults-file=/home/mysql/pxc6000.cnf --ledir=/usr/local/pxc_56/bin/ & 
[1] 56425
[root@zejin241 mysql]# 160528 11:06:03 mysqld_safe Logging to '/home/mysql/I6000/error.log'.
160528 11:06:03 mysqld_safe Starting mysqld daemon with databases from /home/mysql/I6000
160528 11:06:03 mysqld_safe Skipping wsrep-recover for 6c86dc17-246f-11e6-9955-ae4d7e89eed2:8 pair
160528 11:06:03 mysqld_safe Assigning 6c86dc17-246f-11e6-9955-ae4d7e89eed2:8 to wsrep_start_position



查看下啓動過程的日誌：
[root@zejin241 I6000]# cat error.log 
160528 11:06:03 mysqld_safe Starting mysqld daemon with databases from /home/mysql/I6000
160528 11:06:03 mysqld_safe Skipping wsrep-recover for 6c86dc17-246f-11e6-9955-ae4d7e89eed2:8 pair
160528 11:06:03 mysqld_safe Assigning 6c86dc17-246f-11e6-9955-ae4d7e89eed2:8 to wsrep_start_position
2016-05-28 11:06:19 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-05-28 11:06:19 0 [Note] /usr/local/pxc_56/bin//mysqld (mysqld 5.6.26-74.0-25.12-log) starting as process 56931 ...
2016-05-28 11:06:19 56931 [Note] WSREP: Read nil XID from storage engines, skipping position init
2016-05-28 11:06:19 56931 [Note] WSREP: wsrep_load(): loading provider library '/usr/local/pxc_56/lib/libgalera_smm.so'
2016-05-28 11:06:20 56931 [Note] WSREP: wsrep_load(): Galera 3.12.2(rf3e626d) by Codership Oy <info@codership.com> loaded successfully.
2016-05-28 11:06:20 56931 [Note] WSREP: CRC-32C: using hardware acceleration.
2016-05-28 11:06:20 56931 [Note] WSREP: Found saved state: 6c86dc17-246f-11e6-9955-ae4d7e89eed2:8
2016-05-28 11:06:20 56931 [Note] WSREP: Passing config to GCS: base_dir = /home/mysql/I6000/; base_host = 192.168.1.241; base_port = 6000; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /home/mysql/I6000/; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /home/mysql/I6000//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://192.168.1.241:6030; gmcast.segment = 0; gmcast.version = 0; ist.recv_a
2016-05-28 11:06:21 56931 [Note] WSREP: Service thread queue flushed.
2016-05-28 11:06:21 56931 [Note] WSREP: Assign initial position for certification: 8, protocol version: -1
2016-05-28 11:06:21 56931 [Note] WSREP: wsrep_sst_grab()
2016-05-28 11:06:21 56931 [Note] WSREP: Start replication
2016-05-28 11:06:21 56931 [Note] WSREP: Setting initial position to 6c86dc17-246f-11e6-9955-ae4d7e89eed2:8
2016-05-28 11:06:21 56931 [Note] WSREP: protonet asio version 0
2016-05-28 11:06:21 56931 [Note] WSREP: Using CRC-32C for message checksums.
2016-05-28 11:06:21 56931 [Note] WSREP: backend: asio
2016-05-28 11:06:21 56931 [Warning] WSREP: access file(/home/mysql/I6000//gvwstate.dat) failed(No such file or directory)
2016-05-28 11:06:21 56931 [Note] WSREP: restore pc from disk failed
2016-05-28 11:06:21 56931 [Note] WSREP: GMCast version 0
2016-05-28 11:06:21 56931 [Note] WSREP: (275a2ebd, 'tcp://192.168.1.241:6030') listening at tcp://192.168.1.241:6030
2016-05-28 11:06:21 56931 [Note] WSREP: (275a2ebd, 'tcp://192.168.1.241:6030') multicast: , ttl: 1
2016-05-28 11:06:21 56931 [Note] WSREP: EVS version 0
2016-05-28 11:06:21 56931 [Note] WSREP: gcomm: connecting to group 'pxc_zejin', peer '192.168.1.240:6030,192.168.1.241:6030,192.168.1.242:6030'
2016-05-28 11:06:21 56931 [Note] WSREP: (275a2ebd, 'tcp://192.168.1.241:6030') turning message relay requesting on, nonlive peers: 
2016-05-28 11:06:21 56931 [Note] WSREP: declaring 6c48c86c at tcp://192.168.1.240:6030 stable
2016-05-28 11:06:21 56931 [Note] WSREP: Node 6c48c86c state prim
2016-05-28 11:06:21 56931 [Note] WSREP: view(view_id(PRIM,275a2ebd,2) memb {
    275a2ebd,0
    6c48c86c,0
} joined {
} left {
} partitioned {
})
2016-05-28 11:06:21 56931 [Note] WSREP: save pc into disk
2016-05-28 11:06:21 56931 [Note] WSREP: discarding pending addr without UUID: tcp://192.168.1.242:6030
2016-05-28 11:06:22 56931 [Note] WSREP: gcomm: connected
2016-05-28 11:06:22 56931 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2016-05-28 11:06:22 56931 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2016-05-28 11:06:22 56931 [Note] WSREP: Opened channel 'pxc_zejin'
2016-05-28 11:06:22 56931 [Note] WSREP: Waiting for SST to complete.
2016-05-28 11:06:22 56931 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2016-05-28 11:06:22 56931 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 2800468f-2481-11e6-8c7c-de1ade51709a
2016-05-28 11:06:22 56931 [Note] WSREP: STATE EXCHANGE: sent state msg: 2800468f-2481-11e6-8c7c-de1ade51709a
2016-05-28 11:06:22 56931 [Note] WSREP: STATE EXCHANGE: got state msg: 2800468f-2481-11e6-8c7c-de1ade51709a from 0 (node6000_241)
2016-05-28 11:06:22 56931 [Note] WSREP: STATE EXCHANGE: got state msg: 2800468f-2481-11e6-8c7c-de1ade51709a from 1 (node6000_240)
2016-05-28 11:06:22 56931 [Note] WSREP: Quorum results:
    version    = 3,
    component  = PRIMARY,
    conf_id    = 1,
    members    = 1/2 (joined/total),
    act_id     = 10,
    last_appl. = -1,
    protocols  = 0/7/3 (gcs/repl/appl),
    group UUID = 6c86dc17-246f-11e6-9955-ae4d7e89eed2
2016-05-28 11:06:22 56931 [Note] WSREP: Flow-control interval: [23, 23]
2016-05-28 11:06:22 56931 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 10)
2016-05-28 11:06:22 56931 [Note] WSREP: State transfer required: 
    Group state: 6c86dc17-246f-11e6-9955-ae4d7e89eed2:10
    Local state: 6c86dc17-246f-11e6-9955-ae4d7e89eed2:8
2016-05-28 11:06:22 56931 [Note] WSREP: New cluster view: global state: 6c86dc17-246f-11e6-9955-ae4d7e89eed2:10, view# 2: Primary, number of nodes: 2, my index: 0, protocol version 3
2016-05-28 11:06:22 56931 [Warning] WSREP: Gap in state sequence. Need state transfer.
2016-05-28 11:06:22 56931 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.1.241:6020' --datadir '/home/mysql/I6000/' --defaults-file '/home/mysql/pxc6000.cnf' --defaults-group-suffix '' --parent '56931'  '' '
2016-05-28 11:06:24 56931 [Note] WSREP: (275a2ebd, 'tcp://192.168.1.241:6030') turning message relay requesting off
WSREP_SST: [INFO] Streaming with xbstream (20160528 11:06:25.322)
WSREP_SST: [INFO] Using socat as streamer (20160528 11:06:25.324)
WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u TCP-LISTEN:6020,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160528 11:06:25.891)
2016-05-28 11:06:27 56931 [Note] WSREP: Prepared SST request: xtrabackup-v2|192.168.1.241:6020/xtrabackup_sst//1
2016-05-28 11:06:27 56931 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2016-05-28 11:06:27 56931 [Note] WSREP: REPL Protocols: 7 (3, 2)
2016-05-28 11:06:27 56931 [Note] WSREP: Service thread queue flushed.
2016-05-28 11:06:27 56931 [Note] WSREP: Assign initial position for certification: 10, protocol version: 3
2016-05-28 11:06:27 56931 [Note] WSREP: Service thread queue flushed.
2016-05-28 11:06:27 56931 [Note] WSREP: Prepared IST receiver, listening at: tcp://192.168.1.241:6031
2016-05-28 11:06:27 56931 [Note] WSREP: Member 0.0 (node6000_241) requested state transfer from '*any*'. Selected 1.0 (node6000_240)(SYNCED) as donor.
2016-05-28 11:06:27 56931 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 10)
2016-05-28 11:06:27 56931 [Note] WSREP: Requesting state transfer: success, donor: 1
2016-05-28 11:06:31 56931 [Note] WSREP: 1.0 (node6000_240): State transfer to 0.0 (node6000_241) complete.
2016-05-28 11:06:31 56931 [Note] WSREP: Member 1.0 (node6000_240) synced with group.
WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20160528 11:06:31.257)
WSREP_SST: [INFO] Galera co-ords from recovery: 6c86dc17-246f-11e6-9955-ae4d7e89eed2:8 (20160528 11:06:31.261)
WSREP_SST: [INFO] Total time on joiner: 0 seconds (20160528 11:06:31.265)
WSREP_SST: [INFO] Removing the sst_in_progress file (20160528 11:06:31.268)
2016-05-28 11:06:31 56931 [Note] WSREP: SST complete, seqno: 8
2016-05-28 11:06:31 56931 [Note] Plugin 'FEDERATED' is disabled.
2016-05-28 11:06:31 56931 [Note] InnoDB: Using atomics to ref count buffer pool pages
2016-05-28 11:06:31 56931 [Note] InnoDB: The InnoDB memory heap is disabled
2016-05-28 11:06:31 56931 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-05-28 11:06:31 56931 [Note] InnoDB: Memory barrier is not used
2016-05-28 11:06:31 56931 [Note] InnoDB: Compressed tables use zlib 1.2.3
2016-05-28 11:06:31 56931 [Note] InnoDB: Using Linux native AIO
2016-05-28 11:06:31 56931 [Note] InnoDB: Using CPU crc32 instructions
2016-05-28 11:06:31 56931 [Note] InnoDB: Initializing buffer pool, size = 100.0M
2016-05-28 11:06:31 56931 [Note] InnoDB: Completed initialization of buffer pool
2016-05-28 11:06:32 56931 [Note] InnoDB: Highest supported file format is Barracuda.
2016-05-28 11:06:32 56931 [Note] InnoDB: 128 rollback segment(s) are active.
2016-05-28 11:06:32 56931 [Note] InnoDB: Waiting for purge to start
2016-05-28 11:06:33 56931 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 1637910
2016-05-28 11:06:33 56931 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 2e7b3e12-2481-11e6-8fe4-000c2973aa48.
2016-05-28 11:06:33 56931 [Note] RSA private key file not found: /home/mysql/I6000//private_key.pem. Some authentication plugins will not work.
2016-05-28 11:06:33 56931 [Note] RSA public key file not found: /home/mysql/I6000//public_key.pem. Some authentication plugins will not work.
2016-05-28 11:06:33 56931 [Note] Server hostname (bind-address): '*'; port: 6000
2016-05-28 11:06:33 56931 [Note] IPv6 is available.
2016-05-28 11:06:33 56931 [Note]   - '::' resolves to '::';
2016-05-28 11:06:33 56931 [Note] Server socket created on IP: '::'.
2016-05-28 11:06:33 56931 [Warning] 'user' entry 'root@zejin240' ignored in --skip-name-resolve mode.
2016-05-28 11:06:33 56931 [Warning] 'user' entry '@zejin240' ignored in --skip-name-resolve mode.
2016-05-28 11:06:33 56931 [Warning] 'proxies_priv' entry '@ root@zejin240' ignored in --skip-name-resolve mode.
2016-05-28 11:06:33 56931 [Note] Event Scheduler: Loaded 0 events
2016-05-28 11:06:33 56931 [Note] WSREP: Signalling provider to continue.
2016-05-28 11:06:33 56931 [Note] WSREP: Initialized wsrep sidno 2
2016-05-28 11:06:33 56931 [Note] WSREP: SST received: 6c86dc17-246f-11e6-9955-ae4d7e89eed2:8
2016-05-28 11:06:33 56931 [Note] WSREP: Receiving IST: 2 writesets, seqnos 8-10
2016-05-28 11:06:33 56931 [Note] /usr/local/pxc_56/bin//mysqld: ready for connections.
Version: '5.6.26-74.0-25.12-log'  socket: '/home/mysql/I6000/mysql.sock'  port: 6000  Percona XtraDB Cluster binary (GPL) 5.6.26-25.12, Revision 624ef81, wsrep_25.12
2016-05-28 11:06:33 56931 [Note] WSREP: IST received: 6c86dc17-246f-11e6-9955-ae4d7e89eed2:10
2016-05-28 11:06:33 56931 [Note] WSREP: 0.0 (node6000_241): State transfer from 1.0 (node6000_240) complete.
2016-05-28 11:06:33 56931 [Note] WSREP: Shifting JOINER -> JOINED (TO: 10)
2016-05-28 11:06:33 56931 [Note] WSREP: Member 0.0 (node6000_241) synced with group.
2016-05-28 11:06:33 56931 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 10)
2016-05-28 11:06:33 56931 [Note] WSREP: Synchronized with group, ready for connections
2016-05-28 11:06:33 56931 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

成功用IST來恢復了。

step8 登錄新節點node241，查看數據是否正常：

[root@zejin241 I6000]# /usr/local/pxc_56/bin/mysql -uroot -p -h127.0.0.1 -P6000
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.6.26-74.0-25.12-log Percona XtraDB Cluster binary (GPL) 5.6.26-25.12, Revision 624ef81, wsrep_25.12

Copyright (c) 2009-2015 Percona LLC and/or its affiliates
Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| zejin              |
+--------------------+
4 rows in set (0.12 sec)

mysql> use zejin;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select * from t1;
+----+-------+
| id | name  |
+----+-------+
|  1 | chen  |
|  2 | li    |
|  3 | zhang |
|  4 | lin   |
|  5 | liang |
+----+-------+
5 rows in set (0.01 sec)

發現數據也已經所有同步完成。

step9:對比SST時的日誌輸出：

在數據複製到node241上咱們不進行僞造grastate.dat操做，直接啓動節點node241

[root@zejin241 I6000]# /usr/local/pxc_56/bin/mysqld_safe --defaults-file=/home/mysql/pxc6000.cnf --ledir=/usr/local/pxc_56/bin &
[1] 3861
[root@zejin241 I6000]# 160528 15:32:57 mysqld_safe Logging to '/home/mysql/I6000/error.log'.
160528 15:32:57 mysqld_safe Starting mysqld daemon with databases from /home/mysql/I6000
160528 15:32:57 mysqld_safe Skipping wsrep-recover for empty datadir: /home/mysql/I6000
160528 15:32:57 mysqld_safe Assigning 00000000-0000-0000-0000-000000000000:-1 to wsrep_start_position



查看錯誤日誌：
[root@zejin241 I6000]# cat error.log 
160528 15:32:57 mysqld_safe Starting mysqld daemon with databases from /home/mysql/I6000
160528 15:32:57 mysqld_safe Skipping wsrep-recover for empty datadir: /home/mysql/I6000
160528 15:32:57 mysqld_safe Assigning 00000000-0000-0000-0000-000000000000:-1 to wsrep_start_position
2016-05-28 15:32:58 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-05-28 15:32:58 0 [Note] /usr/local/pxc_56/bin/mysqld (mysqld 5.6.26-74.0-25.12-log) starting as process 4359 ...
2016-05-28 15:32:58 4359 [Note] WSREP: Read nil XID from storage engines, skipping position init
2016-05-28 15:32:58 4359 [Note] WSREP: wsrep_load(): loading provider library '/usr/local/pxc_56/lib/libgalera_smm.so'
2016-05-28 15:32:58 4359 [Note] WSREP: wsrep_load(): Galera 3.12.2(rf3e626d) by Codership Oy <info@codership.com> loaded successfully.
2016-05-28 15:32:58 4359 [Note] WSREP: CRC-32C: using hardware acceleration.
2016-05-28 15:32:58 4359 [Warning] WSREP: Could not open state file for reading: '/home/mysql/I6000//grastate.dat'
2016-05-28 15:32:58 4359 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
2016-05-28 15:32:58 4359 [Note] WSREP: Passing config to GCS: base_dir = /home/mysql/I6000/; base_host = 192.168.1.241; base_port = 6000; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /home/mysql/I6000/; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /home/mysql/I6000//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://192.168.1.241:6030; gmcast.segment = 0; gmcast.version = 0; ist.recv_a
2016-05-28 15:32:58 4359 [Note] WSREP: Service thread queue flushed.
2016-05-28 15:32:58 4359 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2016-05-28 15:32:58 4359 [Note] WSREP: wsrep_sst_grab()
2016-05-28 15:32:58 4359 [Note] WSREP: Start replication
2016-05-28 15:32:58 4359 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2016-05-28 15:32:58 4359 [Note] WSREP: protonet asio version 0
2016-05-28 15:32:58 4359 [Note] WSREP: Using CRC-32C for message checksums.
2016-05-28 15:32:58 4359 [Note] WSREP: backend: asio
2016-05-28 15:32:58 4359 [Warning] WSREP: access file(/home/mysql/I6000//gvwstate.dat) failed(No such file or directory)
2016-05-28 15:32:58 4359 [Note] WSREP: restore pc from disk failed
2016-05-28 15:32:58 4359 [Note] WSREP: GMCast version 0
2016-05-28 15:32:58 4359 [Note] WSREP: (666e7891, 'tcp://192.168.1.241:6030') listening at tcp://192.168.1.241:6030
2016-05-28 15:32:58 4359 [Note] WSREP: (666e7891, 'tcp://192.168.1.241:6030') multicast: , ttl: 1
2016-05-28 15:32:58 4359 [Note] WSREP: EVS version 0
2016-05-28 15:32:58 4359 [Note] WSREP: gcomm: connecting to group 'pxc_zejin', peer '192.168.1.240:6030,192.168.1.241:6030,192.168.1.242:6030'
2016-05-28 15:32:58 4359 [Note] WSREP: (666e7891, 'tcp://192.168.1.241:6030') turning message relay requesting on, nonlive peers: 
2016-05-28 15:32:58 4359 [Note] WSREP: declaring 2cadc8a9 at tcp://192.168.1.240:6030 stable
2016-05-28 15:32:58 4359 [Note] WSREP: Node 2cadc8a9 state prim
2016-05-28 15:32:58 4359 [Note] WSREP: view(view_id(PRIM,2cadc8a9,2) memb {
    2cadc8a9,0
    666e7891,0
} joined {
} left {
} partitioned {
})
2016-05-28 15:32:58 4359 [Note] WSREP: save pc into disk
2016-05-28 15:32:58 4359 [Note] WSREP: discarding pending addr without UUID: tcp://192.168.1.242:6030
2016-05-28 15:32:58 4359 [Note] WSREP: discarding pending addr proto entry 0x30f9560
2016-05-28 15:32:59 4359 [Note] WSREP: gcomm: connected
2016-05-28 15:32:59 4359 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2016-05-28 15:32:59 4359 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2016-05-28 15:32:59 4359 [Note] WSREP: Opened channel 'pxc_zejin'
2016-05-28 15:32:59 4359 [Note] WSREP: Waiting for SST to complete.
2016-05-28 15:32:59 4359 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
2016-05-28 15:32:59 4359 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2016-05-28 15:32:59 4359 [Note] WSREP: STATE EXCHANGE: sent state msg: 66ae3372-24a6-11e6-8d13-c77d57a8c632
2016-05-28 15:32:59 4359 [Note] WSREP: STATE EXCHANGE: got state msg: 66ae3372-24a6-11e6-8d13-c77d57a8c632 from 0 (node6000_240)
2016-05-28 15:32:59 4359 [Note] WSREP: STATE EXCHANGE: got state msg: 66ae3372-24a6-11e6-8d13-c77d57a8c632 from 1 (node6000_241)
2016-05-28 15:32:59 4359 [Note] WSREP: Quorum results:
    version    = 3,
    component  = PRIMARY,
    conf_id    = 1,
    members    = 1/2 (joined/total),
    act_id     = 12,
    last_appl. = -1,
    protocols  = 0/7/3 (gcs/repl/appl),
    group UUID = 6c86dc17-246f-11e6-9955-ae4d7e89eed2
2016-05-28 15:32:59 4359 [Note] WSREP: Flow-control interval: [23, 23]
2016-05-28 15:32:59 4359 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 12)
2016-05-28 15:32:59 4359 [Note] WSREP: State transfer required: 
    Group state: 6c86dc17-246f-11e6-9955-ae4d7e89eed2:12
    Local state: 00000000-0000-0000-0000-000000000000:-1
2016-05-28 15:32:59 4359 [Note] WSREP: New cluster view: global state: 6c86dc17-246f-11e6-9955-ae4d7e89eed2:12, view# 2: Primary, number of nodes: 2, my index: 1, protocol version 3
2016-05-28 15:32:59 4359 [Warning] WSREP: Gap in state sequence. Need state transfer.
2016-05-28 15:32:59 4359 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.1.241:6020' --datadir '/home/mysql/I6000/' --defaults-file '/home/mysql/pxc6000.cnf' --defaults-group-suffix '' --parent '4359'  '' '
WSREP_SST: [INFO] Streaming with xbstream (20160528 15:32:59.795)
WSREP_SST: [INFO] Using socat as streamer (20160528 15:32:59.797)
WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u TCP-LISTEN:6020,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160528 15:32:59.840)
2016-05-28 15:33:00 4359 [Note] WSREP: Prepared SST request: xtrabackup-v2|192.168.1.241:6020/xtrabackup_sst//1
2016-05-28 15:33:00 4359 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2016-05-28 15:33:00 4359 [Note] WSREP: REPL Protocols: 7 (3, 2)
2016-05-28 15:33:00 4359 [Note] WSREP: Service thread queue flushed.
2016-05-28 15:33:00 4359 [Note] WSREP: Assign initial position for certification: 12, protocol version: 3
2016-05-28 15:33:00 4359 [Note] WSREP: Service thread queue flushed.
2016-05-28 15:33:00 4359 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (6c86dc17-246f-11e6-9955-ae4d7e89eed2): 1 (Operation not permitted)
     at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
2016-05-28 15:33:00 4359 [Note] WSREP: Member 1.0 (node6000_241) requested state transfer from '*any*'. Selected 0.0 (node6000_240)(SYNCED) as donor.
2016-05-28 15:33:00 4359 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 12)
2016-05-28 15:33:00 4359 [Note] WSREP: Requesting state transfer: success, donor: 0
WSREP_SST: [INFO] Proceeding with SST (20160528 15:33:00.512)
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:6020,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160528 15:33:00.515)
WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20160528 15:33:00.517)
removed `/home/mysql/I6000/ib_logfile0'
removed `/home/mysql/I6000/ib_logfile1'
removed `/home/mysql/I6000/xtrabackup_info'
removed `/home/mysql/I6000/xtrabackup_checkpoints'
removed `/home/mysql/I6000/performance_schema/rwlock_instances.frm'
removed `/home/mysql/I6000/performance_schema/db.opt'
removed `/home/mysql/I6000/performance_schema/session_account_connect_attrs.frm'
removed `/home/mysql/I6000/performance_schema/performance_timers.frm'
removed `/home/mysql/I6000/performance_schema/setup_objects.frm'
removed `/home/mysql/I6000/performance_schema/events_statements_summary_by_user_by_event_name.frm'
removed `/home/mysql/I6000/performance_schema/events_statements_summary_by_digest.frm'
removed `/home/mysql/I6000/performance_schema/events_stages_history.frm'
removed `/home/mysql/I6000/performance_schema/events_waits_summary_by_user_by_event_name.frm'
removed `/home/mysql/I6000/performance_schema/setup_timers.frm'
removed `/home/mysql/I6000/performance_schema/events_waits_summary_by_account_by_event_name.frm'
removed `/home/mysql/I6000/performance_schema/events_waits_summary_by_thread_by_event_name.frm'
removed `/home/mysql/I6000/performance_schema/session_connect_attrs.frm'
removed `/home/mysql/I6000/performance_schema/events_stages_history_long.frm'
removed `/home/mysql/I6000/performance_schema/table_io_waits_summary_by_index_usage.frm'
removed `/home/mysql/I6000/performance_schema/table_io_waits_summary_by_table.frm'
removed `/home/mysql/I6000/performance_schema/host_cache.frm'
removed `/home/mysql/I6000/performance_schema/events_waits_summary_global_by_event_name.frm'
removed `/home/mysql/I6000/performance_schema/events_statements_summary_by_host_by_event_name.frm'
removed `/home/mysql/I6000/performance_schema/events_statements_current.frm'
removed `/home/mysql/I6000/performance_schema/events_statements_summary_by_thread_by_event_name.frm'
removed `/home/mysql/I6000/performance_schema/mutex_instances.frm'
removed `/home/mysql/I6000/performance_schema/events_waits_summary_by_host_by_event_name.frm'
removed `/home/mysql/I6000/performance_schema/objects_summary_global_by_type.frm'
removed `/home/mysql/I6000/performance_schema/events_waits_current.frm'
removed `/home/mysql/I6000/performance_schema/events_stages_summary_global_by_event_name.frm'
removed `/home/mysql/I6000/performance_schema/threads.frm'
removed `/home/mysql/I6000/performance_schema/events_statements_history.frm'
removed `/home/mysql/I6000/performance_schema/file_summary_by_instance.frm'
removed `/home/mysql/I6000/performance_schema/table_lock_waits_summary_by_table.frm'
removed `/home/mysql/I6000/performance_schema/setup_instruments.frm'
removed `/home/mysql/I6000/performance_schema/events_stages_summary_by_thread_by_event_name.frm'
removed `/home/mysql/I6000/performance_schema/users.frm'
removed `/home/mysql/I6000/performance_schema/socket_summary_by_instance.frm'
removed `/home/mysql/I6000/performance_schema/socket_instances.frm'
removed `/home/mysql/I6000/performance_schema/setup_consumers.frm'
removed `/home/mysql/I6000/performance_schema/events_stages_summary_by_host_by_event_name.frm'
removed `/home/mysql/I6000/performance_schema/events_waits_summary_by_instance.frm'
removed `/home/mysql/I6000/performance_schema/events_stages_current.frm'
removed `/home/mysql/I6000/performance_schema/socket_summary_by_event_name.frm'
removed `/home/mysql/I6000/performance_schema/events_statements_history_long.frm'
removed `/home/mysql/I6000/performance_schema/cond_instances.frm'
removed `/home/mysql/I6000/performance_schema/events_waits_history_long.frm'
removed `/home/mysql/I6000/performance_schema/events_waits_history.frm'
removed `/home/mysql/I6000/performance_schema/events_statements_summary_by_account_by_event_name.frm'
removed `/home/mysql/I6000/performance_schema/events_statements_summary_global_by_event_name.frm'
removed `/home/mysql/I6000/performance_schema/events_stages_summary_by_account_by_event_name.frm'
removed `/home/mysql/I6000/performance_schema/file_instances.frm'
removed `/home/mysql/I6000/performance_schema/hosts.frm'
removed `/home/mysql/I6000/performance_schema/setup_actors.frm'
removed `/home/mysql/I6000/performance_schema/events_stages_summary_by_user_by_event_name.frm'
removed `/home/mysql/I6000/performance_schema/accounts.frm'
removed `/home/mysql/I6000/performance_schema/file_summary_by_event_name.frm'
removed directory: `/home/mysql/I6000/performance_schema'
removed `/home/mysql/I6000/zejin/db.opt'
removed `/home/mysql/I6000/zejin/t1.ibd'
removed `/home/mysql/I6000/zejin/t1.frm'
removed directory: `/home/mysql/I6000/zejin'
removed `/home/mysql/I6000/xtrabackup_binlog_info'
removed `/home/mysql/I6000/backup-my.cnf'
removed `/home/mysql/I6000/xtrabackup_binlog_pos_innodb'
removed `/home/mysql/I6000/mysql/time_zone_transition_type.MYI'
removed `/home/mysql/I6000/mysql/time_zone_transition.MYI'
removed `/home/mysql/I6000/mysql/servers.MYD'
removed `/home/mysql/I6000/mysql/ndb_binlog_index.frm'
removed `/home/mysql/I6000/mysql/servers.MYI'
removed `/home/mysql/I6000/mysql/plugin.frm'
removed `/home/mysql/I6000/mysql/time_zone_transition_type.MYD'
removed `/home/mysql/I6000/mysql/proxies_priv.MYD'
removed `/home/mysql/I6000/mysql/proxies_priv.frm'
removed `/home/mysql/I6000/mysql/columns_priv.frm'
removed `/home/mysql/I6000/mysql/help_keyword.MYI'
removed `/home/mysql/I6000/mysql/func.frm'
removed `/home/mysql/I6000/mysql/proc.frm'
removed `/home/mysql/I6000/mysql/innodb_table_stats.ibd'
removed `/home/mysql/I6000/mysql/help_topic.frm'
removed `/home/mysql/I6000/mysql/slow_log.CSM'
removed `/home/mysql/I6000/mysql/help_relation.MYD'
removed `/home/mysql/I6000/mysql/proc.MYD'
removed `/home/mysql/I6000/mysql/help_category.MYD'
removed `/home/mysql/I6000/mysql/help_category.frm'
removed `/home/mysql/I6000/mysql/columns_priv.MYI'
removed `/home/mysql/I6000/mysql/help_topic.MYI'
removed `/home/mysql/I6000/mysql/event.frm'
removed `/home/mysql/I6000/mysql/db.frm'
removed `/home/mysql/I6000/mysql/columns_priv.MYD'
removed `/home/mysql/I6000/mysql/time_zone_transition.MYD'
removed `/home/mysql/I6000/mysql/innodb_index_stats.frm'
removed `/home/mysql/I6000/mysql/time_zone.MYI'
removed `/home/mysql/I6000/mysql/event.MYD'
removed `/home/mysql/I6000/mysql/slave_master_info.ibd'
removed `/home/mysql/I6000/mysql/help_topic.MYD'
removed `/home/mysql/I6000/mysql/help_relation.MYI'
removed `/home/mysql/I6000/mysql/db.MYD'
removed `/home/mysql/I6000/mysql/slave_relay_log_info.ibd'
removed `/home/mysql/I6000/mysql/user.MYI'
removed `/home/mysql/I6000/mysql/plugin.MYI'
removed `/home/mysql/I6000/mysql/proc.MYI'
removed `/home/mysql/I6000/mysql/help_relation.frm'
removed `/home/mysql/I6000/mysql/time_zone_name.MYD'
removed `/home/mysql/I6000/mysql/innodb_table_stats.frm'
removed `/home/mysql/I6000/mysql/time_zone.frm'
removed `/home/mysql/I6000/mysql/servers.frm'
removed `/home/mysql/I6000/mysql/general_log.CSV'
removed `/home/mysql/I6000/mysql/ndb_binlog_index.MYI'
removed `/home/mysql/I6000/mysql/func.MYD'
removed `/home/mysql/I6000/mysql/slave_worker_info.ibd'
removed `/home/mysql/I6000/mysql/innodb_index_stats.ibd'
removed `/home/mysql/I6000/mysql/time_zone_name.frm'
removed `/home/mysql/I6000/mysql/proxies_priv.MYI'
removed `/home/mysql/I6000/mysql/slow_log.frm'
removed `/home/mysql/I6000/mysql/tables_priv.MYI'
removed `/home/mysql/I6000/mysql/tables_priv.MYD'
removed `/home/mysql/I6000/mysql/procs_priv.frm'
removed `/home/mysql/I6000/mysql/help_keyword.MYD'
removed `/home/mysql/I6000/mysql/time_zone_leap_second.MYD'
removed `/home/mysql/I6000/mysql/tables_priv.frm'
removed `/home/mysql/I6000/mysql/ndb_binlog_index.MYD'
removed `/home/mysql/I6000/mysql/slow_log.CSV'
removed `/home/mysql/I6000/mysql/time_zone.MYD'
removed `/home/mysql/I6000/mysql/procs_priv.MYD'
removed `/home/mysql/I6000/mysql/slave_master_info.frm'
removed `/home/mysql/I6000/mysql/general_log.frm'
removed `/home/mysql/I6000/mysql/help_keyword.frm'
removed `/home/mysql/I6000/mysql/time_zone_transition_type.frm'
removed `/home/mysql/I6000/mysql/general_log.CSM'
removed `/home/mysql/I6000/mysql/time_zone_transition.frm'
removed `/home/mysql/I6000/mysql/time_zone_name.MYI'
removed `/home/mysql/I6000/mysql/event.MYI'
removed `/home/mysql/I6000/mysql/time_zone_leap_second.frm'
removed `/home/mysql/I6000/mysql/plugin.MYD'
removed `/home/mysql/I6000/mysql/slave_relay_log_info.frm'
removed `/home/mysql/I6000/mysql/help_category.MYI'
removed `/home/mysql/I6000/mysql/user.frm'
removed `/home/mysql/I6000/mysql/user.MYD'
removed `/home/mysql/I6000/mysql/procs_priv.MYI'
removed `/home/mysql/I6000/mysql/db.MYI'
removed `/home/mysql/I6000/mysql/time_zone_leap_second.MYI'
removed `/home/mysql/I6000/mysql/slave_worker_info.frm'
removed `/home/mysql/I6000/mysql/func.MYI'
removed directory: `/home/mysql/I6000/mysql'
removed `/home/mysql/I6000/ibdata1'
WSREP_SST: [INFO] Cleaning the binlog directory /home/mysql/I6000 as well (20160528 15:33:00.540)
rm: cannot remove `/home/mysql/I6000/*.index': No such file or directory
WSREP_SST: [INFO] Waiting for SST streaming to complete! (20160528 15:33:00.547)
2016-05-28 15:33:01 4359 [Note] WSREP: (666e7891, 'tcp://192.168.1.241:6030') turning message relay requesting off
2016-05-28 15:33:15 4359 [Note] WSREP: 0.0 (node6000_240): State transfer to 1.0 (node6000_241) complete.
2016-05-28 15:33:15 4359 [Note] WSREP: Member 0.0 (node6000_240) synced with group.
WSREP_SST: [INFO] Preparing the backup at /home/mysql/I6000//.sst (20160528 15:33:15.588)
WSREP_SST: [INFO] Evaluating innobackupex --no-version-check  --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20160528 15:33:15.591)
rm: cannot remove `/home/mysql/I6000//innobackup.prepare.log': No such file or directory
rm: cannot remove `/home/mysql/I6000//innobackup.move.log': No such file or directory
WSREP_SST: [INFO] Moving the backup to /home/mysql/I6000/ (20160528 15:33:20.551)
WSREP_SST: [INFO] Evaluating innobackupex --defaults-file=/home/mysql/pxc6000.cnf  --defaults-group=mysqld --no-version-check   --move-back --force-non-empty-directories ${DATA} &>${DATA}/innobackup.move.log (20160528 15:33:20.554)
WSREP_SST: [INFO] Move successful, removing /home/mysql/I6000//.sst (20160528 15:33:20.848)
WSREP_SST: [INFO] Galera co-ords from recovery: 6c86dc17-246f-11e6-9955-ae4d7e89eed2:12 (20160528 15:33:20.854)
WSREP_SST: [INFO] Total time on joiner: 0 seconds (20160528 15:33:20.857)
WSREP_SST: [INFO] Removing the sst_in_progress file (20160528 15:33:20.860)
2016-05-28 15:33:20 4359 [Note] WSREP: SST complete, seqno: 12
2016-05-28 15:33:20 4359 [Note] Plugin 'FEDERATED' is disabled.
2016-05-28 15:33:20 4359 [Note] InnoDB: Using atomics to ref count buffer pool pages
2016-05-28 15:33:20 4359 [Note] InnoDB: The InnoDB memory heap is disabled
2016-05-28 15:33:20 4359 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-05-28 15:33:20 4359 [Note] InnoDB: Memory barrier is not used
2016-05-28 15:33:20 4359 [Note] InnoDB: Compressed tables use zlib 1.2.3
2016-05-28 15:33:20 4359 [Note] InnoDB: Using Linux native AIO
2016-05-28 15:33:20 4359 [Note] InnoDB: Using CPU crc32 instructions
2016-05-28 15:33:20 4359 [Note] InnoDB: Initializing buffer pool, size = 100.0M
2016-05-28 15:33:20 4359 [Note] InnoDB: Completed initialization of buffer pool
2016-05-28 15:33:20 4359 [Note] InnoDB: Highest supported file format is Barracuda.
2016-05-28 15:33:20 4359 [Note] InnoDB: 128 rollback segment(s) are active.
2016-05-28 15:33:20 4359 [Note] InnoDB: Waiting for purge to start
2016-05-28 15:33:21 4359 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 1650198
2016-05-28 15:33:21 4359 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 73f9a99b-24a6-11e6-a163-000c2973aa48.
2016-05-28 15:33:21 4359 [Note] RSA private key file not found: /home/mysql/I6000//private_key.pem. Some authentication plugins will not work.
2016-05-28 15:33:21 4359 [Note] RSA public key file not found: /home/mysql/I6000//public_key.pem. Some authentication plugins will not work.
2016-05-28 15:33:21 4359 [Note] Server hostname (bind-address): '*'; port: 6000
2016-05-28 15:33:21 4359 [Note] IPv6 is available.
2016-05-28 15:33:21 4359 [Note]   - '::' resolves to '::';
2016-05-28 15:33:21 4359 [Note] Server socket created on IP: '::'.
2016-05-28 15:33:21 4359 [Warning] 'user' entry 'root@zejin240' ignored in --skip-name-resolve mode.
2016-05-28 15:33:21 4359 [Warning] 'user' entry '@zejin240' ignored in --skip-name-resolve mode.
2016-05-28 15:33:21 4359 [Warning] 'proxies_priv' entry '@ root@zejin240' ignored in --skip-name-resolve mode.
2016-05-28 15:33:21 4359 [Note] Event Scheduler: Loaded 0 events
2016-05-28 15:33:21 4359 [Note] WSREP: Signalling provider to continue.
2016-05-28 15:33:21 4359 [Note] WSREP: Initialized wsrep sidno 2
2016-05-28 15:33:21 4359 [Note] WSREP: SST received: 6c86dc17-246f-11e6-9955-ae4d7e89eed2:12
2016-05-28 15:33:21 4359 [Note] /usr/local/pxc_56/bin/mysqld: ready for connections.
Version: '5.6.26-74.0-25.12-log'  socket: '/home/mysql/I6000/mysql.sock'  port: 6000  Percona XtraDB Cluster binary (GPL) 5.6.26-25.12, Revision 624ef81, wsrep_25.12
2016-05-28 15:33:21 4359 [Note] WSREP: 1.0 (node6000_241): State transfer from 0.0 (node6000_240) complete.
2016-05-28 15:33:21 4359 [Note] WSREP: Shifting JOINER -> JOINED (TO: 12)
2016-05-28 15:33:21 4359 [Note] WSREP: Member 1.0 (node6000_241) synced with group.
2016-05-28 15:33:21 4359 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 12)
2016-05-28 15:33:21 4359 [Note] WSREP: Synchronized with group, ready for connections
2016-05-28 15:33:21 4359 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

能夠看到裏面有大量的removed操做，它會刪除數據文件夾裏面的全部文件，再從主節點那裏經過SST獲取數據
官方說明：
    XtraBackup SST works in two stages:
      • Stage I on joiner checks if it is SST or IST based on presence of xtrabackup_ist file.
      • In Stage II it starts the data transfer, if it’s SST, it empties the data directory sans few files (galera.cache, sst_in_progress, grastate.dat) and then proceed with the SST or if it’s IST, proceeds as before.

至此咱們完成了pxc環境下避免新節點加入時須要進行SST的方法，在數據量大時SST將會佔用大量的資源，配置的gcache.size足夠容納從備份到恢復此時間段內主節點產生的數據量，建議經過業務低峯期時經過腳原本實現此新節點加入操做。