摘要mysql
GITD是惟一標示符是由服務器的UUID(全局惟一標示對於任何一個服務器都是128位隨機符)並結合事務ID號來組合成一個惟一的標示,某一個主機上某一個事務的標識碼 就叫作GTIDlinux
mysql
sql
MySQL 5.6 基於GTID 進行主從複製docker
1、什麼是GTID?數據庫
2、GTID工做方式vim
3、MySQL主從複製原理安全
4、主從節點數據庫配置服務器
5、演示多線程
MySQL 複製過濾功能介紹異步
GTID全稱GTIDs are unique identifiers comprising the server UUID (of the original master) and a transaction number
GITD是惟一標示符是由服務器的UUID(全局惟一標示對於任何一個服務器都是128位隨機符)並結合事務ID號來組合成一個惟一的標示,某一個主機上某一個事務的標識碼 就叫作GTID 自從引入GTID以後 MySQL5.6的binlog在每個事務的首部都會寫上GTID的標示,在二進制文件中,每個事務的語句記錄下來的時候在這個事件首部會把相關聯的GTID給記錄下來。 所以GTID使得追蹤和比較複製事務變得很是簡單,並且可以實現從崩潰中快速進行恢復。
尤爲是innodb引擎,要想使用高可用功能,必需要依靠GTID實現
Mysql5.6引入了一個新特性多線程複製,有了gtid相關的功能以後,咱們Slave服務器上所謂的多線程複製是咱們Master服務器上有多顆cpu,多個事務同時執行的時候,會比Slave服務器讀取事務的速度要快,從庫存放的只有一個線程。這樣在多線程複製的時候Slave服務器會比Master服務器落後不少. 從服務器能夠開啓多個進程,並行的讀取數據存放在本地。可是好比說咱們bbs庫,2個線程互相讀取數據,會形成問題。 Mysql5.6 多線程複製每一個數據庫也僅能使用一個線程,這樣在咱們多線程複製的時候只有多個庫進行復制,咱們的多線程複製纔有意義
這個主從複製原理基本上是mysql的基礎知識,我這裏在給你們介紹一遍
MySQL的主從複製是一個異步的複製過程(雖然通常狀況下感受是實時的),數據將從一個Mysql數據庫(咱們稱之爲Master)複製到另外一個Mysql數據庫(咱們稱之爲Slave),在Master與Slave之間實現整個主從複製的過程是由三個線程參與完成的。其中有兩個線程(SQL線程和IO線程)在Slave端,另外一個線程(I/O線程)在Master端。 要實現MySQL的主從複製,首先必須打開Master端的binlog記錄功能,不然就沒法實現。由於整個複製過程實際上就是Slave從aster端獲取binlog日誌,而後再在Slave上以相同順序執行獲取的binlog日誌中的記錄的各類SQL操做
從庫生成兩個線程,一個I/O線程,一個SQL線程;
i/o線程去請求主庫 的binlog,並將獲得的binlog日誌寫到relay log(中繼日誌) 文件中;
主庫會生成一個 log dump 線程,用來給從庫 i/o線程傳binlog;
SQL 線程,會讀取relay log文件中的日誌,並解析成具體操做,來實現主從的操做一致,而最終數據一致;
1.1環境以下
關於MySQL 5.6的安裝我就不介紹了,網上太多了,你們自行百度吧!
IP地址 | mysql版本 | 系統版本 | 做用 | 端口 |
---|---|---|---|---|
192.168.11.11 | MySQL5.6 | CentOS Linux release 7.2.1511 | Master | 3306 |
192.168.11.12 | MySQL5.6 | CentOS Linux release 7.2.1511 | Slave | 3306 |
關閉防火牆、Selinux、調整字符集 (2臺都請執行,固然一臺使用多實例也能夠)
systemctl stop firewalldsed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/configgrep SELINUX=disabled /etc/selinux/configsetenforce 0echo '* - nofile 100000 ' >>/etc/security/limits.conf
1.2配置Master節點my.cnf
[client]port = 3306socket = /usr/local/mysql/tmp/mysql.sock[mysqld]########mysql master GTID conf##############binlog-format=ROWlog-bin=master-binlog-slave-updates=truegtid-mode=onenforce-gtid-consistency=truemaster-info-repository=TABLErelay-log-info-repository=TABLEsync-master-info=1slave-parallel-workers=2binlog-checksum=CRC32master-verify-checksum=1slave-sql-verify-checksum=1binlog-rows-query-log_events=1server-id=1report-port=3306report-host=master.abcdocker.com#######################################port = 3306basedir = /usr/local/mysql/socket = /usr/local/mysql/tmp/mysql.sockpid-file = /usr/local/mysql/var/mysql.piddatadir = /usr/local/mysql/var/tmpdir = /usr/local/mysql/tmp/slave-load-tmpdir = /usr/local/mysql/tmp/!include /usr/local/mysql/etc/mysqld.cnf# skip leverskip-name-resolveskip-symbolic-linksskip-external-lockingskip-slave-start#thread leveltable_open_cache = 2048#############connect############back_log = 50max_connections = 1000max_connect_errors = 10000#open_files_limit = 10240##############timeout###########connect-timeout = 10wait-timeout = 800interactive-timeout = 800slave-net-timeout = 60net_read_timeout = 30net_write_timeout = 60net_retry_count = 10net_buffer_length = 16384max_allowed_packet = 64M################# cache #############table_open_cache = 2048thread_stack = 192Kthread_cache_size = 100thread_concurrency = 16# qcache settingsquery_cache_size = 20mquery_cache_limit = 2Mquery_cache_min_res_unit = 2K# default settings# time zonedefault-time-zone = systemcharacter-set-server = utf8default-storage-engine = InnoDB# tmp & heap tmp_table_size = 512Mmax_heap_table_size = 512Mlog-bin = mysql-binlog-bin-index = mysql-bin.indexrelay-log = relay-logrelay_log_index = relay-log.index# warning & error loglog-warnings = 1log-error = /usr/local/mysql/log/mysql.errlog-output = FILE# slow query logslow_query_log = 1long-query-time = 1slow_query_log_file = /usr/local/mysql/log/slow.log#log-queries-not-using-indexes#log-slow-slave-statementsgeneral_log = 1general_log_file = /usr/local/mysql/log/mysql.logmax_binlog_size = 1Gmax_relay_log_size = 1G# if use auto-ex, set to 0relay-log-purge = 1# max binlog keeps daysexpire_logs_days = 7binlog_cache_size = 1M# replicationreplicate-wild-ignore-table = mysql.%replicate-wild-ignore-table = test.%# slave_skip_errors=allkey_buffer_size = 30Msort_buffer_size = 2Mread_buffer_size = 2Mjoin_buffer_size = 8Mread_rnd_buffer_size = 8Mbulk_insert_buffer_size = 64Mmyisam_sort_buffer_size = 64Mmyisam_max_sort_file_size = 10Gmyisam_repair_threads = 1myisam_recovergroup_concat_max_len = 64Ktransaction_isolation = REPEATABLE-READinnodb_file_per_table#############mysql5.5new################innodb_file_format = Barracuda########################################innodb_additional_mem_pool_size = 10Minnodb_buffer_pool_size = 200Minnodb_data_home_dir = /usr/local/mysql/var/innodb_data_file_path = ibdata1:100M:autoextend################mysql5.5new############innodb_read_io_threads = 8innodb_write_io_threads = 8innodb_purge_threads = 1########################################innodb_thread_concurrency = 16innodb_flush_log_at_trx_commit = 1innodb_log_buffer_size = 16Minnodb_log_file_size = 50Minnodb_log_files_in_group = 2innodb_log_group_home_dir = /usr/local/mysql/var/innodb_max_dirty_pages_pct = 90innodb_lock_wait_timeout = 50#innodb_flush_method = O_DSYNC################mysql5.6 new#####################innodb_buffer_pool_instances = 16innodb_change_buffering = allinnodb_adaptive_flushing = 1innodb_io_capacity = 2000innodb_old_blocks_time = 1000innodb_stats_on_metadata = 0#################################################old-passwords = 0[mysqldump]quickmax_allowed_packet = 64M[mysql]no-auto-rehashdefault-character-set = utf8connect-timeout = 3[myisamchk]key_buffer_size = 256Msort_buffer_size = 256Mread_buffer = 2Mwrite_buffer = 2M[mysqlhotcopy]interactive-timeout#提示:配置文件除了上面mysql master GTID conf註釋下的是咱們GTID須要用的,其餘的你們能夠用默認就能夠。我這個配置比較全
1.3配置Master詳解
[mysqld]binlog-format=ROW #行記錄log-bin=master-bin #二進制文件log-slave-updates=true #表示當從服務器從中繼日誌中讀取事件在本地應用的時候,是否把操做記錄到本地的二進制文件中(爲了GTID複製的安全,咱們啓用這項) 主服務器必須啓用,從服務器能夠不啓用gtid-mode=on #gtid-mode=是否啓用GTID功能enforce-gtid-consistency=true #是否強制GTID一致性(所謂GTID一致性指的是在記錄到二進制的時候有些特殊的語句可以設計數據庫修改的語句它是不記錄的,可是能夠靠其餘功能保證數據一致性。主要和臨時建立的表語句有關)在MySQL 5.6.2以前,slave記錄的master信息以及slave應用binlog的信息存放在文件中,即master.info與relay-log.info。在5.6.2版本以後,容許記錄到table中master-info-repository=TABLE #主要用於主服務器記錄從服務鏈接信息和每個從服務器的二進制文件及其相關事件位置等信息記錄到一個文件中(master.info)或一個表relay-log-info-repository=TABLE #讓從服務器本身記錄鏈接的主服務器是誰,主服務器哪個二進制文件,及二進制文件的位置等等對應的表分別爲mysql.slave_master_info與mysql.slave_relay_log_info,且這兩個表均爲innodb引擎表master info與relay info還有3個參數控制刷新:sync_relay_log:默認爲10000,即每10000次sync_relay_log事件會刷新到磁盤。爲0則表示不刷新,交由OS的cache控制。sync_master_info:若master-info-repository爲FILE,當設置爲0,則每次sync_master_info事件都會刷新到磁盤,默認爲10000次刷新到磁盤;若master-info-repository爲TABLE,當設置爲0,則表不作任何更新,設置爲1,則每次事件會更新表 #默認爲10000sync_relay_log_info:若relay_log_info_repository爲FILE,當設置爲0,交由OS刷新磁盤,默認爲10000次刷新到磁盤;若relay_log_info_repository爲TABLE,且爲INNODB存儲,則不管爲任何值,則都每次evnet都會更新表。建議參數設置以下:sync_relay_log = 1sync_master_info = 1sync_relay_log_info = 1在GTID模式下,每個從服務器在連入主服務器的時候,它必需要告訴主服務器本身的主機名和IP地址and端口號這樣咱們在主服務器使用show slave hosts能夠看到當前主服務器鏈接的從服務器信息report-port= 從服務器的端口report-host= 從服務器的主機名report-host=master.abcdocker.comreport-port=3306server-id=1 #同一個複製拓撲中的全部服務器的ID號必須惟一從服務器並行的線程數slave-parallel-workers=2workers指的是進程後面的進程數儘可能等於數據庫的個數或者小於,大於數據庫的數量是沒有意義的若是=0 表示禁用多線程功能 binlog-checksum、master-verify-checksum和slave-sql-verify-checksum:啓用複製有關的全部校驗功能binlog-checksum=CRC32 #主服務器端在啓動的時候要不要校驗binlog自己的校驗碼master-verify-checksum=1slave-sql-verify-checksum=1binlog-rows-query-log_events=1 #用於在二進制日誌詳細記錄事件相關的信息,可下降故障排除的複雜度;datadir"/mydata/datasocket=/tmp/mysql.sock #mysql.sock的做用是server和client在同一臺服務器,而且使用localhost進行連接的時候,就會使用socket來進行鏈接log-slave-updates、gtid-mode、enforce-gtid-consistency、report-port和report-host:用於啓動GTID及知足附屬的其餘需求;
1.4配置Slave詳解
[mysqld]binlog-format=ROWlog-slave-updates=truegtid-mode=onenforce-gtid-consistency=truemaster-info-repository=TABLErelay-log-info-repository=TABLEsync-master-info=1slave-parallel-workers=2binlog-checksum=CRC32master-verify-checksum=1slave-sql-verify-checksum=1binlog-rows-query-log_events=1server-id=11 #須要和master不同report-port=3306port=3306log-bin=mysql.bin.logreport-host=slave.abcdocker.com #名字也要和master不同#從節點不須要定義二進制日誌,也能夠不定義中繼日誌
1.5配置Slave節點配置my.cnf
[client]port = 3306socket = /usr/local/mysql/tmp/mysql.sock[mysqld]#############slave GTID conf#################################binlog-format=ROWlog-slave-updates=truegtid-mode=onenforce-gtid-consistency=truemaster-info-repository=TABLErelay-log-info-repository=TABLEsync-master-info=1slave-parallel-workers=2binlog-checksum=CRC32master-verify-checksum=1slave-sql-verify-checksum=1binlog-rows-query-log_events=1server-id=11report-port=3306report-host=slave.abcdocker.com########################################################port = 3306basedir = /usr/local/mysql/socket = /usr/local/mysql/tmp/mysql.sockpid-file = /usr/local/mysql/var/mysql.piddatadir = /usr/local/mysql/var/tmpdir = /usr/local/mysql/tmp/slave-load-tmpdir = /usr/local/mysql/tmp/!include /usr/local/mysql/etc/mysqld.cnf# skip leverskip-name-resolveskip-symbolic-linksskip-external-lockingskip-slave-start#thread leveltable_open_cache = 2048#############connect############back_log = 50max_connections = 1000max_connect_errors = 10000#open_files_limit = 10240##############timeout###########connect-timeout = 10wait-timeout = 800interactive-timeout = 800slave-net-timeout = 60net_read_timeout = 30net_write_timeout = 60net_retry_count = 10net_buffer_length = 16384max_allowed_packet = 64M################# cache #############table_open_cache = 2048thread_stack = 192Kthread_cache_size = 100thread_concurrency = 16# qcache settingsquery_cache_size = 20mquery_cache_limit = 2Mquery_cache_min_res_unit = 2K# default settings# time zonedefault-time-zone = systemcharacter-set-server = utf8default-storage-engine = InnoDB# tmp & heap tmp_table_size = 512Mmax_heap_table_size = 512Mlog-bin = mysql-binlog-bin-index = mysql-bin.indexrelay-log = relay-logrelay_log_index = relay-log.index# warning & error loglog-warnings = 1log-error = /usr/local/mysql/log/mysql.errlog-output = FILE# slow query logslow_query_log = 1long-query-time = 1slow_query_log_file = /usr/local/mysql/log/slow.log#log-queries-not-using-indexes#log-slow-slave-statementsgeneral_log = 1general_log_file = /usr/local/mysql/log/mysql.logmax_binlog_size = 1Gmax_relay_log_size = 1G# if use auto-ex, set to 0relay-log-purge = 1# max binlog keeps daysexpire_logs_days = 7binlog_cache_size = 1M# replicationreplicate-wild-ignore-table = mysql.%replicate-wild-ignore-table = test.%# slave_skip_errors=allkey_buffer_size = 30Msort_buffer_size = 2Mread_buffer_size = 2Mjoin_buffer_size = 8Mread_rnd_buffer_size = 8Mbulk_insert_buffer_size = 64Mmyisam_sort_buffer_size = 64Mmyisam_max_sort_file_size = 10Gmyisam_repair_threads = 1myisam_recovergroup_concat_max_len = 64Ktransaction_isolation = REPEATABLE-READinnodb_file_per_table#############mysql5.5new################innodb_file_format = Barracuda########################################innodb_additional_mem_pool_size = 10Minnodb_buffer_pool_size = 200Minnodb_data_home_dir = /usr/local/mysql/var/innodb_data_file_path = ibdata1:100M:autoextend################mysql5.5new############innodb_read_io_threads = 8innodb_write_io_threads = 8innodb_purge_threads = 1########################################innodb_thread_concurrency = 16innodb_flush_log_at_trx_commit = 1innodb_log_buffer_size = 16Minnodb_log_file_size = 50Minnodb_log_files_in_group = 2innodb_log_group_home_dir = /usr/local/mysql/var/innodb_max_dirty_pages_pct = 90innodb_lock_wait_timeout = 50#innodb_flush_method = O_DSYNC################mysql5.6 new#####################innodb_buffer_pool_instances = 16innodb_change_buffering = allinnodb_adaptive_flushing = 1innodb_io_capacity = 2000innodb_old_blocks_time = 1000innodb_stats_on_metadata = 0#################################################old-passwords = 0[mysqldump]quickmax_allowed_packet = 64M[mysql]no-auto-rehashdefault-character-set = utf8connect-timeout = 3[myisamchk]key_buffer_size = 256Msort_buffer_size = 256Mread_buffer = 2Mwrite_buffer = 2M[mysqlhotcopy]interactive-timeout
1.6 Master 上建立複製用戶
grant replication slave on *.* to repl@192.168.11.12 identified by '123456';
說明:192.168.11.12是從節點服務器,若是想一次性受權更多的節點,能夠自行根據須要修改。
1.7 爲備節點提供初始數據集
鎖定主表,備份主節點上的數據,將其還原至從節點;若是沒有GTID,在備份時須要在master上使用show master status命令查看二進制文件名稱及事件位置,以便後面啓動slave節點時使用。
1.8 啓動Slave節點的複製線程
若是啓動了GTID功能,則使用以下命令;
CHANGE MASTER TO MASTER_HOST='master.abcdocker.com',MASTER_USER='repl',MASTER_PASSWORD='123456',MASTER_AUTO_POSITION=1;
MASTER_AUTO_POSITION 該參數在mysql5.6.5版本引入,若是進行change master to時使用MASTER_AUTO_POSITION = 1,slave鏈接master將使用基於GTID的複製協議。
使用基於GTID協議的複製,slave會告訴master它已經接收到或執行了哪些事務。計算這個集,slave須要讀取全局參數gtid_executed以及經過show slave status獲取的參數Retrieved_gtid_set。
結果集做爲初次握手的一部分,發送到master,master發回它已經執行的且不在結果集這部分的全部事務。若是這些事務在master的binlog文件中已經被清除,master將會發送一個
提示:主從服務器節點時間要同步,主機名最好能夠解析。
啓動從庫
mysql> start slave;
查看主從狀態
mysql> show slave status\G*************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: master.abcdocker.com Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000005 Read_Master_Log_Pos: 814 Relay_Log_File: relay-log.000002 Relay_Log_Pos: 1024 Relay_Master_Log_File: mysql-bin.000005 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: mysql.%,test.% Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 814 Relay_Log_Space: 1222 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_UUID: a961eee5-ae7a-11e7-89f1-000c29020966 Master_Info_File: mysql.slave_master_info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: a961eee5-ae7a-11e7-89f1-000c29020966:1-3 Executed_Gtid_Set: a961eee5-ae7a-11e7-89f1-000c29020966:1-3 Auto_Position: 11 row in set (0.00 sec)
1.1 主庫建立庫
master-mysql> create database shopping;Query OK, 1 row affected (0.00 sec)
從庫查看
slave-mysql> show databases;+--------------------+| Database |+--------------------+| information_schema || mysql || performance_schema || shopping || test |+--------------------+5 rows in set (0.01 sec)
這裏已經出來了,咱們導入點數據
1.2 導出一份數據
[root@abcdocker ~]# mysqldump -uroot -p --hex-blob --routines --triggers --single-transaction --default-character-set=utf8 shopping >slave.sqlEnter password: 關於mysqldump相關參數請前往:https://www.abcdocker.com/abcdocker/20
如今咱們已經將數據導入到主庫,如今去從庫看一下
master-mysql> show tables;+------------------------------+| Tables_in_shopping |+------------------------------+| aaaaaa_activity || aaaaaa_activity_detail || aaaaa_address || aaaaa_admin |這時候主庫和從庫看到的數據應該和咱們這裏面顯示的相同
經常使用命令
中止從庫: mysql> stop slave; 開啓從庫 mysql> start slave; 清除主從信息:(關閉從庫才能夠執行) mysql> reset slave;
數據庫過濾,數據同步的時候只複製某一個庫或者某一個表或某些表的操做
1.1 配置介紹
主 [Master]
binlog-do-db-d 表明白名單(僅將指定數據庫的相關修改操做記入二進制日誌) 缺點:若是主服務器宕了,恢復的時候只能夠恢復記錄到二進制文件日誌裏面的數據量。
binlog-ignore-db 表明黑名單(只要和這裏面設置相關的庫,都不記錄到二進制文件 )
不論是do-db仍是ignore-db咱們都不建議在主端操做,一旦主庫掛了,任何不被記錄的數據都將沒法還原。
咱們不開啓do和ignore這個功能,這時候咱們主庫的binlog是完整的。可是在從庫會有不少的二進制能夠進行過濾
從[Slave]
replicate-do-db 白名單(只應用哪一個數據庫到本地 )
replicate-ignore-db 黑名單(忽略哪些數據庫到本地)
過濾的功能還能夠在表級別實現
replicate-do-table= 應用那些表到本地
replicate-ignore-table= 忽略哪些表到本地在實現表同步的時候還可使用通配符 replicate-wild-do-table= (abcdocker.tb%)abcdocker下面的全部tb開頭的表 replicate-wild-ignore-table=
通配符%,_ 咱們能夠一個指令使用屢次,只要每次使用的值不一樣就能夠
1.2 例子
例子:例如咱們在mysql主從上,只複製abcdocker上的全部數據
從庫設置以下
vim /etc/my.confrelay-log = relay-logrelay-log-index =relay-log-indexreplicate-do-db = abcdocker [庫]
重啓mysqld
重啓從庫以後咱們須要開啓主從鏈接,在查看就能夠看到咱們這裏只同步abcdocker庫
<img src="http://static.zybuluo.com/abcdocker/dc2nrjy2yfndvse3vx7omo8j/image_1bs59gqgd1r461fq5jloltf176qm.png" alt="image_1bs59gqgd1r461fq5jloltf176qm.png-56.6kB"></p>
主庫設置以下
咱們在主庫建立2個databasescreate database abcdocker;create database discuz;由於咱們同步discuz庫,而不一樣步abc庫。
主庫有以下庫
master-mysql> show databases;+--------------------+| Database |+--------------------+| information_schema || abcdocker || discuz || mysql || performance_schema || shopping || test |+--------------------+7 rows in set (0.00 sec)
咱們在去從庫使用show databases;查看庫是否有discuz和abcdocker庫就能夠
此時只有abcdocker庫一個,由於咱們受權的只有一個
mysql> show databases;+--------------------+| Database |+--------------------+| information_schema || abcdocker || mysql || performance_schema || shopping || test |+--------------------+6 rows in set (0.00 sec)