mysql主從之多線程複製

時間 2019-11-24

標籤 mysql 主從之多線程複製欄目 MySQL 简体版

原文原文鏈接

多線程複製mysql

mysql 主從複製原理：sql

1. master 節點上的binlogdump 線程，在slave 與其正常鏈接的狀況下，將binlog 發送到slave 上。數據庫

2. slave 節點的I/O Thread ，經過讀取master 節點binlog 日誌名稱以及偏移量信息將其拷貝到本地relay log 日誌文件。服務器

3. slave 節點的SQL Thread，該線程讀取relay log 日誌信息，將在master 節點上提交的事務在本地回放，達到與主庫數據保持一致的目的。多線程

MySQL5.5 及之前的複製併發

通常主從複製有三個線程且都是單線程：socket

Binlog Dump（主） ‐‐> IO Thread（從） ‐‐> SQL Thread（從）。性能

而master 這邊是經過併發線程提交，事務經過LSN 寫入binlog；可是Slave 只有一個IO 線程和SQL 線程，是單線程，因此在業務大的狀況下就很容易形成主從延時.優化

若是在MySQL 5.6 版本開啓並行複製功能（slave_parallel_workers > 0），那麼SQL 線程就變爲了coordinator 線程，coordinator 線程主要負責如下兩部份內容：ui

Coordinator+worker（多個）

若判斷能夠並行執行，那麼選擇worker 線程執行事務的二進制日誌。

若判斷不能夠並行執行，如該操做是DDL，亦或者是事務跨schema 操做，則等待全部的worker 線程執行完成以後，再執行當前的日誌。

這意味着coordinator 線程並非僅將日誌發送給worker 線程，本身也能夠回放日誌，可是全部能夠並行的操做交付由worker 線程完成。

上述機制實現的基於schema 的並行複製，存在的問題是：

這樣設計的並行複製效果並不高，若是用戶實例僅有一個庫，那麼就沒法實現並行回放，甚至性能會比原來的單線程更差，而單庫多表是比多庫多表更爲常見的一種情形。

MySQL5.7 的MTS（Enhanced Muti‐threadedslaves）MySQL 5.7 引入了新的機制來實現並行複製，再也不有基於庫的並行複製限制，主要思想就是slave 服務器的回放與主機是一致的，即master 服務器上是怎麼並行執行，slave 上就怎樣進行並行回放。

mysql v5.7.2 進行了優化，增長了參數slave_parallel_type，參數有兩個選項：

LOGICAL_CLOCK：基於邏輯時鐘，能夠在一個DATABASE 中併發執行relay log 事務DATABASE：基於數據庫，v5.6 默認是這個參數，改參數每一個庫只能一個線程；

slave‐parallel‐type，其能夠配置的值有：

DATABASE：默認值，基於庫的並行複製方式

LOGICAL_CLOCK：基於組提交的並行複製方式

從庫:

mysql> show variables like 'slave_parallel%';
+------------------------+----------+
| Variable_name          | Value    |
+------------------------+----------+
| slave_parallel_type    | DATABASE |
| slave_parallel_workers | 0        |
+------------------------+----------+
mysql> stop slave;
mysql> set global slave_parallel_type='LOGICAL_CLOCK';
Query OK, 0 rows affected (0.00 sec)
mysql> set global slave_parallel_type='LOGICAL_CLOCK';
Query OK, 0 rows affected (0.00 sec)
mysql> set global slave_parallel_workers=2;
Query OK, 0 rows affected (0.00 sec)
mysql> start slave;
Query OK, 0 rows affected (0.01 sec)
mysql> show processlist;
+-------+-------------+-----------+------+---------+------+--------------------------------------------------------+------------------+
| Id    | User        | Host      | db   | Command | Time | State                                                  | Info             |
+-------+-------------+-----------+------+---------+------+--------------------------------------------------------+------------------+
| 26077 | root        | localhost | NULL | Query   |    0 | starting                                               | show processlist |
| 26141 | system user |           | NULL | Connect |   40 | Waiting for master to send event                       | NULL             |
| 26142 | system user |           | NULL | Connect |   40 | Slave has read all relay log; waiting for more updates | NULL             |
| 26143 | system user |           | NULL | Connect |   40 | Waiting for an event from Coordinator                  | NULL             |
| 26144 | system user |           | NULL | Connect |   40 | Waiting for an event from Coordinator                  | NULL             |
+-------+-------------+-----------+------+---------+------+--------------------------------------------------------+------------------+
mysql> show variables like 'slave_parallel%';
+------------------------+---------------+
| Variable_name          | Value         |
+------------------------+---------------+
| slave_parallel_type    | LOGICAL_CLOCK |
| slave_parallel_workers | 2             |
+------------------------+---------------+

master_info_repository=TABLE （開啓MTS 功能後，會頻繁更新master.info，設置爲TABLE 減少開銷）relay_log_info_repository=TABLE

slave_master_info 記錄了首次同步master 的位置relay_log_recovery=ON （slave IO 線程crash，若是relay‐log損壞，則自動放棄全部未執行的relay‐log，從新從master 上獲取日誌，保證relay‐log 的完整性）

slave_preserve_commit_order=ON （保證提交的順序性）在slave 上應用事務的順序是無序的，和relay log 中記錄的事務順序不同，這樣數據一致性是沒法保證的，爲了保證事務是按照relay log 中記錄的順序來回放，就須要開啓參數slave_preserve_commit_order。

雖然mysql5.7 添加MTS 後，雖然slave 能夠並行應用relay log，但commit 部分仍然是順序提交，其中可能會有等待的狀況。

優化參數

[mysqld]
bind-address=0.0.0.0
port=3306
datadir=/data/mysql
socket=/data/mysql/mysql.sock
user=mysql
skip-name-resolve
slow_query_log=on
long_query_time=1
slow_query_log_file=/data/mysql/mysql-slow.log
innodb-file-per-table=1
innodb_flush_log_at_trx_commit = 2
log_warnings = 1
connect_timeout = 60
net_read_timeout = 120
performance_schema_max_table_instances = 400
server-id = 2
relay-log = relay-log
relay-log-index = relay-log.index
slave_parallel_type=LOGICAL_CLOCK
slave_parallel_workers=2
master_info_repository=TABLE
relay_log_info_repository=TABLE
relay_log_recovery=ON
slave_preserve_commit_order=ON
log_bin=mysql‐bin
log_slave_updates=ON

[mysqld_safe]
log-error=/data/mysql/mysqld.log
pid-file=/data/mysql/mysqld.pid

重啓mysqld進入數據庫

mysql> show variables like '%slave_parallel_type%';
+---------------------+---------------+
| Variable_name       | Value         |
+---------------------+---------------+
| slave_parallel_type | LOGICAL_CLOCK |
+---------------------+---------------+
mysql> show variables like '%slave_parallel_worker%';
+------------------------+-------+
| Variable_name          | Value |
+------------------------+-------+
| slave_parallel_workers | 2     |
+------------------------+-------+
mysql> show variables like '%master_info_repository%';
+------------------------+-------+
| Variable_name          | Value |
+------------------------+-------+
| master_info_repository | TABLE |
+------------------------+-------+
mysql> set global relay_log_info_repository='TABLE';
Query OK, 0 rows affected (0.00 sec)

mysql> show variables like '%relay_log_info_repository%';
+---------------------------+-------+
| Variable_name             | Value |
+---------------------------+-------+
| relay_log_info_repository | TABLE |
+---------------------------+-------+
mysql> show variables like '%relay_log_recovery%';
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| relay_log_recovery | ON    |
+--------------------+-------+
1 row in set (0.00 sec)

mysql> show variables like '%slave_preserve_commit_order%';
+-----------------------------+-------+
| Variable_name               | Value |
+-----------------------------+-------+
| slave_preserve_commit_order | ON    |
+-----------------------------+-------+

開啓slave

mysql> start slave;
Query OK, 0 rows affected (0.00 sec)

mysql> show slave  status\G;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.132.121
                  Master_User: replication
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: master-bin.000002
          Read_Master_Log_Pos: 40870
               Relay_Log_File: relay-log.000010
                Relay_Log_Pos: 321
        Relay_Master_Log_File: master-bin.000002
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 40870
              Relay_Log_Space: 689
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 1
                  Master_UUID: 6b00724f-a094-11e9-8f47-000c2991dd19
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version:

主庫寫入數據

[root@master mysql]#mysql -uroot -p123456  -e 'use darren;insert into test values (1);
[root@master mysql]#mysql -uroot -p123456  -e 'use darren;insert into test values (1);
[root@master mysql]#mysql -uroot -p123456
mysql> mysql> show master status;
+-------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+-------------------+----------+--------------+------------------+-------------------+
| master-bin.000002 | 41388 | | | |
+-------------------+----------+--------------+------------------+-------------------+

從庫

mysql> use mysql;
Database changed
mysql> show tables;
+---------------------------+
| Tables_in_mysql           |
+---------------------------+
| columns_priv              |
| db                        |
| engine_cost               |
| event                     |
| func                      |
| general_log               |
| gtid_executed             |
| help_category             |
| help_keyword              |
| help_relation             |
| help_topic                |
| innodb_index_stats        |
| innodb_table_stats        |
| ndb_binlog_index          |
| plugin                    |
| proc                      |
| procs_priv                |
| proxies_priv              |
| server_cost               |
| servers                   |
| slave_master_info         |
| slave_relay_log_info      |
| slave_worker_info         |
| slow_log                  |
| tables_priv               |
| time_zone                 |
| time_zone_leap_second     |
| time_zone_name            |
| time_zone_transition      |
| time_zone_transition_type |
| user                      |
+---------------------------+
mysql> select * from slave_master_info;
+-----------------+-------------------+----------------+-----------------+-------------+---------------+------+---------------+-------------+--------+------------+----------+------------+---------+------------------------+-----------+------+--------------------+--------------------------------------+-------------+---------+-------------+-----------------------+--------------+-------------+
| Number_of_lines | Master_log_name   | Master_log_pos | Host            | User_name   | User_password | Port | Connect_retry | Enabled_ssl | Ssl_ca | Ssl_capath | Ssl_cert | Ssl_cipher | Ssl_key | Ssl_verify_server_cert | Heartbeat | Bind | Ignored_server_ids | Uuid                                 | Retry_count | Ssl_crl | Ssl_crlpath | Enabled_auto_position | Channel_name | Tls_version |
+-----------------+-------------------+----------------+-----------------+-------------+---------------+------+---------------+-------------+--------+------------+----------+------------+---------+------------------------+-----------+------+--------------------+--------------------------------------+-------------+---------+-------------+-----------------------+--------------+-------------+
|              25 | master-bin.000002 |          40870 | 192.168.132.121 | replication | 1234567       | 3306 |            60 |           0 |        |            |          |            |         |                      0 |        30 |      | 0                  | 6b00724f-a094-11e9-8f47-000c2991dd19 |       86400 |         |             |                     0 |              |             |
+-----------------+-------------------+----------------+-----------------+-------------+---------------+------+---------------+-------------+--------+------------+----------+------------+---------+------------------------+-----------+------+--------------------+--------------------------------------+-------------+---------+-------------+-----------------------+--------------+-------------+
mysql> select * from slave_relay_log_info;
+-----------------+--------------------+---------------+-------------------+----------------+-----------+-------------------+----+--------------+
| Number_of_lines | Relay_log_name     | Relay_log_pos | Master_log_name   | Master_log_pos | Sql_delay | Number_of_workers | Id | Channel_name |
+-----------------+--------------------+---------------+-------------------+----------------+-----------+-------------------+----+--------------+
|               7 | ./relay-log.000010 |           839 | master-bin.000002 |          41388 |         0 |                 2 |  1 |              |
+-----------------+--------------------+---------------+-------------------+----------------+-----------+-------------------+----+--------------+

查看線程

+----+-------------+-----------+-------+---------+------+--------------------------------------------------------+-------------------+
| Id | User        | Host      | db    | Command | Time | State                                                  | Info              |
+----+-------------+-----------+-------+---------+------+--------------------------------------------------------+-------------------+
|  7 | root        | localhost | mysql | Query   |    0 | starting                                               | show  processlist |
| 36 | system user |           | NULL  | Connect |  762 | Waiting for master to send event                       | NULL              |
| 37 | system user |           | NULL  | Connect |  567 | Slave has read all relay log; waiting for more updates | NULL              |
| 38 | system user |           | NULL  | Connect |  567 | Waiting for an event from Coordinator                  | NULL              |
| 39 | system user |           | NULL  | Connect |  762 | Waiting for an event from Coordinator                  | NULL              |
+----+-------------+-----------+-------+---------+------+--------------------------------------------------------+-------------------+