ProxySQL Query Rewrite 使用示例

時間 2020-01-05

標籤 proxysql query rewrite 使用示例简体版

原文原文鏈接

在這篇文章中，我將從新探究ProxySQL中的Query Rewrite功能，由於query rewriting是建立ProxySQL的最根本初衷。 html

爲何咱們須要重寫查詢？mysql

你已經肯定了一個觸發性能瓶頸或致使系統緩慢的查詢
你沒法（快速的）修改應用代碼
某些特殊的操做須要「重定向查詢」

這兒舉例你做爲DBA發現了一個「壞查詢」，你確認是它致使了服務緩慢，而且可能會致使服務不可用。那這個查詢必須被優化，你和開發溝通要修正這個SQL，可是開發反饋回來的信息是能改，可是因爲技術的非技術的種種緣由吧，沒有那麼快。這時你怎麼辦，等着？顯然不能，你能夠在開發完成修正以前經過ProxySQL的Query Rewrite功能重寫某些查詢來完成優化同時對應用保持透明。nginx

如何重寫查詢？經過ProxySQL有兩種方式來完成（譯者注：其實應該理解爲兩種匹配查詢的方式）。sql

Query rewrite其實就是經過 mysql_query_rules 表中一個 match_pattern + replace_pattern 的過程，而match_digest （注意區分 match_pattern 和 match_digest ）僅用來匹配一個查詢，而非重寫它。邏輯上講，match_digest 和 username，schemaname，proxy_addr 等字段的做用是同樣的，僅用來匹配查詢。數據庫

這兩種不一樣的機制爲不一樣的查詢類型（例如DML操做，SELECT等）提供了靈活高效匹配方式。注意若是你但願重寫查詢，那麼規則中的match_pattern必須能匹配到原始的查詢。查詢規則按照rule_id字段的升序順序處理，而且只有在active字段爲1的前提下才會處理。app

下面是咱們如何在咱們的測試環境演示 match_digest ide

mysql> SELECT hostgroup hg, sum_time, count_star, digest_text FROM stats_mysql_query_digest ORDER BY sum_time DESC limit 10;
+----+-----------+------------+-----------------------------------+
| hg | sum_time  | count_star | digest_text                       |
+----+-----------+------------+-----------------------------------+
| 0  | 243549572 | 85710      | SELECT c FROM sbtest10 WHERE id=? |
| 0  | 146324255 | 42856      | COMMIT                            |
| 0  | 126643488 | 44310      | SELECT c FROM sbtest7 WHERE id=?  |
| 0  | 126517140 | 42927      | BEGIN                             |
| 0  | 123797307 | 43820      | SELECT c FROM sbtest1 WHERE id=?  |
| 0  | 123345775 | 43460      | SELECT c FROM sbtest6 WHERE id=?  |
| 0  | 122121030 | 43010      | SELECT c FROM sbtest9 WHERE id=?  |
| 0  | 121245265 | 42400      | SELECT c FROM sbtest8 WHERE id=?  |
| 0  | 120554811 | 42520      | SELECT c FROM sbtest3 WHERE id=?  |
| 0  | 119244143 | 42070      | SELECT c FROM sbtest5 WHERE id=?  |
+----+-----------+------------+-----------------------------------+
10 rows in set (0.00 sec)
mysql> INSERT INTO mysql_query_rules (rule_id,active,username,match_digest, match_pattern,replace_pattern,apply) VALUES (10,1,'root','SELECT.*WHERE id=?','sbtest2','sbtest10',1);
Query OK, 1 row affected (0.00 sec)
mysql> LOAD MYSQL QUERY RULES TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)
mysql> SELECT hits, mysql_query_rules.rule_id,digest,active,username, match_digest, match_pattern, replace_pattern, cache_ttl, apply FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules ORDER BY mysql_query_rules.rule_id;
+------+---------+--------+--------+----------+--------------------+---------------+-----------------+-----------+-------+
| hits | rule_id | digest | active | username | match_digest       | match_pattern | replace_pattern | cache_ttl | apply |
+------+---------+--------+--------+----------+--------------------+---------------+-----------------+-----------+-------+
| 0    | 10      | NULL   | 1      | root     | SELECT.*WHERE id=? | sbtest2       | sbtest10        | NULL      | 1     |
+------+---------+--------+--------+----------+--------------------+---------------+-----------------+-----------+-------+
1 row in set (0.00 sec)
mysql> SELECT hits, mysql_query_rules.rule_id,digest,active,username, match_digest, match_pattern, replace_pattern, cache_ttl, apply FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules ORDER BY mysql_query_rules.rule_id;
+------+---------+--------+--------+----------+--------------------+---------------+-----------------+-----------+-------+
| hits | rule_id | digest | active | username | match_digest       | match_pattern | replace_pattern | cache_ttl | apply |
+------+---------+--------+--------+----------+--------------------+---------------+-----------------+-----------+-------+
| 593  | 10      | NULL   | 1      | root     | SELECT.*WHERE id=? | sbtest2       | sbtest10        | NULL      | 1     |
+------+---------+--------+--------+----------+--------------------+---------------+-----------------+-----------+-------+
1 row in set (0.00 sec)

若是想清空 query rules 的統計信息，使用下列方法性能

mysql> SELECT 1 FROM stats_mysql_query_digest_reset LIMIT 1;
+---+
| 1 |
+---+
| 1 |
+---+
1 row in set (0.01 sec)
mysql> LOAD MYSQL QUERY RULES TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)

接下來是 match_pattern 示例：測試

mysql> SELECT hostgroup hg, sum_time, count_star, digest_text FROM stats_mysql_query_digest ORDER BY sum_time DESC limit 5;
+----+----------+------------+----------------------------------+
| hg | sum_time | count_star | digest_text                      |
+----+----------+------------+----------------------------------+
| 0  | 98753983 | 16292      | BEGIN                            |
| 0  | 84613532 | 16232      | COMMIT                           |
| 1  | 49327292 | 16556      | SELECT c FROM sbtest3 WHERE id=? |
| 1  | 49027118 | 16706      | SELECT c FROM sbtest2 WHERE id=? |
| 1  | 48095847 | 16396      | SELECT c FROM sbtest4 WHERE id=? |
+----+----------+------------+----------------------------------+
5 rows in set (0.01 sec)

mysql> INSERT INTO mysql_query_rules (rule_id,active,username,match_pattern,replace_pattern,apply) VALUES (20,1,'root','DISTINCT(.*)ORDER BY c','DISTINCT1',1);
Query OK, 1 row affected (0.00 sec)

mysql> LOAD MYSQL QUERY RULES TO RUNTIME;
Query OK, 0 rows affected (0.01 sec)

mysql> SELECT hits, mysql_query_rules.rule_id,digest,active,username, match_digest, match_pattern, replace_pattern, cache_ttl, apply FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules ORDER BY mysql_query_rules.rule_id;
+------+---------+--------+--------+----------+--------------------+------------------------+-----------------+-----------+-------+
| hits | rule_id | digest | active | username | match_digest       | match_pattern          | replace_pattern | cache_ttl | apply |
+------+---------+--------+--------+----------+--------------------+------------------------+-----------------+-----------+-------+
| 0    | 10      | NULL   | 1      | root     | SELECT.*WHERE id=? | sbtest2                | sbtest10        | NULL      | 1     |
| 0    | 20      | NULL   | 1      | root     | NULL               | DISTINCT(.*)ORDER BY c | DISTINCT1      | NULL      | 1     |
+------+---------+--------+--------+----------+--------------------+------------------------+-----------------+-----------+-------+
2 rows in set (0.01 sec)

mysql> SELECT hits, mysql_query_rules.rule_id,digest,active,username, match_digest, match_pattern, replace_pattern, cache_ttl, apply FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules ORDER BY mysql_query_rules.rule_id;
+------+---------+--------+--------+----------+--------------------+------------------------+-----------------+-----------+-------+
| hits | rule_id | digest | active | username | match_digest       | match_pattern          | replace_pattern | cache_ttl | apply |
+------+---------+--------+--------+----------+--------------------+------------------------+-----------------+-----------+-------+
| 9994 | 10      | NULL   | 1      | root     | SELECT.*WHERE id=? | sbtest2                | sbtest10        | NULL      | 1     |
| 6487 | 20      | NULL   | 1      | root     | NULL               | DISTINCT(.*)ORDER BY c | DISTINCT1      | NULL      | 1     |
+------+---------+--------+--------+----------+--------------------+------------------------+-----------------+-----------+-------+
2 rows in set (0.00 sec)

mysql> SELECT 1 FROM stats_mysql_query_digest_reset LIMIT 1;
+---+
| 1 |
+---+
| 1 |
+---+
1 row in set (0.00 sec)

mysql>  LOAD MYSQL QUERY RULES TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)

路由規則中一個關鍵點是 mysql_query_rules 的 apply 字段優化

apply = 1（默認）表示查詢一旦匹配到一條規則就再也不匹配剩餘的規則
apply = 0 表示繼續嘗試匹配後續的規則

（譯者注：相似於nginx rewrite 指令中的 break 參數）

以下面測試中所展現的，全部匹配rule_id = 10 或 rule_id = 20 的查詢都準確的匹配上了。實際上，如今全部的規則在 runtime_mysql_query_rules 表中都是激活的。若是咱們想禁用 mysql_query_rules 表中某條規則，設置 active = 0

mysql> update mysql_query_rules set apply = 1 where rule_id in (10);
Query OK, 1 row affected (0.00 sec)
mysql> update mysql_query_rules set apply = 0 where rule_id in (20);
Query OK, 1 row affected (0.00 sec)
mysql>  LOAD MYSQL QUERY RULES TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)
mysql> SELECT hits, mysql_query_rules.rule_id,digest,active,username, match_digest, match_pattern, replace_pattern, cache_ttl, apply FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules ORDER BY mysql_query_rules.rule_id;
+------+---------+--------+--------+----------+--------------------+------------------------+-----------------+-----------+-------+
| hits | rule_id | digest | active | username | match_digest       | match_pattern          | replace_pattern | cache_ttl | apply |
+------+---------+--------+--------+----------+--------------------+------------------------+-----------------+-----------+-------+
| 0    | 10      | NULL   | 1      | root     | SELECT.*WHERE id=? | sbtest2                | sbtest10        | NULL      | 1     |
| 0    | 20      | NULL   | 1      | root     | NULL               | DISTINCT(.*)ORDER BY c | DISTINCT1      | NULL      | 0     |
+------+---------+--------+--------+----------+--------------------+------------------------+-----------------+-----------+-------+
2 rows in set (0.00 sec)
mysql> SELECT hits, mysql_query_rules.rule_id,digest,active,username, match_digest, match_pattern, replace_pattern, flagIN, apply FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules ORDER BY mysql_query_rules.rule_id;
+-------+---------+--------+--------+----------+--------------------+------------------------+-----------------+--------+-------+
| hits  | rule_id | digest | active | username | match_digest       | match_pattern          | replace_pattern | flagIN | apply |
+-------+---------+--------+--------+----------+--------------------+------------------------+-----------------+--------+-------+
| 10195 | 10      | NULL   | 1      | root     | SELECT.*WHERE id=? | sbtest2                | sbtest10        | 0      | 1     |
| 6599  | 20      | NULL   | 1      | root     | NULL               | DISTINCT(.*)ORDER BY c | DISTINCT1      | 0      | 0     |
+-------+---------+--------+--------+----------+--------------------+------------------------+-----------------+--------+-------+
2 rows in set (0.00 sec)
mysql> SELECT hits, mysql_query_rules.rule_id,digest,active,username, match_digest, match_pattern, replace_pattern, flagIN, apply FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules ORDER BY mysql_query_rules.rule_id;
+-------+---------+--------+--------+----------+--------------------+------------------------+-----------------+--------+-------+
| hits  | rule_id | digest | active | username | match_digest       | match_pattern          | replace_pattern | flagIN | apply |
+-------+---------+--------+--------+----------+--------------------+------------------------+-----------------+--------+-------+
| 20217 | 5       | NULL   | 1      | root     | NULL               | DISTINCT(.*)ORDER BY c | DISTINCT1      | 0      | 1     |
| 27020 | 10      | NULL   | 1      | root     | SELECT.*WHERE id=? | sbtest2                | sbtest10        | 0      | 0     |
+-------+---------+--------+--------+----------+--------------------+------------------------+-----------------+--------+-------+
2 rows in set (0.00 sec)
mysql> update mysql_query_rules set active = 0 where rule_id = 5;
Query OK, 1 row affected (0.00 sec)
mysql>  LOAD MYSQL QUERY RULES TO RUNTIME;
Query OK, 0 rows affected (0.02 sec)
mysql> SELECT hits, mysql_query_rules.rule_id,digest,active,username, match_digest, match_pattern, replace_pattern, cache_ttl, apply FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules ORDER BY mysql_query_rules.rule_id;
+------+---------+--------+--------+----------+--------------------+---------------+-----------------+-----------+-------+
| hits | rule_id | digest | active | username | match_digest       | match_pattern | replace_pattern | cache_ttl | apply |
+------+---------+--------+--------+----------+--------------------+---------------+-----------------+-----------+-------+
| 0    | 10      | NULL   | 1      | root     | SELECT.*WHERE id=? | sbtest2       | sbtest10        | NULL      | 0     |
+------+---------+--------+--------+----------+--------------------+---------------+-----------------+-----------+-------+
1 row in set (0.00 sec)
mysql> SELECT hits, mysql_query_rules.rule_id,digest,active,username, match_digest, match_pattern, replace_pattern, cache_ttl, apply FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules ORDER BY mysql_query_rules.rule_id;
+------+---------+--------+--------+----------+--------------------+---------------+-----------------+-----------+-------+
| hits | rule_id | digest | active | username | match_digest       | match_pattern | replace_pattern | cache_ttl | apply |
+------+---------+--------+--------+----------+--------------------+---------------+-----------------+-----------+-------+
| 4224 | 10      | NULL   | 1      | root     | SELECT.*WHERE id=? | sbtest2       | sbtest10        | NULL      | 0     |
+------+---------+--------+--------+----------+--------------------+---------------+-----------------+-----------+-------+
1 row in set (0.01 sec)

另外，ProxySQL還能幫忙識別出「低效的查詢」，登陸管理界面按以下操做
找出總耗時最多的查詢

mysql> SELECT SUM(sum_time), SUM(count_star), digest_text FROM stats_mysql_query_digest GROUP BY digest ORDER BY SUM(sum_time) DESC LIMIT 3G
*************************** 1. row ***************************
  SUM(sum_time): 95053795
SUM(count_star): 13164
    digest_text: BEGIN
*************************** 2. row ***************************
  SUM(sum_time): 85094367
SUM(count_star): 13130
    digest_text: COMMIT
*************************** 3. row ***************************
  SUM(sum_time): 52110099
SUM(count_star): 13806
    digest_text: SELECT c FROM sbtest3 WHERE id=?
3 rows in set (0.00 sec)

找出平均耗時最高的查詢

mysql> SELECT SUM(sum_time), SUM(count_star), SUM(sum_time)/SUM(count_star) avg,  digest_text FROM stats_mysql_query_digest GROUP BY digest ORDER BY SUM(sum_time)/SUM(count_star) DESC limit 1;
+---------------+-----------------+--------+--------------------------------+
| SUM(sum_time) | SUM(count_star) | avg    | digest_text                    |
+---------------+-----------------+--------+--------------------------------+
| 972162        | 1               | 972162 | CREATE INDEX k_5 ON sbtest5(k) |
+---------------+-----------------+--------+--------------------------------+
1 row in set (0.00 sec)

我發現關於ProxySQL query rewrite 的「最好」的文檔在IBM，這裏介紹了查詢重寫的原理和示例，值得一讀。

還有一些別的場景你可能須要重寫查詢，試想有一張表的自增ID列已經達到了int類型的最大值，你能夠將新插入的數據重定向到另外一張表同時你經過alter命令來修正原表的問題，在這期間全部的查詢還將訪問原表，等alter原表完成後，將新表的數據導入的原表，便可達到不停機修DDL的效果。

從MySQL 5.7.6 起，MySQL以插件形式提供了 query rewrite 功能，你能夠在這裏找到相關文檔。MySQL內建的查詢重寫功能的一個最大的劣勢在於重寫規則僅做用於當前MySQL實例，這也是相比之下ProxySQL 的優點所在：它處在應用和數據庫之間，因此它的重寫規則是全局的。

原文連接