MySQL線程處於Waiting for table flush的分析

 

最近遇到一個案例，不少查詢被阻塞沒有返回結果，使用show processlist查看，發現很多MySQL線程處於Waiting for table flush狀態，查詢語句一直被阻塞，只能經過Kill進程來解決。那麼咱們先來看看Waiting for table flush的官方解釋：https://dev.mysql.com/doc/refman/5.6/en/general-thread-states.html

 

Waiting for table flush

 

The thread is executing FLUSH TABLES and is waiting for all threads to close their tables, or the thread got a notification that the underlying structure for a table has changed and it needs to reopen the table to get the new structure. However, to reopen the table, it must wait until all other threads have closed the table in question.

This notification takes place if another thread has used FLUSH TABLES or one of the following statements on the table in question: FLUSH TABLES tbl_name, ALTER TABLE, RENAME TABLE, REPAIR TABLE, ANALYZE TABLE, or OPTIMIZE TABLE.

 

 

那麼咱們接下來模擬一下線程處於Waiting for table flush狀態的狀況，如所示：

 

在第一個會話鏈接（connection id=13）中，咱們使用lock table 鎖定表test。 

 

mysql> use MyDB;

Database changed

mysql> select connection_id();

+-----------------+

| connection_id() |

+-----------------+

|              13 |

+-----------------+

1 row in set (0.00 sec)

mysql> lock table test read;

Query OK, 0 rows affected (0.00 sec)

mysql> 

 

 

 

在第二個會話鏈接（connection id=17）中，咱們執行flush table 或 flush table test 皆可。此時你會發現flush table處於阻塞狀態。

 

mysql> use MyDB;

Reading table information for completion of table and column names

You can turn off this feature to get a quicker startup with -A

Database changed

mysql> select connection_id();

+-----------------+

| connection_id() |

+-----------------+

|              17 |

+-----------------+

1 row in set (0.00 sec)

mysql> flush table test;

 

 

 

 

在第三個會話/鏈接中，當你切換到MyDB時，就會提示「You can turn off this feature to get a quicker startup with -A」 ，此時處於阻塞狀態。此時你退出會話，使用參數-A登陸數據庫後，你若是查詢test表，就會處於阻塞狀態（固然查詢其它表不會被阻塞）。以下所示：

 

mysql> use MyDB;

Reading table information for completion of table and column names

You can turn off this feature to get a quicker startup with -A

 

 

mysql> use MyDB;

Database changed

mysql> select * from test;

 

 

 

在第四個會話/鏈接，咱們用show processlist查看到當前數據庫全部鏈接線程狀態，你會看到1七、18都處於Waiting for table flush的狀態。以下截圖所示：

 

mysql> show processlist;

+----+------+-----------+------+---------+------+-------------------------+--------------------+

| Id | User | Host      | db   | Command | Time | State                   | Info               |

+----+------+-----------+------+---------+------+-------------------------+--------------------+

| 13 | root | localhost | MyDB | Sleep   |   90 |                         | NULL               |

| 14 | root | localhost | NULL | Query   |    0 | init                    | show processlist   |

| 17 | root | localhost | MyDB | Query   |   52 | Waiting for table flush | flush table test   |

| 18 | root | localhost | MyDB | Query   |    9 | Waiting for table flush | select * from test |

+----+------+-----------+------+---------+------+-------------------------+--------------------+

4 rows in set (0.00 sec)

mysql> 

 

 

mysql> show processlist;

+----+------+-----------+------+---------+------+-------------------------+--------------------+

| Id | User | Host      | db   | Command | Time | State                   | Info               |

+----+------+-----------+------+---------+------+-------------------------+--------------------+

| 13 | root | localhost | MyDB | Sleep   |   90 |                         | NULL               |

| 14 | root | localhost | NULL | Query   |    0 | init                    | show processlist   |

| 17 | root | localhost | MyDB | Query   |   52 | Waiting for table flush | flush table test   |

| 18 | root | localhost | MyDB | Query   |    9 | Waiting for table flush | select * from test |

+----+------+-----------+------+---------+------+-------------------------+--------------------+

4 rows in set (0.00 sec)

mysql> 

mysql> 

mysql> 

mysql> 

mysql> show open tables where in_use >=1;

+----------+-------+--------+-------------+

| Database | Table | In_use | Name_locked |

+----------+-------+--------+-------------+

| MyDB     | test  |      1 |           0 |

+----------+-------+--------+-------------+

1 row in set (0.00 sec)

mysql> kill 17;

Query OK, 0 rows affected (0.00 sec)

mysql> show processlist;

+----+------+-----------+------+---------+------+-------------------------+--------------------+

| Id | User | Host      | db   | Command | Time | State                   | Info               |

+----+------+-----------+------+---------+------+-------------------------+--------------------+

| 13 | root | localhost | MyDB | Sleep   |  442 |                         | NULL               |

| 14 | root | localhost | NULL | Query   |    0 | init                    | show processlist   |

| 18 | root | localhost | MyDB | Query   |  361 | Waiting for table flush | select * from test |

+----+------+-----------+------+---------+------+-------------------------+--------------------+

3 rows in set (0.00 sec)

mysql> kill 13;

Query OK, 0 rows affected (0.00 sec)

mysql> show processlist;

+----+------+-----------+------+---------+------+-------+------------------+

| Id | User | Host      | db   | Command | Time | State | Info             |

+----+------+-----------+------+---------+------+-------+------------------+

| 14 | root | localhost | NULL | Query   |    0 | init  | show processlist |

| 18 | root | localhost | MyDB | Sleep   |  427 |       | NULL             |

+----+------+-----------+------+---------+------+-------+------------------+

2 rows in set (0.00 sec)

mysql> 

 

|

 

注意：咱們須要Kill線程13， Kill掉線程17是解決不了問題的。

 

 

 

生產環境中，不少時候可能不是lock table read引發的阻塞，而是因爲慢查詢，致使flush table一直沒法關閉該表而一直處於等待狀態，例以下面測試案例中，我使用同一張大表作笛卡爾積模擬一個慢查詢，其它操做相同，以下所示，你會看到一樣產生了Waiting for table flush

 

mysql> SELECT T.* FROM TEST1 T, TEST1 L;

 

 

 

另外，網上有個案例，mysqldump備份時，若是沒有使用參數—single-transaction 或因爲同時使用了flush-logs與—single-transaction兩個參數也可能引發這樣的等待場景，這個兩個參數放在一塊兒，會在開始dump數據以前先執行一個FLUSH TABLES操做。

 

 

 

解決方案：

 

 

出現Waiting for table flush時，咱們通常須要找到那些表被lock住或那些慢查詢致使flush table一直在等待而沒法關閉該表。而後Kill掉對應的線程便可，可是如何精準定位是一個挑戰，尤爲是生產環境，你使用show processlist會看到大量的線程。讓你眼花繚亂的，怎麼一會兒定位問題呢？

 

對於慢查詢引發的其它線程處於Waiting for table flush狀態的情形：

 

能夠查看show processlist中Time值很大的線程。而後甄別確認後Kill掉，如上截圖所示，會話鏈接14就是引發阻塞的源頭SQL。有種規律就是這個線程的Time列值一定比被阻塞的線程要高。這個就能過濾不少記錄。

 

對於lock table read引發的其它線程處於Waiting for table flush狀態的情形：

 

對於實驗中使用lock table read這種狀況，這種會話可能處於Sleep狀態，並且它也不會出如今show engine innodb status \G命令的輸出信息中。 即便show open tables where in_use >=1;能找到是那張表被lock住了，可是沒法定位到具體的線程（鏈接），其實這個是一個頭痛的問題。可是inntop這款利器就能夠定位到，以下所示，線程17鎖住了表test，在innotop裏面就能定位到是線程17。所謂工欲善其事必先利其器！

 

 

 

 

 

另外，在官方文檔中ALTER TABLE, RENAME TABLE, REPAIR TABLE, ANALYZE TABLE, or OPTIMIZE TABLE都能引發這類等待，下面也作了一些簡單測試，以下所示：

 

 

 

Waiting for table flush的另一個場景

 

會話鏈接（connection id=18)執行下面SQL語句，模擬一個慢查詢SQL

 

mysql> select connection_id();

+-----------------+

| connection_id() |

+-----------------+

|              18 |

+-----------------+

1 row in set (0.00 sec)

mysql> select name, sleep(64) from test;

 

會話鏈接（connection id=6)執行下面SQL語句，分析表test

 

mysql> select connection_id();

+-----------------+

| connection_id() |

+-----------------+

|               6 |

+-----------------+

1 row in set (0.00 sec)

mysql> analyze table test;

+-----------+---------+----------+----------+

| Table     | Op      | Msg_type | Msg_text |

+-----------+---------+----------+----------+

| MyDB.test | analyze | status   | OK       |

+-----------+---------+----------+----------+

1 row in set (0.04 sec)

mysql> 

 

會話鏈接（connection id=8)執行下面SQL語句

 

mysql> select connection_id();

+-----------------+

| connection_id() |

+-----------------+

|               8 |

+-----------------+

1 row in set (0.00 sec)

mysql> select * from test;

 

查看線程的狀態，你會發現被阻塞的會話處於 Waiting for table flush狀態。 由於當對錶作了ANALYZE TABLE後，後臺針對該表的查詢須要等待，由於MySQL已經檢測到該表內部變化，須要使用FLUSH TABLE關閉而後從新打開該表，因此當你查詢該表時，就會處於 Waiting for table flush

 

mysql> show processlist;

+----+------+-----------+------+---------+------+-------------------------+----------------------------------+

| Id | User | Host      | db   | Command | Time | State                   | Info                             |

+----+------+-----------+------+---------+------+-------------------------+----------------------------------+

|  6 | root | localhost | MyDB | Sleep   |   22 |                         | NULL                             |

|  8 | root | localhost | MyDB | Query   |   14 | Waiting for table flush | select * from test               |

| 15 | root | localhost | NULL | Sleep   |    3 |                         | NULL                             |

| 16 | root | localhost | NULL | Query   |    0 | init                    | show processlist                 |

| 18 | root | localhost | MyDB | Query   |   46 | User sleep              | select name, sleep(64) from test |

+----+------+-----------+------+---------+------+-------------------------+----------------------------------+

5 rows in set (0.00 sec)

mysql> 

 

 

 

 

Waiting for table metadata lock

 

 

會話鏈接（connection id=17)執行下面SQL語句，模擬一個慢查詢SQL

 

 

mysql> select connection_id();

+-----------------+

| connection_id() |

+-----------------+

|              17 |

+-----------------+

1 row in set (0.00 sec)

mysql> select name, sleep(100) from test;

 

 

會話鏈接（connection id=6）執行下面SQL語句， 修改表結構操做

 

mysql> select connection_id();

+-----------------+

| connection_id() |

+-----------------+

|               6 |

+-----------------+

1 row in set (0.00 sec)

mysql> alter table test add tname varchar(10); // rename table test to kkk 一樣會引發Waiting for table metadata lock

 

 

會話鏈接（connection id=8）執行下面SQL語句，查詢表test

 

mysql> select connection_id();

+-----------------+

| connection_id() |

+-----------------+

|               8 |

+-----------------+

1 row in set (0.00 sec)

mysql> select * from test;

 

 

查看線程的狀態，你會發現被阻塞的會話處於 Waiting for table metadata lock狀態。

 

 

mysql> show processlist;

+----+------+-----------+------+---------+------+---------------------------------+----------------------------------------+

| Id | User | Host      | db   | Command | Time | State                           | Info                                   |

+----+------+-----------+------+---------+------+---------------------------------+----------------------------------------+

|  6 | root | localhost | MyDB | Query   |   19 | Waiting for table metadata lock | alter table test add tname varchar(10) |

|  8 | root | localhost | MyDB | Query   |    6 | Waiting for table metadata lock | select * from test                     |

| 15 | root | localhost | NULL | Sleep   |    8 |                                 | NULL                                   |

| 16 | root | localhost | NULL | Query   |    0 | init                            | show processlist                       |

| 17 | root | localhost | MyDB | Query   |   55 | User sleep                      | select name, sleep(100) from test      |

+----+------+-----------+------+---------+------+---------------------------------+----------------------------------------+

5 rows in set (0.00 sec)

mysql> 

 

 

 

 

 

參考資料：

 

https://www.percona.com/blog/2013/02/27/mysql-optimizer-analyze-table-and-waiting-for-table-flush/

http://www.cnblogs.com/jackhub/p/3841004.html

http://myrock.github.io/2014/11/20/mysql-waiting-for-table-flush/

http://mysql.taobao.org/monthly/2016/03/10/