做者:高鵬
文章末尾有他著做的《深刻理解MySQL主從原理 32講》,深刻透徹理解MySQL主從,GTID相關技術知識。本案例由徐晨亮提供,而且一塊兒探討。html
本文中
FTWRL = 「flush table with read lock」
關於經常使用操做加MDL LOCK鎖類型參考文章:
http://blog.itpub.net/7728585...
測試版本 : MySQL 5.7.22mysql
首先創建一張有幾條數據的表就能夠了,我這裏是baguait1表了。sql
SESSION1 | SESSION2 | SESSION3 |
---|---|---|
步驟1:select sleep(1000) from baguait1 for update; | ||
步驟2:flush table with read lock;堵塞 | ||
步驟3:kill session2 | ||
步驟4:select * from baguait1 limit 1;成功 |
步驟2 「flush table with read lock;」操做等待狀態爲「Waiting for global read lock」,以下:數據庫
mysql> select Id,State,Info from information_schema.processlist where command<>'sleep'; +----+------------------------------+------------------------------------------------------------------------------------+ | Id | State | Info | +----+------------------------------+------------------------------------------------------------------------------------+ | 1 | Waiting on empty queue | NULL | | 18 | Waiting for global read lock | flush table with read lock | | 3 | User sleep | select sleep(1000) from baguait1 for update | | 6 | executing | select Id,State,Info from information_schema.processlist where command<>'sleep' | +----+------------------------------+------------------------------------------------------------------------------------+
這裏比較奇怪了,實際上我好久之前就遇到過和測試過可是沒有仔細研究過,此次恰好詳細看看。緩存
SESSION1 | SESSION2 | SESSION3 |
---|---|---|
步驟1:select sleep(1000) from baguait1 | ||
步驟2:flush table with read lock;堵塞 | ||
步驟3:kill session2 | ||
步驟4:select * from baguait1 limit 1;堵塞 |
步驟2 「flush table with read lock;」操做等待狀態爲 「Waiting for table flush」,狀態以下:微信
mysql> select Id,State,Info from information_schema.processlist where command<>'sleep'; +----+-------------------------+------------------------------------------------------------------------------------+ | Id | State | Info | +----+-------------------------+------------------------------------------------------------------------------------+ | 1 | Waiting on empty queue | NULL | | 26 | User sleep | select sleep(1000) from baguait1 | | 23 | Waiting for table flush | flush table with read lock | | 6 | executing | select Id,State,Info from information_schema.processlist where command<>'sleep' | +----+-------------------------+------------------------------------------------------------------------------------+
步驟4 「select * from testmts.baguait1 limit 1」操做等待狀態爲 「Waiting for table flush」,這個現象看起來很是奇怪沒有任何特殊的其餘操做,select竟然堵塞了。session
mysql> select Id,State,Info from information_schema.processlist where command<>'sleep'; +----+-------------------------+------------------------------------------------------------------------------------+ | Id | State | Info | +----+-------------------------+------------------------------------------------------------------------------------+ | 1 | Waiting on empty queue | NULL | | 26 | User sleep | select sleep(1000) from baguait1 | | 27 | executing | select Id,State,Info from information_schema.processlist where command<>'sleep' | | 6 | Waiting for table flush | select * from testmts.baguait1 limit 1 | +----+-------------------------+------------------------------------------------------------------------------------+
若是仔細對比兩個案例實際上區別僅僅在於 步驟1中的select 語句是否加了for update,案例2中咱們發現即使咱們將「flush table with read lock;」會話KILL掉也會堵塞隨後的關於本表上所有操做(包括select),這個等待實際上會持續到步驟1的sleep操做完成事後。app
對於線上數據庫的話,若是在長時間的select大表期間執行「flush table with read lock;」就會出現這種狀況,這種狀況會形成所有關於本表的操做等待,即使你發現後殺掉了FTWRL會話也無濟於事,等待會持續到select操做完成後,除非你KILL掉長時間的select操做。函數
爲何會出現這種狀況呢?咱們接下來慢慢分析。工具
關於本案例中我使用 sleep 函數來代替 select 大表操做作爲測試,在這裏這個代替是成立的。爲何成立呢咱們來看一下sleep函數的生效點以下:
T@3: | | | | | | | | >evaluate_join_record T@3: | | | | | | | | | enter: join: 0x7ffee0007350 join_tab index: 0 table: tii cond: 0x0 T@3: | | | | | | | | | counts: evaluate_join_record join->examined_rows++: 1 T@3: | | | | | | | | | >end_send T@3: | | | | | | | | | | >Query_result_send::send_data T@3: | | | | | | | | | | | >send_result_set_row T@3: | | | | | | | | | | | | >THD::enter_cond T@3: | | | | | | | | | | | | | THD::enter_stage: 'User sleep' /mysqldata/percona-server-locks-detail-5.7.22/sql/item_func.cc:6057 T@3: | | | | | | | | | | | | | >PROFILING::status_change T@3: | | | | | | | | | | | | | <PROFILING::status_change 384 T@3: | | | | | | | | | | | | <THD::enter_cond 3405
這裏看出sleep的生效點實際上每次Innodb層返回一行數據通過where條件判斷後,再觸發sleep函數,也就是每行通過where條件過濾的數據在發送給客戶端以前都會進行一次sleep操做。這個時候實際上該打開表的和該上MDL LOCK的都已經完成了,所以使用sleep函數來模擬大表select操做致使的FTWRL堵塞是能夠的。
實際上這部分咱們能夠在函數mysql_execute_command尋找case SQLCOM_FLUSH 的部分,實際上主要調用函數爲reload_acl_and_cache,其中核心部分爲:
if (thd->global_read_lock.lock_global_read_lock(thd))//加 MDL GLOBAL 級別S鎖 return 1; // Killed if (close_cached_tables(thd, tables, //關閉表操做釋放 share 和 cache ((options & REFRESH_FAST) ? FALSE : TRUE), thd->variables.lock_wait_timeout)) //等待時間受lock_wait_timeout影響 { /* NOTE: my_error() has been already called by reopen_tables() within close_cached_tables(). */ result= 1; } if (thd->global_read_lock.make_global_read_lock_block_commit(thd)) // MDL COMMIT 鎖 { /* Don't leave things in a half-locked state */ thd->global_read_lock.unlock_global_read_lock(thd); return 1; }
更具體的關閉表的操做和釋放table緩存的部分包含在函數close_cached_tables中,我就不詳細寫了。可是咱們須要明白table緩存實際上包含兩個部分:
這裏我統稱爲table緩存,好了下面是我總結的 FTWRL 的大概步驟:
第一步: 加MDL LOCK類型爲GLOBAL 級別爲S。若是出現等待狀態爲‘Waiting for global read lock’。注意select語句不會上GLOBAL級別上鎖,可是DML/DDL/FOR UPDATE語句會上GLOBAL級別的IX鎖,IX鎖和S鎖不兼容會出現這種等待。下面是這個兼容矩陣:
| Type of active | Request | scoped lock | type | IS(*) IX S X | ---------+------------------+ IS | + + + + | IX | + + - - | S | + - + - | X | + - - - |
第二步:推動全局表緩存版本。源碼中就是一個全局變量 refresh_version++。
第三步:釋放沒有使用的table 緩存。可自行參考函數close_cached_tables函數。
第四步:判斷是否有正在佔用的table緩存,若是有則等待,等待佔用者釋放。等待狀態爲'Waiting for table flush'。這一步會去判斷table緩存的版本和全局表緩存版本是否匹配,若是不匹配則等待以下:
for (uint idx=0 ; idx < table_def_cache.records ; idx++) { share= (TABLE_SHARE*) my_hash_element(&table_def_cache, idx); //尋找整個 table cache shared hash結構 if (share->has_old_version()) //若是版本 和 當前 的 refresh_version 版本不一致 { found= TRUE; break; //跳出第一層查找 是否有老版本 存在 } } ... if (found)//若是找到老版本,須要等待 { /* The method below temporarily unlocks LOCK_open and frees share's memory. */ if (share->wait_for_old_version(thd, &abstime, MDL_wait_for_subgraph::DEADLOCK_WEIGHT_DDL)) { mysql_mutex_unlock(&LOCK_open); result= TRUE; goto err_with_reopen; } }
而等待的結束就是佔用的table緩存的佔用者釋放,這個釋放操做存在於函數close_thread_table中,以下:
if (table->s->has_old_version() || table->needs_reopen() || table_def_shutdown_in_progress) { tc->remove_table(table);//關閉 table cache instance mysql_mutex_lock(&LOCK_open); intern_close_table(table);//去掉 table cache define mysql_mutex_unlock(&LOCK_open); }
最終會調用函數MDL_wait::set_status將FTWRL喚醒,也就是說對於正在佔用的 table 緩存釋而言,放者不是 FTWRL 會話線程而是佔用者本身的會話線程。無論怎麼樣最終整個table緩存將會被清空,若是通過FTWRL後去查看Open_table_definitions和Open_tables將會發現從新計數了。下面是喚醒函數的代碼,也很明顯:
bool MDL_wait::set_status(enum_wait_status status_arg) open_table { bool was_occupied= TRUE; mysql_mutex_lock(&m_LOCK_wait_status); if (m_wait_status == EMPTY) { was_occupied= FALSE; m_wait_status= status_arg; mysql_cond_signal(&m_COND_wait_status);//喚醒 } mysql_mutex_unlock(&m_LOCK_wait_status);//解鎖 return was_occupied; }
第五步:加MDL LOCK類型COMMIT 級別爲S。若是出現等待狀態爲‘Waiting for commit lock’。若是有大事務的提交極可能出現這種等待。
步驟1 咱們使用select for update語句,這個語句會加GLOBAL級別的IX鎖,持續到語句結束(注意實際上還會加對象級別的MDL_SHARED_WRITE(SW)鎖持續到事務結束,和FTWRL無關不作描述)
步驟2 咱們使用FTWRL語句,根據上面的分析須要獲取GLOBAL級別的S鎖,不兼容,所以出現了等待‘Waiting for global read lock’
步驟3 咱們KILL掉了FTWRL會話,這種狀況下會話退出,FTWRL就像沒有執行過同樣不會有任何影響,由於它在第一步就堵塞了。
步驟4 咱們的select操做不會受到任何影響,由於在GLOBAL級別select不會加MDL LOCK,對象級別MDL LOCK select/select for update是兼容的(即MDL_SHARED_READ(SR)和MDL_SHARED_WRITE(SW)兼容),且FTWRL尚未執行實際的操做。
步驟1 咱們使用select 語句,這個語句不會在GLOBAL級別上任何的鎖(注意實際上還會加對象級別的MDL_SHARED_READ(SR)鎖持續到事務結束,和FTWRL無關不作描述)
步驟2 咱們使用FTWRL語句,根據上面的分析咱們發現FTWRL語句能夠獲取了GLOBAL 級別的S鎖,由於單純的select 語句不會在GLOBAL級別上任何鎖。同時會將全局表緩存版本推動而後釋放掉沒有使用的table 緩存。可是在第三節的第四步中咱們發現由於baguait1的表緩存正在被佔用,所以出現了等待,等待狀態爲'Waiting for table flush'。
步驟3 咱們KILL掉了FTWRL會話,這種狀況下雖然GLOBAL 級別的S鎖會釋放,可是全局表緩存版本已經推動了,同時沒有使用的table 緩存已經釋放掉了。
步驟4 再次執行一個baguait1表上的select 查詢操做,這個時候在打開表的時候會去判斷是否table緩存的版本和全局表緩存版本匹配若是不匹配進入等待,等待爲‘Waiting for table flush’,下面是這個判斷:
if (share->has_old_version()) { /* We already have an MDL lock. But we have encountered an old version of table in the table definition cache which is possible when someone changes the table version directly in the cache without acquiring a metadata lock (e.g. this can happen during "rolling" FLUSH TABLE(S)). Release our reference to share, wait until old version of share goes away and then try to get new version of table share. */ release_table_share(share); ... wait_result= tdc_wait_for_old_version(thd, table_list->db, table_list->table_name, ot_ctx->get_timeout(), deadlock_weight);
整個等待操做和FTWRL同樣,會等待佔用者釋放table緩存後纔會醒來繼續。
所以後續本表的全部select/DML/DDL都會堵塞,代價極高,即使KILL掉FTWRL會話也無用。
最後提醒一下不少備份工具都要執行FTWRL操做,必定要注意它的堵塞/被堵塞場景和特殊場景。
(1)使用的斷點
(2)FTWRL堵塞棧幀因爲select堵塞棧幀:
(gdb) bt #0 0x00007ffff7bd3a5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000192027b in native_cond_timedwait (cond=0x7ffedc007c78, mutex=0x7ffedc007c30, abstime=0x7fffec5bbb90) at /mysqldata/percona-server-locks-detail-5.7.22/include/thr_cond.h:129 #2 0x00000000019205ea in safe_cond_timedwait (cond=0x7ffedc007c78, mp=0x7ffedc007c08, abstime=0x7fffec5bbb90, file=0x204cdd0 "/mysqldata/percona-server-locks-detail-5.7.22/sql/mdl.cc", line=1899) at /mysqldata/percona-server-locks-detail-5.7.22/mysys/thr_cond.c:88 #3 0x00000000014b9f21 in my_cond_timedwait (cond=0x7ffedc007c78, mp=0x7ffedc007c08, abstime=0x7fffec5bbb90, file=0x204cdd0 "/mysqldata/percona-server-locks-detail-5.7.22/sql/mdl.cc", line=1899) at /mysqldata/percona-server-locks-detail-5.7.22/include/thr_cond.h:180 #4 0x00000000014ba484 in inline_mysql_cond_timedwait (that=0x7ffedc007c78, mutex=0x7ffedc007c08, abstime=0x7fffec5bbb90, src_file=0x204cdd0 "/mysqldata/percona-server-locks-detail-5.7.22/sql/mdl.cc", src_line=1899) at /mysqldata/percona-server-locks-detail-5.7.22/include/mysql/psi/mysql_thread.h:1229 #5 0x00000000014bb702 in MDL_wait::timed_wait (this=0x7ffedc007c08, owner=0x7ffedc007b70, abs_timeout=0x7fffec5bbb90, set_status_on_timeout=true, wait_state_name=0x2d897b0) at /mysqldata/percona-server-locks-detail-5.7.22/sql/mdl.cc:1899 #6 0x00000000016cdb30 in TABLE_SHARE::wait_for_old_version (this=0x7ffee0a4fc30, thd=0x7ffedc007b70, abstime=0x7fffec5bbb90, deadlock_weight=100) at /mysqldata/percona-server-locks-detail-5.7.22/sql/table.cc:4717 #7 0x000000000153829b in close_cached_tables (thd=0x7ffedc007b70, tables=0x0, wait_for_refresh=true, timeout=31536000) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_base.cc:1291 #8 0x00000000016123ec in reload_acl_and_cache (thd=0x7ffedc007b70, options=16388, tables=0x0, write_to_binlog=0x7fffec5bc9dc) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_reload.cc:224 #9 0x00000000015cee9c in mysql_execute_command (thd=0x7ffedc007b70, first_level=true) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_parse.cc:4433 #10 0x00000000015d2fde in mysql_parse (thd=0x7ffedc007b70, parser_state=0x7fffec5bd600) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_parse.cc:5901 #11 0x00000000015c6b72 in dispatch_command (thd=0x7ffedc007b70, com_data=0x7fffec5bdd70, command=COM_QUERY) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_parse.cc:1490
(3)殺點FTWRL會話後其餘select操做等待棧幀:
#0 MDL_wait::timed_wait (this=0x7ffee8008298, owner=0x7ffee8008200, abs_timeout=0x7fffec58a600, set_status_on_timeout=true, wait_state_name=0x2d897b0) at /mysqldata/percona-server-locks-detail-5.7.22/sql/mdl.cc:1888 #1 0x00000000016cdb30 in TABLE_SHARE::wait_for_old_version (this=0x7ffee0011620, thd=0x7ffee8008200, abstime=0x7fffec58a600, deadlock_weight=0) at /mysqldata/percona-server-locks-detail-5.7.22/sql/table.cc:4717 #2 0x000000000153b6ba in tdc_wait_for_old_version (thd=0x7ffee8008200, db=0x7ffee80014a0 "testmts", table_name=0x7ffee80014a8 "tii", wait_timeout=31536000, deadlock_weight=0) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_base.cc:2957 #3 0x000000000153ca97 in open_table (thd=0x7ffee8008200, table_list=0x7ffee8001708, ot_ctx=0x7fffec58aab0) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_base.cc:3548 #4 0x000000000153f904 in open_and_process_table (thd=0x7ffee8008200, lex=0x7ffee800a830, tables=0x7ffee8001708, counter=0x7ffee800a8f0, flags=0, prelocking_strategy=0x7fffec58abe0, has_prelocking_list=false, ot_ctx=0x7fffec58aab0) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_base.cc:5213 #5 0x0000000001540a58 in open_tables (thd=0x7ffee8008200, start=0x7fffec58aba0, counter=0x7ffee800a8f0, flags=0, prelocking_strategy=0x7fffec58abe0) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_base.cc:5831 #6 0x0000000001541e93 in open_tables_for_query (thd=0x7ffee8008200, tables=0x7ffee8001708, flags=0) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_base.cc:6606 #7 0x00000000015d1dca in execute_sqlcom_select (thd=0x7ffee8008200, all_tables=0x7ffee8001708) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_parse.cc:5416 #8 0x00000000015ca380 in mysql_execute_command (thd=0x7ffee8008200, first_level=true) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_parse.cc:2939 #9 0x00000000015d2fde in mysql_parse (thd=0x7ffee8008200, parser_state=0x7fffec58c600) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_parse.cc:5901 #10 0x00000000015c6b72 in dispatch_command (thd=0x7ffee8008200, com_data=0x7fffec58cd70, command=COM_QUERY) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_parse.cc:1490
(4)佔用者釋放喚醒FTWRL棧幀:
Breakpoint 3, MDL_wait::set_status (this=0x7ffedc000c78, status_arg=MDL_wait::GRANTED) at /mysqldata/percona-server-locks-detail-5.7.22/sql/mdl.cc:1832 1832 bool was_occupied= TRUE; (gdb) bt #0 MDL_wait::set_status (this=0x7ffedc000c78, status_arg=MDL_wait::GRANTED) at /mysqldata/percona-server-locks-detail-5.7.22/sql/mdl.cc:1832 #1 0x00000000016c2483 in free_table_share (share=0x7ffee0011620) at /mysqldata/percona-server-locks-detail-5.7.22/sql/table.cc:607 #2 0x0000000001536a22 in table_def_free_entry (share=0x7ffee0011620) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_base.cc:524 #3 0x00000000018fd7aa in my_hash_delete (hash=0x2e4cfe0, record=0x7ffee0011620 "\002") at /mysqldata/percona-server-locks-detail-5.7.22/mysys/hash.c:625 #4 0x0000000001537673 in release_table_share (share=0x7ffee0011620) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_base.cc:949 #5 0x00000000016cad10 in closefrm (table=0x7ffee000f280, free_share=true) at /mysqldata/percona-server-locks-detail-5.7.22/sql/table.cc:3597 #6 0x0000000001537d0e in intern_close_table (table=0x7ffee000f280) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_base.cc:1109 #7 0x0000000001539054 in close_thread_table (thd=0x7ffee0000c00, table_ptr=0x7ffee0000c68) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_base.cc:1780 #8 0x00000000015385fe in close_open_tables (thd=0x7ffee0000c00) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_base.cc:1443 #9 0x0000000001538d4a in close_thread_tables (thd=0x7ffee0000c00) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_base.cc:1722 #10 0x00000000015d19bc in mysql_execute_command (thd=0x7ffee0000c00, first_level=true) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_parse.cc:5307 #11 0x00000000015d2fde in mysql_parse (thd=0x7ffee0000c00, parser_state=0x7fffec5ee600) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_parse.cc:5901 #12 0x00000000015c6b72 in dispatch_command (thd=0x7ffee0000c00, com_data=0x7fffec5eed70, command=COM_QUERY) at /mysqldata/percona-server-locks-detail-5.7.22/sql/sql_parse.cc:1490
最後推薦高鵬的專欄《深刻理解MySQL主從原理 32講》,想要透徹瞭解學習MySQL 主從原理的朋友不容錯過。做者微信:gp_22389860