MySQL參數max_connect_errors分析釋疑

 

最近一MySQL服務器，因爲一些特殊因素遇到「ERROR 1129 (00000): Host 'xxx' is blocked because of many connection errors. Unblock with 'mysqladmin flush-hosts'」，在問題解決後，在詳細瞭解參數max_connect_errors的過程當中，有些不一樣網絡資料的矛盾描述確實讓我有點迷惑和混淆（關於這個錯誤，本質緣由是由於同一個IP在短期內產生太多中斷的數據庫鏈接（超過max_connect_errors的最大值）而致使的），下面介紹個人探索問題、分析問題、釋疑的一個過程。

 

首先，我在網上搜索了一些資料，很多資料信誓旦旦的介紹，密碼輸入錯誤的嘗試次數超過max_connect_errors變量，MySQL就會阻塞這個客戶端登陸，而後我找到了官方資料關於max_connect_errors的介紹，以下所示，MySQL 5.6/5.7的介紹一致

 

 

If more than this many successive connection requests from a host are interrupted without a successful connection, the server blocks that host from further connections. You can unblock blocked hosts by flushing the host cache. To do so, issue a FLUSH HOSTS statement or execute a mysqladmin flush-hosts command. If a connection is established successfully within fewer than max_connect_errors attempts after a previous connection was interrupted, the error count for the host is cleared to zero. However, once a host is blocked, flushing the host cache is the only way to unblock it. The default is 100.

 

如上所示，翻譯出來的話，大體以下：若是MySQL服務器連續接收到了來自於同一個主機的請求，並且這些連續的請求所有都沒有成功的創建鏈接就被中斷了，當這些連續的請求的累計值大於max_connect_errors的設定值時，MySQL服務器就會阻止這臺主機後續的全部請求。相信一開始你看到這些資料，也會被「many successive connection requests from a host are interrupted without a successful connection」給弄懵，其實這個就是由於因爲網絡異常而停止數據庫鏈接。網上搜索到這麼一個資料：

 

There seems to be confusion around that variable. It does not really block hosts for repeated invalid passwords but for aborted connections due to network errors.

 

 

 

好吧，那麼咱們本身動手實驗驗證一下，就能弄明白到底那個是正確的。在MySQL數據庫裏面建立一個test帳號，而後咱們將max_connect_errors變量設置爲3.

 

 

 

mysql> show variables like '%max_connect_errors%';

+--------------------+-------+

| Variable_name      | Value |

+--------------------+-------+

| max_connect_errors | 100   |

+--------------------+-------+

1 row in set (0.00 sec)

mysql> set global max_connect_errors=3;

Query OK, 0 rows affected (0.00 sec)

mysql> show variables like '%max_connect_error%';

+--------------------+-------+

| Variable_name      | Value |

+--------------------+-------+

| max_connect_errors | 3     |

+--------------------+-------+

1 row in set (0.00 sec)

 

 

 

而後咱們在另一臺測試機器，以錯誤的密碼去鏈接這個MySQL數據庫，以下所示，即便前面輸入了三次錯誤密碼，第四次輸入是也沒有碰到上面錯誤。那麼能夠排除這個變量與密碼錯誤輸入有關係。

 

 

[root@mytestlnx02 tmp]# mysql -h10.20.57.24 -utest -p

Enter password:

ERROR 1045 (28000): Access denied for user 'test'@'mytestlnx02' (using password: YES)

[root@mytestlnx02 tmp]# mysql -h10.20.57.24 -utest -p

Enter password:

ERROR 1045 (28000): Access denied for user 'test'@'mytestlnx02' (using password: YES)

[root@mytestlnx02 tmp]# mysql -h10.20.57.24 -utest -p

Enter password:

ERROR 1045 (28000): Access denied for user 'test'@'mytestlnx02' (using password: YES)

[root@mytestlnx02 tmp]# mysql -h10.20.57.24 -utest -p

Enter password:

ERROR 1045 (28000): Access denied for user 'test'@'mytestlnx02' (using password: YES)

[root@mytestlnx02 tmp]#

 

 

其實，關於某個IP輸入了錯誤密碼，MySQL會在performance_schema數據庫下的host_cache表中記錄。它會累計記錄在COUNT_AUTHENTICATION_ERRORS字段，以下所示：

 

 

mysql> use performance_schema;

Database changed

mysql> select * from host_cache\G;

*************************** 1. row ***************************

                                        IP: 192.168.27.180

                                      HOST: gettestlnx02

                            HOST_VALIDATED: YES

                        SUM_CONNECT_ERRORS: 0

                 COUNT_HOST_BLOCKED_ERRORS: 0

           COUNT_NAMEINFO_TRANSIENT_ERRORS: 0

           COUNT_NAMEINFO_PERMANENT_ERRORS: 0

                       COUNT_FORMAT_ERRORS: 0

           COUNT_ADDRINFO_TRANSIENT_ERRORS: 0

           COUNT_ADDRINFO_PERMANENT_ERRORS: 0

                       COUNT_FCRDNS_ERRORS: 0

                     COUNT_HOST_ACL_ERRORS: 0

               COUNT_NO_AUTH_PLUGIN_ERRORS: 0

                  COUNT_AUTH_PLUGIN_ERRORS: 0

                    COUNT_HANDSHAKE_ERRORS: 0

                   COUNT_PROXY_USER_ERRORS: 0

               COUNT_PROXY_USER_ACL_ERRORS: 0

               COUNT_AUTHENTICATION_ERRORS: 4

                          COUNT_SSL_ERRORS: 0

         COUNT_MAX_USER_CONNECTIONS_ERRORS: 0

COUNT_MAX_USER_CONNECTIONS_PER_HOUR_ERRORS: 0

             COUNT_DEFAULT_DATABASE_ERRORS: 0

                 COUNT_INIT_CONNECT_ERRORS: 0

                        COUNT_LOCAL_ERRORS: 0

                      COUNT_UNKNOWN_ERRORS: 0

                                FIRST_SEEN: 2018-01-31 16:28:19

                                 LAST_SEEN: 2018-01-31 16:28:26

                          FIRST_ERROR_SEEN: 2018-01-31 16:28:19

                           LAST_ERROR_SEEN: 2018-01-31 16:28:26

1 row in set (0.00 sec)

ERROR: 

No query specified

 

 

 

 

 

官方資料介紹，host_cache的字段是統計被視爲「阻塞」的鏈接錯誤的數量（根據max_connect_errors系統變量進行評估）。 只計算協議握手錯誤，而且僅用於經過驗證的主機（HOST_VALIDATED = YES）。

 

SUM_CONNECT_ERRORS

The number of connection errors that are deemed 「blocking」 (assessed against the max_connect_errors system variable). Only protocol handshake errors are counted, and only for hosts that passed validation (HOST_VALIDATED = YES).

 

MySQL客戶端與數據庫創建鏈接須要發起三次握手協議，正常狀況下，這個時間很是短，可是一旦網絡異常，網絡超時等因素出現，就會致使這個握手協議沒法完成，MySQL有個參數connect_timeout，它是MySQL服務端進程mysqld等待鏈接創建完成的時間，單位爲秒。若是超過connect_timeout時間範圍內，仍然沒法完成協議握手話，MySQL客戶端會收到異常，異常消息相似於: Lost connection to MySQL server at 'XXX', system error: errno，該變量默認是10秒： 

 

 

 

mysql> show variables like 'connect_timeout';

+-----------------+-------+

| Variable_name   | Value |

+-----------------+-------+

| connect_timeout | 10    |

+-----------------+-------+

1 row in set (0.00 sec)

mysql> 

 

 

 

那麼咱們就構造一個網絡超時引發的數據庫鏈接被中斷案例吧，咱們用Linux下的netem與tc命令模擬構造出複雜環境下的網絡傳輸延時案例，以下設置後，此時從測試服務器去訪問MySQL服務器，都會出現延時11秒：

 

 

 

 

[root@gettestlnx02 ~]# ping 10.20.57.24

PING 10.20.57.24 (10.20.57.24) 56(84) bytes of data.

64 bytes from 10.20.57.24: icmp_seq=1 ttl=62 time=0.251 ms

64 bytes from 10.20.57.24: icmp_seq=2 ttl=62 time=0.330 ms

64 bytes from 10.20.57.24: icmp_seq=3 ttl=62 time=0.362 ms

64 bytes from 10.20.57.24: icmp_seq=4 ttl=62 time=0.316 ms

64 bytes from 10.20.57.24: icmp_seq=5 ttl=62 time=0.281 ms

64 bytes from 10.20.57.24: icmp_seq=6 ttl=62 time=0.377 ms

^C

--- 10.20.57.24 ping statistics ---

6 packets transmitted, 6 received, 0% packet loss, time 5716ms

rtt min/avg/max/mdev = 0.251/0.319/0.377/0.047 ms

[root@gettestlnx02 ~]# tc qdisc add dev eth0 root netem delay 11000ms

[root@gettestlnx02 ~]# ping 10.20.57.24

PING 10.20.57.24 (10.20.57.24) 56(84) bytes of data.

64 bytes from 10.20.57.24: icmp_seq=1 ttl=62 time=11000 ms

64 bytes from 10.20.57.24: icmp_seq=2 ttl=62 time=11000 ms

64 bytes from 10.20.57.24: icmp_seq=3 ttl=62 time=11000 ms

64 bytes from 10.20.57.24: icmp_seq=4 ttl=62 time=11000 ms

64 bytes from 10.20.57.24: icmp_seq=5 ttl=62 time=11000 ms

64 bytes from 10.20.57.24: icmp_seq=6 ttl=62 time=11000 ms

64 bytes from 10.20.57.24: icmp_seq=7 ttl=62 time=11000 ms

 

 

 

 

 

咱們在測試服務器gettestlnx02鏈接MySQL數據庫，以下所示（注意，若是你是在經過ssh鏈接這臺服務器的話，此時在gettestlnx02上操做會至關慢。固然你也能夠在MySQL服務器模擬網絡延時，或者你將connect_timeout和網絡延時都設小一點）

 

 

 

[root@gettestlnx02 ~]# mysql -h10.20.57.24 -utest -p

Enter password:

ERROR 2013 (HY000): Lost connection to MySQL server at 'reading authorization packet', system error: 0

[root@gettestlnx02 ~]#

 

 

 

如上所示，因爲網絡延時超過10秒，致使鏈接MySQL失敗，此時，你在MySQL服務器上查詢host_cache表時,那麼你就會看到SUM_CONNECT_ERRORS變成1了，COUNT_HANDSHAKE_ERRORS也變成了1.

 

 

 

 

 

 

那麼咱們反覆這樣折騰三次，那麼你會看到SUM_CONNECT_ERRORS變成3了，COUNT_HANDSHAKE_ERRORS也變成了3了.

 

 

 

 

mysql> select * from host_cache\G;

*************************** 1. row ***************************

                                        IP: 192.168.27.180

                                      HOST: gettestlnx02

                            HOST_VALIDATED: YES

                        SUM_CONNECT_ERRORS: 3

                 COUNT_HOST_BLOCKED_ERRORS: 1

           COUNT_NAMEINFO_TRANSIENT_ERRORS: 0

           COUNT_NAMEINFO_PERMANENT_ERRORS: 0

                       COUNT_FORMAT_ERRORS: 0

           COUNT_ADDRINFO_TRANSIENT_ERRORS: 0

           COUNT_ADDRINFO_PERMANENT_ERRORS: 0

                       COUNT_FCRDNS_ERRORS: 0

                     COUNT_HOST_ACL_ERRORS: 0

               COUNT_NO_AUTH_PLUGIN_ERRORS: 0

                  COUNT_AUTH_PLUGIN_ERRORS: 0

                    COUNT_HANDSHAKE_ERRORS: 3

                   COUNT_PROXY_USER_ERRORS: 0

               COUNT_PROXY_USER_ACL_ERRORS: 0

               COUNT_AUTHENTICATION_ERRORS: 4

                          COUNT_SSL_ERRORS: 0

         COUNT_MAX_USER_CONNECTIONS_ERRORS: 0

COUNT_MAX_USER_CONNECTIONS_PER_HOUR_ERRORS: 0

             COUNT_DEFAULT_DATABASE_ERRORS: 0

                 COUNT_INIT_CONNECT_ERRORS: 0

                        COUNT_LOCAL_ERRORS: 0

                      COUNT_UNKNOWN_ERRORS: 0

                                FIRST_SEEN: 2018-01-31 16:28:19

                                 LAST_SEEN: 2018-01-31 17:02:10

                          FIRST_ERROR_SEEN: 2018-01-31 16:28:19

                           LAST_ERROR_SEEN: 2018-01-31 17:02:10

1 row in set (0.00 sec)

ERROR: 

No query specified

 

 

 

 

而後咱們用netem與tc 命令在測試服務器上取消網絡延遲模擬，而後去測試鏈接MySQL數據庫，以下測試所示：

 

 

[root@gettestlnx02 ~]# tc qdisc del dev eth0 root netem delay 11000ms

[root@gettestlnx02 ~]# mysql -h10.20.57.24 -utest -p

Enter password:

ERROR 1129 (HY000): Host '192.168.27.180' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'

[root@gettestlnx02 ~]#

 

 

 

 

 

 

 

此時就能構造出「ERROR 1129 (HY000): Host '192.168.27.180' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'」錯誤了。

 

 

 

 

 

解決方案

 

 

 

 解決ERROR 1129 (00000): Host 'xxx' is blocked because of many connection errors. Unblock with 'mysqladmin flush-hosts'這個錯誤的方法比較多，不過有些方案都是臨時方案。臨時方案是指標不治本。關鍵仍是須要解決網絡錯誤（這個每每須要求助網絡管理人員或系統管理員）

 

解決方法：

 

 

1、將變量max_connection_errors的值設置爲一個更大的值

　

 

mysql> show variables like 'max_connect_errors';

+--------------------+-------+

| Variable_name      | Value |

+--------------------+-------+

| max_connect_errors | 3     |

+--------------------+-------+

1 row in set (0.01 sec)

mysql> set global max_connect_errors=150;

Query OK, 0 rows affected (0.00 sec)

mysql> 

 

 

這個臨時方案只是延遲觸發IP被禁止訪問的條件而已，並且在複雜狀況或高併發的狀況下，須要設置一個很大的值，不然很容易就會再次被觸發。另外，變量只對當前環境生效，若是重啓就會失效，若是須要永久有效，能夠在my.cnf配置文件裏面配置。

 

 

2：使用flush hosts

 

mysql> flush hosts;

Query OK, 0 rows affected (0.00 sec)

 

mysql> select * from performance_schema.host_cache;

Empty set (0.00 sec)

 

mysql>

 

固然你也能夠mysqladmin flush-hosts 命令清理一下hosts cache信息

 

[root@DB-Server ~]# mysqladmin  --port=3306 -uroot -p flush-host

Enter password:

 

那麼host cache是什麼呢？ 官方介紹以下：

 

The MySQL server maintains a host cache in memory that contains information about clients: IP address, host name, and error information. The server uses this cache for nonlocal TCP connections. It does not use the cache for TCP connections established using a loopback interface address (127.0.0.1 or ::1), or for connections established using a Unix socket file, named pipe, or shared memory.

 

 

簡單來講，就是MySQL服務器在內存中維護一個包含客戶端信息的緩存：IP地址，主機名和錯誤信息等。 服務器會將非本地TCP鏈接信息緩存起來。它不會緩存使用環回接口地址（127.0.0.1或者:: 1）創建的TCP鏈接，或者使用Unix套接字文件，命名管道或共享內存創建的鏈接。host cache信息能夠經過performance_schema數據庫下的host_cache表查詢。

 

3：將變量host_cache_size設置爲0

 

 

其實我想說這是一個最不靠譜的解決方法，只是讓MySQL服務器不記錄host cache信息而已。徹底能夠忽略這個方法。 

 

mysql> show variables like '%host_cache_size%';

+-----------------+-------+

| Variable_name   | Value |

+-----------------+-------+

| host_cache_size | 279   |

+-----------------+-------+

1 row in set (0.00 sec)

mysql> set global host_cache_size=0;

Query OK, 0 rows affected (0.00 sec)

mysql> select * from performance_schema.host_cache;

Empty set (0.00 sec)

mysql> 

 

 

 

知識點延伸

 

 

    關於參數max_connect_errors，不要誤解其功能，這個不能做爲防止窮舉密碼攻擊的手段。若是您擔憂SYN氾濫攻擊，max_connect_errors可能會在特定狀況下幫助您。 MySQL 5.6中的PERFORMANCE_SCHEMA改進提供了有關潛在強力攻擊的有意義的信息，可是隻有在涉及主機緩存的狀況下。關於這個能夠參考博文Understanding max_connect_errors

 

 

 

參考資料：

 

http://mysqlblog.fivefarmers.com/2013/08/08/understanding-max_connect_errors/

https://dev.mysql.com/doc/refman/5.6/en/host-cache.html

https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_max_connect_errors

https://dev.mysql.com/doc/refman/5.7/en/blocked-host.html