深刻分析Tomcat無響應問題及解決方法

問題描述

生產環境下有幾臺tomcat,但忽然某個時候發現全部的請求都不能響應了,因爲咱們的web server使用的是nginx,會將請求反向到tomcat上,因此起初懷疑是nginx就沒有收到請求,但查看日誌後發現,nginx中大量出現499的返回,這說明問題仍是出在tomcat上.

問題排查

首先我想到的是否是CPU跑滿了,雖然說CPU沒有報警但仍是本能的top命令看下系統負載,發現系統只有0.x的負載,cpu,內存消耗都是正常的.
因爲CPU沒有出現異常,因此應該不是GC出現了問題,但仍是檢查了下GC log,果真GC也沒問題
此時必須讓jstack上場了,果真在使用jstack後發現不少線程都是WAITING狀態

"http-nio-127.0.0.1-801-exec-498" daemon prio=10 tid=0x00002ada7c14f800 nid=0x16a6 waiting on condition [0x00002ada9c905000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007873e6990> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:133)
        at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:282)
        at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
        at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:177)
        at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:170)
        at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:102)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:240)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:227)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:173)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
        at com.weimai.utils.HttpClientUtil.doGet(HttpClientUtil.java:105)
        at com.weimai.utils.HttpClientUtil.doGet(HttpClientUtil.java:87)
        at com.weimai.utils.WeiBoUtil.checkUser(WeiBoUtil.java:214)
        at com.weimai.web.UserInfoController.newWeiboLogin(UserInfoController.java:1223)
        at sun.reflect.GeneratedMethodAccessor390.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        ...
 
此時意識到問題應該出現http鏈接上,立刻用netstat查看下801端口的鏈接狀態,果真發現不少請求都是CLOSE_WAIT,這裏簡單解釋下CLOSE_WAIT狀態,若是咱們的client程序處於CLOSE_WAIT狀態的話,說明套接字是被動關閉的,整個流程應該是這樣

由於若是是server端主動斷掉當前鏈接的話,那麼雙方關閉這個TCP鏈接共須要四個packet
 
server -> FIN -> client
 
server <- ACK <- client
 
這時候server端處於FIN_WAIT_2狀態,而咱們的程序處於CLOSE_WAIT狀態
 
server <- FIN <- client
 
這時client發送FIN給server,client就置爲LAST_ACK狀態。
 
server -> ACK -> client
 
server迴應了ACK,那麼client的套接字纔會真正置爲CLOSED狀態
咱們的請求處於CLOSE_WAIT狀態,而不是LAST_ACK狀態,說明尚未發FIN給server,那麼很簡單,去看HttpClientUtil中如何處理就知道了,果真在查看HttpClientUtil代碼中發現對於非正常關閉的http鏈接沒有作abort,補充完善好try catch finally塊後問題獲得解決.