系統變慢調整小結

時間 2019-12-06

標籤系統變慢調整小結简体版

原文原文鏈接

客戶反映系統越用越慢，重啓服務器後段時間內系統恢復正常java

老年代垃圾回收不了。 full gc 每分鐘執行40次左右。web

開始覺得是內存泄漏，導出jvm內存快照spring

快照中能夠看到線程池裏有問題。可是看不出具體的問題緣由apache

導出線程快照沒有發現可疑線程信息。。。鬱悶了。。（當前時間19:21已經沒有客戶在操做系統了）服務器

次日繼續今天的問題。上午客戶打電話說又有問題了。app

趕忙導出線程快照jvm

$jstack pidsocket

"Running HTTPHandler-26" prio=10 tid=0x00007f2140001000 nid=0x3c9b waiting for monitor entry [0x00007f21f4d50000]spa

java.lang.Thread.State: BLOCKED (on object monitor)操作系統

at com.sinosoft.xf.petition.petitionaccept.manager.RepeatJudgeManager.repeatJudgeMethod(RepeatJudgeManager.java:266)

- waiting to lock <0x00000007a994d600> (a com.sinosoft.xf.petition.petitionaccept.manager.RepeatJudgeManager)

at com.sinosoft.xf.petition.petitionaccept.manager.RepeatJudgeManager$$FastClassByCGLIB$$971a2e6a.invoke(<generated>)

at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)

at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)

at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)

at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:635)

at com.sinosoft.xf.petition.petitionaccept.manager.RepeatJudgeManager$$EnhancerByCGLIB$$1805e0ee.repeatJudgeMethod(<generated>)

at com.sinosoft.xf.petition.petitionaccept.web.RepeatJudgeAction.repeatJudgeMethod(RepeatJudgeAction.java:385)

at com.sinosoft.xf.petition.petitionaccept.web.RepeatJudgeAction$$FastClassByCGLIB$$720d47c0.invoke(<generated>)

at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)

java.lang.Thread.State: BLOCKED (on object monitor)：線程阻塞

- waiting to lock <0x00000007a994d600> (a com.sinosoft.xf.petition.petitionaccept.manager.RepeatJudgeManager)：線程在等待<0x00000007a994d600>這個鎖。由於沒法獲取到因此發生阻塞。

#cat statck.txt | grep 'java.lang.Thread.State' | sort | uniq -c | sort -n > 1.txt

18 java.lang.Thread.State: TIMED_WAITING (sleeping)

18 java.lang.Thread.State: WAITING (parking)

45 java.lang.Thread.State: TIMED_WAITING (parking)

63 java.lang.Thread.State: WAITING (on object monitor)

132 java.lang.Thread.State: RUNNABLE

135 java.lang.Thread.State: BLOCKED (on object monitor)

300 java.lang.Thread.State: TIMED_WAITING (on object monitor)

發現居然有135個線程發生了阻塞

經過導出線程信息定位到具體的代碼位置。。

public synchronized String repeatJudgeMethod() {
    ...
}

根據業務需求拆分，調整該方法。

以後有又觀察一兩天時間發現系統仍是很慢，繼續導出線程快照

發現該方法執行了一個批量更新，並且這邊更新過程比較慢，鎖住了一大批的數據致使

"Running HTTPHandler-10" prio=10 tid=0x00007f21b0004800 nid=0x48c1 runnable [0x00007f21f5afa000]

java.lang.Thread.State: RUNNABLE

at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.read(SocketInputStream.java:129)

at com.ibm.db2.jcc.t4.ab.b(ab.java:193)

at com.ibm.db2.jcc.t4.ab.c(ab.java:237)

at com.ibm.db2.jcc.t4.ab.c(ab.java:348)

at com.ibm.db2.jcc.t4.ab.v(ab.java:1133)

at com.ibm.db2.jcc.t4.eb.c(eb.java:30)

at com.ibm.db2.jcc.t4.u.a(u.java:32)

at com.ibm.db2.jcc.t4.j.Xb(j.java:245)

at com.ibm.db2.jcc.b.jk.X(jk.java:3412)

at com.ibm.db2.jcc.t4.d.lb(d.java:1928)

at com.ibm.db2.jcc.t4.d.n(d.java:2115)

at com.ibm.db2.jcc.t4.d.o(d.java:2127)

at com.ibm.db2.jcc.t4.d.a(d.java:163)

at com.ibm.db2.jcc.t4.d.b(d.java:478)

at com.ibm.db2.jcc.b.bc.a(bc.java:213)

at com.ibm.db2.jcc.b.jk.c(jk.java:296)

at com.ibm.db2.jcc.b.jk.next(jk.java:258)

- locked <0x00000007b02094a8> (a com.ibm.db2.jcc.t4.b)

at com.apusic.jdbc.adapter.ResultSetHandle.next(Unknown Source)

at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:91)

at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:448)

at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:396)

at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:458)

at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:466)

at org.springframework.jdbc.core.JdbcTemplate.queryForList(JdbcTemplate.java:497)

at com.sinosoft.xf.petition.petitionaccept.dao.RepeatJudgeDao.getRepeatInfoTreeList(RepeatJudgeDao.java:429)

at com.sinosoft.xf.petition.petitionaccept.dao.RepeatJudgeDao$$FastClassByCGLIB$$f0ee3a4a.invoke(<generated>)

at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)

at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)

at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:635)

at com.sinosoft.xf.petition.petitionaccept.dao.RepeatJudgeDao$$EnhancerByCGLIB$$e0e3d2b0.getRepeatInfoTreeList(<generated>)

at com.sinosoft.xf.petition.petitionaccept.manager.RepeatJudgeManager.editRepeatInfoRepeatNum(RepeatJudgeManager.java:476)

at com.sinosoft.xf.petition.petitionaccept.manager.RepeatJudgeManager.repeatJudgeMethod(RepeatJudgeManager.java:370)

- locked <0x00000007a994d600> (a com.sinosoft.xf.petition.petitionaccept.manager.RepeatJudgeManager)