weblogic調優以及線程阻塞引發的宕機

由weblogic相關日誌分析後認爲2020-1-19 下午03時04分43秒宕機緣由爲weblogic的線程阻塞，引發weblogic宕機；

####<2020-1-19 下午02時53分43秒 CST> <Info> <Health> <VM_35_2_redhat> <nc1> <weblogic.GCMonitor> <<anonymous>> <> <> <1579416823035> <BEA-310002> <49% of the total memory in the server is free> 
####<2020-1-19 下午02時55分43秒 CST> <Info> <Health> <VM_35_2_redhat> <nc1> <weblogic.GCMonitor> <<anonymous>> <> <> <1579416943036> <BEA-310002> <32% of the total memory in the server is free> 
####<2020-1-19 下午02時56分43秒 CST> <Info> <Health> <VM_35_2_redhat> <nc1> <weblogic.GCMonitor> <<anonymous>> <> <> <1579417003036> <BEA-310002> <43% of the total memory in the server is free> 
####<2020-1-19 下午03時00分11秒 CST> <Error> <HTTP> <VM_35_2_redhat> <nc1> <[ACTIVE] ExecuteThread: '28' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1579417211420> <BEA-101017> <[ServletContext@620021214[app:nc65 module:/ path: spec-version:2.5]] Root cause of ServletException.
javax.servlet.ServletException: not supported
	at nc.bs.framework.comn.serv.CommonServletDispatcher.doGet(CommonServletDispatcher.java:39)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
	at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
	at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)
	at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
	at nc.bs.framework.server.LoggerServletFilter.doFilter(LoggerServletFilter.java:35)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
	at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273)
	at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179)
	at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
> 
####<2020-1-19 下午03時04分43秒 CST> <Info> <Health> <VM_35_2_redhat> <nc1> <weblogic.GCMonitor> <<anonymous>> <> <> <1579417483038> <BEA-310002> <32% of the total memory in the server is free>

 

經排查後進行如下優化：

1）優化JDK內存；

2）修改StuckThreadMaxTime參數；

3)   增大線程數，防止線程阻塞問題。

 

優化JDK內存

setDomainEnv.sh

修改前：

if [ "${JAVA_VENDOR}" = "Sun" ] ; then

       WLS_MEM_ARGS_64BIT="-Xms256m -Xmx512m"

       export WLS_MEM_ARGS_64BIT

       WLS_MEM_ARGS_32BIT="-Xms256m -Xmx512m"

       export WLS_MEM_ARGS_32BIT

else

       WLS_MEM_ARGS_64BIT="-Xms512m -Xmx512m"

       export WLS_MEM_ARGS_64BIT

       WLS_MEM_ARGS_32BIT="-Xms512m -Xmx512m"

       export WLS_MEM_ARGS_32BIT

fi

 

MEM_PERM_SIZE_64BIT="-XX:PermSize=128m"

MEM_PERM_SIZE_32BIT="-XX:PermSize=48m"

 

MEM_MAX_PERM_SIZE_64BIT="-XX:MaxPermSize=256m"

MEM_MAX_PERM_SIZE_32BIT="-XX:MaxPermSize=128m"

 

MEM_ARGS="-Xms512m -Xmx1536m -XX:PermSize=256m -XX:MaxPermSize=512m -XX:CompileThreshold=8000 -Dorg.owasp.esapi.resources=/home/ap/nc65/ierp/bin/esapi  -Dfile.encoding=utf-8"

 

修改後：

if [ "${JAVA_VENDOR}" = "Sun" ] ; then

       WLS_MEM_ARGS_64BIT="-Xms4096m -Xmx4096m"

       export WLS_MEM_ARGS_64BIT

       WLS_MEM_ARGS_32BIT="-Xms1024m -Xmx1024m"

       export WLS_MEM_ARGS_32BIT

else

       WLS_MEM_ARGS_64BIT="-Xms4096m -Xmx4096m"

       export WLS_MEM_ARGS_64BIT

       WLS_MEM_ARGS_32BIT="-Xms1024m -Xmx1024m"

       export WLS_MEM_ARGS_32BIT

fi

 

MEM_PERM_SIZE_64BIT="-XX:PermSize=1024m"

MEM_PERM_SIZE_32BIT="-XX:PermSize=512m"

 

MEM_MAX_PERM_SIZE_64BIT="-XX:MaxPermSize=2048m"

MEM_MAX_PERM_SIZE_32BIT="-XX:MaxPermSize=1024m"

 

MEM_ARGS="-Xms4096m -Xmx4096m -XX:PermSize=1024m -XX:MaxPermSize=2048m -XX:CompileThreshold=8000 -Dorg.owasp.esapi.resources=/home/ap/nc65/ierp/bin/esapi  -Dfile.encoding=utf-8"

 

修改StuckThreadMaxTime參數

config.xml 配置文件

修改全部 server下stuck-thread-max-time 參數

修改前：

<stuck-thread-max-time>600</stuck-thread-max-time>

修改後：

<stuck-thread-max-time>60000</stuck-thread-max-time>

 

增大線程數（64位操做系統 3000 ~ 5000 左右）

在全部server中增長參數

<self-tuning-thread-pool-size-min>2000</self-tuning-thread-pool-size-min>

         <self-tuning-thread-pool-size-max>4000</self-tuning-thread-pool-size-max>