關於一次Weblogic活動線程的問題處理

時間 2019-11-16

原文原文鏈接

Weblogic控制檯監控發現環境>>服務器>>你的服務器>>監控>>線程中活動執行線程居然是2000多。同一套系統在另外一套平臺上，而且訪問的人很多，也沒有超過100。重啓此應用，活動進程依然沒有變化，真是奇怪。java

查看轉儲線程堆：linux

==== FULL THREAD DUMP===============

            Tue Aug 12 10:54:48 2014

            Oracle JRockit(R)R28.1.0-123-138454-1.6.0_20-20101014-1350-linux-x86_64

            "Main Thread" id=1idx=0x4 tid=12399 prio=5 alive, waiting, native_blocked

                -- Waiting for notification on:weblogic/t3/srvr/T3Srvr@0xc70a6538[fat lock]

                atjrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                atjava/lang/Object.wait(J)V(Native Method)

                atjava/lang/Object.wait(Object.java:485)

                atweblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:981)

                ^-- Lock released whilewaiting: weblogic/t3/srvr/T3Srvr@0xc70a6538[fat lock]

                atweblogic/t3/srvr/T3Srvr.run(T3Srvr.java:490)

                atweblogic/Server.main(Server.java:71)

                atjrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

            "(Signal Handler)" id=2idx=0x8 tid=12400 prio=5 alive, native_blocked, daemon

            "(OC Main Thread)" id=3idx=0xc tid=12401 prio=5 alive, native_waiting, daemon

            "(GC Worker Thread 1)" id=?idx=0x10 tid=12402 prio=5 alive, daemon

            "(GC Worker Thread 2)"id=? idx=0x14 tid=12403 prio=5 alive, daemon

            "(GC Worker Thread 3)"id=? idx=0x18 tid=12404 prio=5 alive, daemon

            "(GC Worker Thread 4)"id=? idx=0x1c tid=12405 prio=5 alive, daemon

            "(Code Generation Thread1)" id=4 idx=0x20 tid=12406 prio=5 alive, native_waiting, daemon

            "(Code Optimization Thread1)" id=5 idx=0x24 tid=12407 prio=5 alive, native_waiting, daemon

            "(VM Periodic Task)" id=6idx=0x28 tid=12408 prio=10 alive, native_blocked, daemon

            "Finalizer" id=7 idx=0x2ctid=12409 prio=8 alive, native_waiting, daemon

                atjrockit/memory/Finalizer.waitForFinalizees(J[Ljava/lang/Object;)I(NativeMethod)

                atjrockit/memory/Finalizer.access$700(Finalizer.java:12)

                atjrockit/memory/Finalizer$4.run(Finalizer.java:189)

                atjava/lang/Thread.run(Thread.java:619)

                atjrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

            "Reference Handler" id=8idx=0x30 tid=12410 prio=10 alive, native_waiting, daemon

                atjava/lang/ref/Reference.waitForActivatedQueue(J)Ljava/lang/ref/Reference;(NativeMethod)

                atjava/lang/ref/Reference.access$100(Reference.java:11)

                atjava/lang/ref/Reference$ReferenceHandler.run(Reference.java:82)

                atjrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

            "(Sensor Event Thread)"id=9 idx=0x34 tid=12411 prio=5 alive, native_blocked, daemon

            "VM JFR Buffer Thread"id=10 idx=0x38 tid=12412 prio=5 alive, in native, daemon

            "Timer-0" id=13 idx=0x3ctid=12415 prio=5 alive, waiting, native_blocked, daemon

                -- Waiting for notification on:java/util/TaskQueue@0xc504e198[fat lock]

                atjrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                at java/lang/Object.wait(J)V(NativeMethod)

                atjava/lang/Object.wait(Object.java:485)

                atjava/util/TimerThread.mainLoop(Timer.java:483)

                ^-- Lock released whilewaiting: java/util/TaskQueue@0xc504e198[fat lock]

                atjava/util/TimerThread.run(Timer.java:462)

                atjrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

            "Timer-1" id=14 idx=0x40 tid=12416prio=5 alive, waiting, native_blocked, daemon

                -- Waiting for notification on:java/util/TaskQueue@0xc504e548[fat lock]

                atjrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                atjava/lang/Object.wait(J)V(Native Method)

                atjava/util/TimerThread.mainLoop(Timer.java:509)

                ^-- Lock released whilewaiting: java/util/TaskQueue@0xc504e548[fat lock]

                atjava/util/TimerThread.run(Timer.java:462)

                atjrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

            "[ACTIVE] ExecuteThread: '0'for queue: 'weblogic.kernel.Default (self-tuning)'" id=15 idx=0x44tid=12417 prio=5 alive, waiting, native_blocked, daemon

                -- Waiting for notification on:weblogic/work/ExecuteThread@0xc6e748b0[fat lock]

                at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(NativeMethod)

                atjava/lang/Object.wait(J)V(Native Method)

                atjava/lang/Object.wait(Object.java:485)

                atweblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)

                ^-- Lock released whilewaiting: weblogic/work/ExecuteThread@0xc6e748b0[fat lock]

                atweblogic/work/ExecuteThread.run(ExecuteThread.java:183)

                atjrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

            "[ACTIVE] ExecuteThread: '1'for queue: 'weblogic.kernel.Default (self-tuning)'" id=16 idx=0x48tid=12418 prio=5 alive, waiting, native_blocked, daemon

                -- Waiting for notification on:weblogic/work/ExecuteThread@0xc6d42d00[fat lock]

                atjrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                atjava/lang/Object.wait(J)V(Native Method)

                atjava/lang/Object.wait(Object.java:485)

                atweblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)

                ^-- Lock released while waiting:weblogic/work/ExecuteThread@0xc6d42d00[fat lock]

                atweblogic/work/ExecuteThread.run(ExecuteThread.java:183)

                atjrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

            "[ACTIVE] ExecuteThread: '2'for queue: 'weblogic.kernel.Default (self-tuning)'" id=17 idx=0x4ctid=12419 prio=5 alive, waiting, native_blocked, daemon

                -- Waiting for notification on:weblogic/work/ExecuteThread@0xc6e1c620[fat lock]

                atjrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                atjava/lang/Object.wait(J)V(Native Method)

                atjava/lang/Object.wait(Object.java:485)

                atweblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)

                ^-- Lock released whilewaiting: weblogic/work/ExecuteThread@0xc6e1c620[fat lock]

                atweblogic/work/ExecuteThread.run(ExecuteThread.java:183)

                atjrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

…………此處省略大量重複的信息web

在下對weblogic瞭解甚少，對java開發也是丈二和尚，身邊也沒有懂weblogic的人，看來只能藉助網絡搜索了。百度、ITpub、CSDN全試了都沒有相關問題。後來加了幾個weblogic的qq羣，在羣裏發問基本也是沒人管。哎，這世界怎麼就沒有個好心人呢？不過好心人最終仍是出現了，他直接就找到了問題所在——這個問題的緣由就在啓動參數上。（就是啊，從新啓動都不行，那十有八九是啓動參數的問題啊，我這笨腦子！）服務器

形成這種狀況的參數在weblogic\user_projects\domains\base_domain\bin下的setDomainEnvNaNd中的Dweblogic.threadpool.MinPoolSize。個人配置是這樣的：網絡

JAVA_OPTIONS="${JAVA_OPTIONS}${JAVA_PROPERTIES} -Dwlw.iterativeDev=${iterativeDevFlag}-Dwlw.testConsole=${testConsoleFlag} -Dweblogic.threadpool.MinPoolSize=2000 -Dweblogic.threadpool.MaxPoolSize=4000-Dwlw.logErrorsToConsole=${logErrorsToConsoleFlag}"多線程

exportJAVA_OPTIONSdom

這裏的-Dweblogic.threadpool.MinPoolSize=2000意思是默認線程池大小，這個參數設置多少合適我也不知道，具體設置於不設置有什麼大的區別也不知道……額，反正是它在搞鬼。我將此參數去掉以後，從新啓動，活動線程變少了。oop

網抄一些weblogic的知識，留做備查——————優化

檢查線程數spa

經過weblogic控制檯能夠查看線程數的統計信息。weblogic9及以上的線程是自優化的。但應該查看系統的線程最大數是否過大，若是過大，就要注意系統爲何會有這麼大的壓力。以下爲示例截圖

對應中文翻譯：

ActiveExecute Threads：在活動的線程池內處理請求的線程個數

ExecuteThread Total Count：線程池內線程的總數

ExecuteThread Idle Count：池內的空閒線程數。它不包含stuck和standby的線程數。它是指等待接收新請求到來並處理的線程個數

queuelength ：在等待隊列裏的請求數，一般保留默認值 65536 ，隊列長度代表了同時發來請求的最大數， 65536 個請求是個很大的數，即便達到這個最大數，也是不多見的。若是達到最大隊列長度，WebLogic 會自動成倍增加隊列大小，以處理額外的工做。注意：超過 65536 個請求預示隊列中的線程有問題，不單單只是隊列自己的長度問題，實踐代表在隊列中有堵塞線程或線程數不足的狀況存在。

hoggingthread count ：線程處理一個請求時間超過必定值被視爲hogging狀態，若是繼續處理請求超過必定時間將被視爲stuck，或處理完請求後被放回線程池

standbythread count ：統計在standby（備用）線程池內的線程數。這些線程不須要處理當前請求被放入standby池內，當活動的線程池內須要更多線程時，這些線程將被激活。

Execute Thread Total Count= Active Execute Threads+ standbythread count