tomcat 6.0.44 「has failed to stop it. This is very likely to create a memory leak」問題調查

時間 2019-11-22

標籤 tomcat 6.0.44 failed stop likely create memory leak 問題調查欄目 Tomcat 简体版

原文原文鏈接

1. 問題原由java

咱們項目中緩存模塊某個實現採用了ehcache(2.4.3)，當項目部署到tomcat中後，對tomcat作中止服務操做(點擊eclipse的console紅色的中止按鈕，奇怪的是有小几率的狀況不能復現這個問題？？)，發現tomcat不能正常中止，報錯 appears to have started a thread named [xxx] but has failed to stop it. This is very likely to create a memory leak.java進程不能正常結束，須要手動kill進程。mysql

信息: Stopping service Catalina
2015-9-26 11:40:17 org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
嚴重: The web application [/hd] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
2015-9-26 11:40:17 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
嚴重: The web application [/hd] appears to have started a thread named [pool-2-thread-1] but has failed to stop it. This is very likely to create a memory leak.
2015-9-26 11:40:17 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
嚴重: The web application [/hd] appears to have started a thread named [net.sf.ehcache.CacheManager@268d15] but has failed to stop it. This is very likely to create a memory leak.
2015-9-26 11:40:17 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
嚴重: The web application [/hd] appears to have started a thread named [org.hibernate.cache.spi.UpdateTimestampsCache.data] but has failed to stop it. This is very likely to create a memory leak.
2015-9-26 11:40:17 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
嚴重: The web application [/hd] appears to have started a thread named [org.hibernate.cache.internal.StandardQueryCache.data] but has failed to stop it. This is very likely to create a memory leak.
2015-9-26 11:40:17 org.apache.coyote.http11.Http11Protocol destroy
信息: Stopping Coyote HTTP/1.1 on http-8080

2. ehcache 線程不會自動中止問題web

分析後，發現是ehcache的三個線程不能自動中止(或者說ehcache未在應用中止/jvm中止的時候將其結束)。sql

經分析後發現能夠在web應用中加listener，在應用destory的時候觸發ehcache的釋放資源與線程的調用。apache

3. 必定要加listener嗎？緩存

分析ehcache代碼後發現，使用系統參數-Dnet.sf.ehcache.enableShutdownHook=true能啓用ehcache的jvm shutdown hook。tomcat

net.sf.ehcache.CacheManagerapp

/**
     * Some caches might be persistent, so we want to add a shutdown hook if that is the
     * case, so that the data and index can be written to disk.
     */
    private void addShutdownHookIfRequired() {

        String shutdownHookProperty = System.getProperty(ENABLE_SHUTDOWN_HOOK_PROPERTY);
        boolean enabled = PropertyUtil.parseBoolean(shutdownHookProperty);
        if (!enabled) {
            return;
        } else {
            LOG.info("The CacheManager shutdown hook is enabled because {} is set to true.", ENABLE_SHUTDOWN_HOOK_PROPERTY);

            Thread localShutdownHook = new Thread() {
                @Override
                public void run() {
                    synchronized (this) {
                        if (status.equals(Status.STATUS_ALIVE)) {
                            // clear shutdown hook reference to prevent
                            // removeShutdownHook to remove it during shutdown
                            shutdownHook = null;
                            LOG.info("VM shutting down with the CacheManager still active. Calling shutdown.");
                            shutdown();
                        }
                    }
                }
            };

            Runtime.getRuntime().addShutdownHook(localShutdownHook);
            shutdownHook = localShutdownHook;
        }
    }

jvm shutdown hook注意點：eclipse

~~tomcat自帶的shutdown.sh/bat不能給java進程發送退出信號，jvm shutdown hook不能被觸發~~ 。待調查webapp

須要使用kill $pid,jvm shutdown hook 才能被觸發

4. 以前的項目用了hibernate3，二級緩存依賴ehcache，爲何沒有這個問題

難道是在什麼地方加了destory？

準備demo，驗證，存在同樣的問題。只是因爲hibernate3(3.2.5ga)依賴較低版本的ehcache(1.2.3)，因此沒有被停掉的線程是hibernate和其餘一些部件的。

日誌以下：

2015-9-26 10:53:15 org.apache.catalina.core.StandardService stop
信息: Stopping service Catalina
2015-9-26 10:53:15 org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
嚴重: The web application [/hd] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
2015-9-26 10:53:15 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
嚴重: The web application [/hd] appears to have started a thread named [Store org.hibernate.cache.UpdateTimestampsCache Spool Thread] but has failed to stop it. This is very likely to create a memory leak.
2015-9-26 10:53:15 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
嚴重: The web application [/hd] appears to have started a thread named [Store org.hibernate.cache.UpdateTimestampsCache Expiry Thread] but has failed to stop it. This is very likely to create a memory leak.
2015-9-26 10:53:15 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
嚴重: The web application [/hd] appears to have started a thread named [Store org.hibernate.cache.StandardQueryCache Spool Thread] but has failed to stop it. This is very likely to create a memory leak.
2015-9-26 10:53:15 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
嚴重: The web application [/hd] appears to have started a thread named [Store org.hibernate.cache.StandardQueryCache Expiry Thread] but has failed to stop it. This is very likely to create a memory leak.
2015-9-26 10:53:15 org.apache.coyote.http11.Http11Protocol destroy
信息: Stopping Coyote HTTP/1.1 on http-8080
2015-9-26 10:53:15 org.apache.catalina.loader.WebappClassLoader loadClass

那之前的項目都上線了，爲何沒這個問題？

猜想是:線上用的weblogic,開發時用tomcat 也應該存在這個問題,只不過點擊eclipse控制檯的中止按鈕並不能重現這個問題

5. tomcat檢查泄露的代碼

org.apache.catalina.loader.WebappClassLoader.clearReferencesThreads()方法中

if (isRequestThread(thread)) {
    log.error(sm.getString("webappClassLoader.warnRequestThread",
            contextName, thread.getName()));
} else {
    log.error(sm.getString("webappClassLoader.warnThread",
            contextName, thread.getName()));
}

6. 延伸閱讀

http://blog.csdn.net/monkeyking1987/article/details/9182201

http://stackoverflow.com/questions/4899205/tomcat-6-memory-leaks-log-entries

http://stackoverflow.com/questions/11872316/tomcat-guice-jdbc-memory-leak

http://stackoverflow.com/questions/5292349/is-this-very-likely-to-create-a-memory-leak-in-tomcat