原文java
今日生產環境發現有些容器中止響應了,可是容器沒有死,docker exec -it <container-name> /bin/bash
也能正常使用。git
在容器內部使用jstack <pid>
發現log4j2的Console Appender一直處於運行狀態:github
"AsyncAppender-asyncConsole" #21 daemon prio=5 os_prio=0 tid=0x00007fd968d07000 nid=0x1f runnable [0x00007fd91bffd000] java.lang.Thread.State: RUNNABLE at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:326) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122) - locked <0x00000000f002b408> (a java.io.BufferedOutputStream) at java.io.PrintStream.write(PrintStream.java:480) - locked <0x00000000f002b3e8> (a java.io.PrintStream) at org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.java:53) at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStreamManager.java:262) - locked <0x00000000f021d848> (a org.apache.logging.log4j.core.appender.OutputStreamManager) at org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager.java:294) - locked <0x00000000f021d848> (a org.apache.logging.log4j.core.appender.OutputStreamManager) at org.apache.logging.log4j.core.appender.OutputStreamManager.drain(OutputStreamManager.java:351) at org.apache.logging.log4j.core.layout.TextEncoderHelper.drainIfByteBufferFull(TextEncoderHelper.java:260) - locked <0x00000000f021d848> (a org.apache.logging.log4j.core.appender.OutputStreamManager) at org.apache.logging.log4j.core.layout.TextEncoderHelper.writeAndEncodeAsMuchAsPossible(TextEncoderHelper.java:199) at org.apache.logging.log4j.core.layout.TextEncoderHelper.encodeChunkedText(TextEncoderHelper.java:159) - locked <0x00000000f021d848> (a org.apache.logging.log4j.core.appender.OutputStreamManager) at org.apache.logging.log4j.core.layout.TextEncoderHelper.encodeText(TextEncoderHelper.java:58) at org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:68) at org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:32) at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:220) at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:58) at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:177) at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:170) at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:161) at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156) at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129) at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120) at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84) at org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.callAppenders(AsyncAppender.java:459) at org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.run(AsyncAppender.java:412)
但用docker logs -f <container-name>
沒有發現有新的日誌輸出,且訪問該應用確定會輸出日誌的接口也是沒有任何日誌輸出,所以懷疑log4j2阻塞住了。docker
Google到有人在log4j提出了相似了問題LOG4J2-2239,官方給出的解釋是問題出在log4j2以外。apache
因而查一下logback是否也有相似問題,找到LOGBACK-1422,一樣給出的解釋是問題出在logback以外。bash
兩個問題的共通點都是用docker運行,因而把應用直接進程方式運行,沒有出現問題。app
因而Google搜索docker logging to stdout hangs
,找到SO的這個回答,以及這個issue,解決方案將Docker升級到18.06。async
查看生產環境的docker版本是18.03,升級到18.09後問題解決。post