一次永久代泄漏（perm泄漏）排查
寫在前面
網上關於perm區泄露的文章比較少，特別是對於動態類加載方面問題的分析比較少，在此記錄下。html
perm區問題通常兩種解決方案：java
- 啓動時perm區問題，通常修改PermSize,MaxPermSize參數便可
- 運行時動態生成類加載問題，這種問題比較難搞，須要關心動態加載了哪些類
週末早晨收到幾臺機器告警(fullgc告警)(perm大於90%告警)，爲快速解決問題，先把幾臺機器重啓解決，留下了一臺禁用端口保留現場進行問題分析。平時上線發版比較頻繁，發版後jvm回收，若是較長時間沒有發版可能會形成泄露，收到告警。python
吃完早飯後登上機器進行排查。git
排查過程
登入機器，查看內存使用高的進程：github
topweb
pid:15298apache
既然是perm區問題，查看永久代狀況：json
jmap -permstat pid > 15298dump.permstatbootstrap
class_loader classes bytes parent_loader alive? type <bootstrap> 3630 21866152 null live <internal> 0x0000000705e4df00 1 2008 0x00000006c002e450 dead sun/reflect/DelegatingClassLoader@0x00000007c0050c30 0x00000007282e1040 4 23480 0x00000006c002e450 dead com/facebook/swift/codec/internal/compiler/DynamicClassLoader@0x00000007c377f040
經過awk統計type類型，查看加載了哪一種類型的類：swift
awk '{ arr[$6]+=$3 } END { for (key in arr) printf("%s\t%s\n", key, arr[key]) }' 15298dump.permstat | sort -k2,2
ava/net/URLClassLoader@0x00000007c01c1598 0 java/util/ResourceBundle$RBClassLoader@0x00000007c0337380 0 type 0 sun/reflect/DelegatingClassLoader@0x00000007c0050c30 11361872 sun/reflect/misc/MethodUtil@0x00000007c777bf60 134528 <internal> 21866152 com/alibaba/fastjson/util/ASMClassLoader@0x00000007c44de0e0 4785232 N/A 5146 sun/misc/Launcher$AppClassLoader@0x00000007c021c2d8 5423216 com/facebook/swift/codec/internal/compiler/DynamicClassLoader@0x00000007c377f040 5488288 sun/misc/Launcher$ExtClassLoader@0x00000007c01c1978 615216 org/eclipse/jetty/webapp/WebAppClassLoader@0x00000007c06e0da0 738107040
WebAppClassLoader加載最多，達到了738107040，近738m。
知道了加載的類最多，怎麼分析具體加載了哪一種類呢？
以前通常經過Java -verbose查看啓動類加載過程。
對於運行時貌似有兩種方式：
- 基於ClassFilter寫一個攔截器，dump特定的類，使用SA的jar($JAVA_HOME/lib/sa-jdi.jar)編譯好類，在編譯好的類目錄下調用下面的命令進行dump
- 問了(政威老師)，準備試試arthas，https://alibaba.github.io/arthas/classloader.html
機器上安裝好arthas-boot.jar，並啓動：
java -jar arthas-boot.jar
選擇進程，執行類後臺導出，便後續分析：
classloader -a >> &
打開導出文件：
hash:null, BootstrapClassLoader [B [C [D [F [I [J [Lcom.sun.activation.registries.MimeTypeFile; [Lcom.sun.imageio.plugins.jpeg.DHTMarkerSegment$Htable; [Lcom.sun.imageio.plugins.jpeg.ImageTypeProducer; [Lcom.sun.imageio.plugins.jpeg.JPEGImageReader$CallBackLock$State; [Lcom.sun.imageio.plugins.jpeg.JPEGImageWriter$CallBackLock$State; [Lcom.sun.imageio.plugins.jpeg.SOFMarkerSegment$ComponentSpec; [Lcom.sun.imageio.plugins.jpeg.SOSMarkerSegment$ScanComponentSpec; [Lcom.sun.jmx.mbeanserver.ClassLoaderRepositorySupport$LoaderEntry; [Lcom.sun.jmx.mbeanserver.MXBeanMapping; [Lcom.sun.org.apache.xalan.internal.utils.FeatureManager$Feature; [Lcom.sun.org.apache.xalan.internal.utils.FeaturePropertyBase$State; [Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityManager$Limit; [Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityManager$NameMap; [Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityManager$State; [Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityPropertyManager$Property; [Lcom.sun.org.apache.xerces.internal.impl.XMLEntityManager$CharacterBuffer; [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.CMLeaf; [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.CMNode; [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.CMStateSet; [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.ContentModelValidator; [Lcom.sun.org.apache.xerces.internal.impl.dv.DatatypeValidator; [Lcom.sun.org.apache.xerces.internal.impl.dv.XSSimpleType; [Lcom.sun.org.apache.xerces.internal.impl.dv.xs.AbstractDateTimeDV$DateTimeData; [Lcom.sun.org.apache.xerces.internal.impl.dv.xs.TypeValidator; [Lcom.sun.org.apache.xerces.internal.impl.dv.xs.XSSimpleTypeDecl; [Lcom.sun.org.apache.xerces.internal.impl.xpath.regex.RegularExpression; [Lcom.sun.org.apache.xerces.internal.impl.xs.SchemaGrammar; [Lcom.sun.org.apache.xerces.internal.impl.xs.SubstitutionGroupHandler$OneSubGroup; [Lcom.sun.org.apache.xerces.internal.impl.xs.XSAnnotationImpl; [Lcom.sun.org.apache.xerces.internal.impl.xs.XSAttributeUseImpl; [Lcom.sun.org.apache.xerces.internal.impl.xs.XSComplexTypeDecl; [Lcom.sun.org.apache.xerces.internal.impl.xs.XSElementDecl; [Lcom.sun.org.apache.xerces.internal.impl.xs.XSGroupDecl; [Lcom.sun.org.apache.xerces.internal.impl.xs.XSNotationDecl; [Lcom.sun.org.apache.xerces.internal.impl.xs.XSParticleDecl; [Lcom.sun.org.apache.xerces.internal.impl.xs.identity.IdentityConstraint; [Lcom.sun.org.apache.xerces.internal.impl.xs.identity.XPathMatcher; [Lcom.sun.org.apache.xerces.internal.impl.xs.models.XSCMLeaf; [Lcom.sun.org.apache.xerces.internal.impl.xs.models.XSCMValidator; [Lcom.sun.org.apache.xerces.internal.impl.xs.opti.DefaultNode; [Lcom.sun.org.apache.xerces.internal.impl.xs.opti.NodeImpl; [Lcom.sun.org.apache.xerces.internal.impl.xs.traversers.OneAttr; [Lcom.sun.org.apache.xerces.internal.impl.xs.traversers.XSDocumentInfo; [Lcom.sun.org.apache.xerces.internal.impl.xs.util.SimpleLocator; [Lcom.sun.org.apache.xerces.internal.impl.xs.util.XInt; [Lcom.sun.org.apache.xerces.internal.util.Status; [Lcom.sun.org.apache.xerces.internal.util.SymbolHash$Entry; [Lcom.sun.org.apache.xerces.internal.util.SymbolTable$Entry; [Lcom.sun.org.apache.xerces.internal.util.XMLAttributesImpl$Attribute; [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityManager$Limit; [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityManager$NameMap; [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityManager$State; [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityPropertyManager$Property; [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityPropertyManager$State; [Lcom.sun.org.apache.xerces.internal.xni.QName; [Lcom.sun.org.apache.xerces.internal.xni.XMLLocator; [Lcom.sun.org.apache.xerces.internal.xni.grammars.Grammar; [Lcom.sun.org.apache.xerces.internal.xni.grammars.XSGrammar; [Lcom.sun.org.apache.xerces.internal.xs.ShortList; [Lcom.sun.org.apache.xerces.internal.xs.XSAnnotation; [Lcom.sun.org.apache.xerces.internal.xs.XSAttributeUse; [Lcom.sun.org.apache.xerces.internal.xs.XSComplexTypeDefinition; [Lcom.sun.org.apache.xerces.internal.xs.XSElementDeclaration; [Lcom.sun.org.apache.xerces.internal.xs.XSIDCDefinition; [Lcom.sun.org.apache.xerces.internal.xs.XSModelGroupDefinition; [Lcom.sun.org.apache.xerces.internal.xs.XSNamespaceItem; [Lcom.sun.org.apache.xerces.internal.xs.XSNotationDeclaration; [Lcom.sun.org.apache.xerces.internal.xs.XSObject; [Lcom.sun.org.apache.xerces.internal.xs.XSParticle; [Lcom.sun.org.apache.xerces.internal.xs.XSSimpleTypeDefinition; [Lcom.sun.org.apache.xerces.internal.xs.XSTerm; [Lcom.sun.org.apache.xerces.internal.xs.XSTypeDefinition; [Lcom.sun.org.apache.xerces.internal.xs.datatypes.XSDateTime; [Lcom.sun.org.apache.xml.internal.dtm.DTM; [Lcom.sun.org.apache.xml.internal.dtm.DTMAxisTraverser; [Lcom.sun.org.apache.xml.internal.dtm.DTMIterator; [Lcom.sun.org.apache.xml.internal.dtm.ref.ExpandedNameTable$HashEntry; [Lcom.sun.org.apache.xml.internal.dtm.ref.ExtendedType; [Lcom.sun.org.apache.xpath.internal.Expression; [Lcom.sun.org.apache.xpath.internal.ExpressionNode; [Lcom.sun.org.apache.xpath.internal.XPathVisitable; [Lcom.sun.org.apache.xpath.internal.axes.LocPathIterator; [Lcom.sun.org.apache.xpath.internal.axes.PathComponent; [Lcom.sun.org.apache.xpath.internal.axes.PredicatedNodeTest; [Lcom.sun.org.apache.xpath.internal.axes.SubContextList; [Lcom.sun.org.apache.xpath.internal.objects.XObject; [Lcom.sun.org.apache.xpath.internal.patterns.NodeTest; [Lcom.sun.xml.internal.ws.org.objectweb.asm.Item; [Lcom.sun.xml.internal.ws.org.objectweb.asm.Type; [Ljava.awt.AWTKeyStroke; [Ljava.awt.Dimension; [Ljava.awt.Queue; [Ljava.awt.event.ActionListener; [Ljava.awt.event.ComponentListener; [Ljava.awt.event.FocusListener; [Ljava.awt.event.HierarchyBoundsListener; ......
統計最多的類：
#!/usr/bin/python
from collections import Counter
package_name_count_dic = {}
for s in open('14.txt'):
full_qualified_name = s.strip()
if not full_qualified_name:
continue
class_name_index = full_qualified_name.rfind('.')
if class_name_index >= 0:
package_name, class_name = full_qualified_name[:class_name_index], full_qualified_name[class_name_index + 1:]
else:
package_name = class_name = full_qualified_name
if package_name in package_name_count_dic:
package_name_count_dic[package_name] += 1
else:
package_name_count_dic[package_name] = 1
k = Counter(package_name_count_dic)
high = k.most_common(5)
for p, c in high:
print p, c
最多的幾個類：
真兇：ma.glasnost.orika.generated。
查看代碼中誰使用了orika類庫。
發現是閃購同窗，因爲咱們的系統目前對接多方，閃購同窗還在咱們系統作代碼開發，在codereview上存在一些問題。
@Component
public class SGMapperFactory implements FactoryBean<MapperFactory> {
@Override
public MapperFactory getObject() {
return new DefaultMapperFactory.Builder().build();
}
@Override
public Class<?> getObjectType() {
return MapperFactory.class;
}
@Override
public boolean isSingleton() {
return true;
}
}
每次調用getObject都會新建立DefaultMapperFactory對象。MapperGenerator 每次會動態產生類。
至此問題排查，推進閃購同窗改動。
後記
看代碼是3月份代碼寫的，爲什麼到如今才發現問題呢？
首先是平時發版比較頻繁，jvm發版後問題解決。 最近發版比較少，同時這些是擴容機器，在最近的幾回發版中並無發版，因此形成類加載持續一段時間，最後形成永久代泄露。 爲儘早發現問題須要在流程上進行控制，好比增長codereview細緻程度，在灰度發版後對發版機器進行引流壓測，儘早發現問題，解決問題。
相關文章
- 1. 一次永久代泄漏（perm泄漏）排查
- 2. Metaspace泄漏排查
- 3. 一次 Java 內存泄漏的排查
- 4. JS內存泄漏排查
- 5. jvisualvm內存泄漏排查
- 6. Java內存泄漏排查
- 7. 區塊鏈項目：golang內存泄漏，goroutine泄漏排查
- 8. DFT泄漏 頻譜泄漏
- 9. csdn 代碼泄漏
- 10. 一次畢生難忘的 Java 內存泄漏排查經歷
- 更多相關文章...
- • 一對一關聯查詢 - MyBatis教程
- • 一對多關聯查詢 - MyBatis教程
- • 算法總結-二分查找法
- • 算法總結-歸併排序
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
最新文章
- 1. 升級Gradle後報錯Gradle‘s dependency cache may be corrupt (this sometimes occurs
- 2. Smarter, Not Harder
- 3. mac-2019-react-native 本地環境搭建(xcode-11.1和android studio3.5.2中Genymotion2.12.1 和VirtualBox-5.2.34 )
- 4. 查看文件中關鍵字前後幾行的內容
- 5. XXE萌新進階全攻略
- 6. Installation failed due to: ‘Connection refused: connect‘安卓studio端口占用
- 7. zabbix5.0通過agent監控winserve12
- 8. IT行業UI前景、潛力如何？
- 9. Mac Swig 3.0.12 安裝
- 10. Windows上FreeRDP-WebConnect是一個開源HTML5代理，它提供對使用RDP的任何Windows服務器和工作站的Web訪問
歡迎關注本站公眾號,獲取更多信息
相關文章
- 1. 一次永久代泄漏（perm泄漏）排查
- 2. Metaspace泄漏排查
- 3. 一次 Java 內存泄漏的排查
- 4. JS內存泄漏排查
- 5. jvisualvm內存泄漏排查
- 6. Java內存泄漏排查
- 7. 區塊鏈項目：golang內存泄漏，goroutine泄漏排查
- 8. DFT泄漏 頻譜泄漏
- 9. csdn 代碼泄漏
- 10. 一次畢生難忘的 Java 內存泄漏排查經歷