JVM調優實戰

對項目 i..net 網站進行了接口的壓測，壓測過程當中發現各微服務的 heap 內存設置太小（-Xms64m –Xmx128m），致使壓測過程當中服務假死（頻繁fullgc致使服務假死）和發生OOM。

一、xx接口的壓測狀況以下，150併發進行壓測，發現響應時間在小於1s一段時間後，響應時間陡升到3s。

二、 TPS 也是從420 陡降到 60 左右。

三、trace 調用鏈跟蹤發現 xxxxsevice中的 xxxxWorktime方法調用耗時3S

四、繼續跟蹤該方法的調用鏈，發現調用 attend-xxxx項目中的接口耗時3S

五、查看後臺日誌發現：atten-xxxx 服務發生了 OOM；經過telnet命令鏈接xxxx的端口，發現沒法鏈接端口，服務處於假死狀態。

六、查看該服務的heap使用狀況，基本上被耗盡

七、jstat 發現 發生了 頻繁的 FullGC

八、壓測過程當中發如今進行資源爭奪時，有時還把mongo 給拖死：

ERROR c.y.a.sign.exception.BaseControllerException - attend-xxxx統一異常捕獲...
org.springframework.dao.DataAcce***esourceFailureException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is
Caused by: com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=xx.xx.0.40:47020, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=xx.xx.0.40:47020, type=REPLICA_SET_ARBITER, roundTripTime=0.6 ms, state=CONNECTED}, {address=xx.xx.0.40:47020, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
at com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:377)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:104)
at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.<init>(ClusterBinding.java:75)
at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.<init>(ClusterBinding.java:71)
at com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:411)
at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:133)
at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:60)
at com.mongodb.Mongo.execute(Mongo.java:845)
at com.mongodb.Mongo$2.execute(Mongo.java:828)
at com.mongodb.DBCollection.executeWriteOperation(DBCollection.java:342)
at com.mongodb.DBCollection.insert(DBCollection.java:337)
at com.mongodb.DBCollection.insert(DBCollection.java:328)
at com.mongodb.DBCollection.insert(DBCollection.java:298)
at com.mongodb.DBCollection.insert(DBCollection.java:264)
at com.mongodb.DBCollection.insert(DBCollection.java:186)
at com.mongodb.DBCollection.save(DBCollection.java:407)
at com.mongodb.DBCollection.save(DBCollection.java:382)

九、數據庫在壓測過程當中被壓死：

綜上所述，調優建議以下：

一、 微服務的heap 設置過小了，須要擴容內存，建議設置爲（-Xms512m –Xmx1024m）

二、 數據庫服務服務器建議單獨部署物理機，且數據庫內存建議設置64G以上；建議採用多主從或分片部署。