【高併發優化實踐】10倍請求壓力來襲，你的系統會被擊垮嗎？【石杉的架構筆記】

歡迎關注微信公衆號：石杉的架構筆記（id：shishan100）

個人新課**《C2C 電商系統微服務架構120天實戰訓練營》在公衆號儒猿技術窩**上線了，感興趣的同窗，能夠點擊下方連接瞭解詳情：

《C2C 電商系統微服務架構120天實戰訓練營》

「 上篇文章一次 JVM FullGC的背後，竟隱藏着驚心動魄的線上生產事故！，給你們講了一個線上系統由於JVM FullGC異常宕機的case。
這篇文章，咱們繼續給你們聊聊另一個線上系統在生產環境遇到的問題。

1、背景介紹

背景狀況是這樣：線上一個系統，在某次高峯期間MQ中間件故障的狀況下，觸發了降級機制，結果降級機制觸發以後運行了一小會兒，忽然系統就徹底卡死，沒法響應任何請求。

給你們簡單介紹一下這個系統的總體架構，這個系統簡單來講就是有一個很是核心的行爲，就是往MQ裏寫入數據，可是這個往MQ裏寫入的數據是很是核心及關鍵的，絕對不允許有丟失。

因此最初就設計了一個降級機制，若是一旦MQ中間件故障，那麼這個系統立馬就會把核心數據寫入本地磁盤文件。

額外提一句，若是有同窗不太清楚MQ中間件的概念，建議看一下以前發的一篇文章「Java進階面試系列之一」大家系統架構中爲什麼要引入消息中間件？，先對MQ中間件這個東西作一個基本的瞭解。

可是若是說在高峯期併發量比較高的狀況下，接收到一條數據立馬同步寫本地磁盤文件，這個性能絕對是極其差的，會致使系統自身的吞吐量瞬間大幅度降低，這個降級機制是絕對沒法在生產環境運行的，由於本身就會被高併發請求壓垮。

所以當時設計的時候，對降級機制進行了一番精心的設計。

咱們的核心思路是一旦MQ中間件故障，觸發降級機制以後，系統接收到一條請求不是立馬寫本地磁盤，而是採用內存雙緩衝 + 批量刷磁盤的機制。

簡單來講，系統接收到一條消息就會立馬寫內存緩衝，而後開啓一個後臺線程把內存緩衝的數據刷新到磁盤上去。

整個過程，你們看看下面的圖，就知道了。

這個內存緩衝實際在設計的時候，分爲了兩個區域。

一個是current區域，用來供系統寫入數據，另一個是ready區域，用來供後臺線程刷新數據到磁盤裏去。

每一塊內存區域設置的緩衝大小是512kb，系統接收到請求就寫current緩衝區，可是current緩衝區總共就512kb的內存空間，所以必定會寫滿。

一樣，你們結合下面的圖，一塊兒來看看。

current緩衝區寫滿以後，就會交換current緩衝區和ready緩衝區。交換事後，ready緩衝區承載了以前寫滿的512kb的數據。

而後current緩衝區此時是空的，能夠繼續接着系統繼續將新來的數據寫入交換後的新的current緩衝區。

整個過程以下圖所示：

此時，後臺線程就能夠將ready緩衝區中的數據經過Java NIO的API，直接高性能append方式的寫入到本地磁盤文件裏。

固然，這裏後臺線程會有一整套完善的機制，好比說一個磁盤文件有固定大小，若是達到了必定大小，自動開啓一個新的磁盤文件來寫入數據。

2、埋下隱患

好！經過上面一套機制，即便是高峯期，也能順利的抗住高併發的請求，一切看起來都很美好！

可是，當時這個降級機制在開發時，咱們採起的思路，爲後面埋下了隱患！

當時採起的思路是：若是current緩衝區寫滿了以後，全部的線程所有陷入一個while循環無限等待。

等到何時呢？一直須要等到ready緩衝區的數據被刷到磁盤文件以後，清空掉ready緩衝區，而後跟current緩衝區進行交換。

這樣current緩衝區要再次變爲空的緩衝區，纔可讓工做線程繼續寫入數據。

可是你們有沒有考慮過一個異常的狀況有可能會發生？

就是後臺線程刷新ready緩衝區的數據到磁盤文件，實際上也是須要一點時間的。

萬一在他刷新數據到磁盤文件的過程當中，current緩衝區忽然也被寫滿了呢？

此時就會致使系統的全部工做線程沒法寫入current緩衝區，線程所有卡死。

給你們上一張圖，看看這個問題！

這個就是系統的降級機制的雙緩衝機制最根本的問題了，在開發好這套降級機制以後，採用正常的請求壓力測試過，發現兩塊緩衝區在設置爲512kb的狀況下，運做良好，沒有什麼問題。

3、高峯請求，問題爆發

可是問題就出在高峯期上了。某一次高峯期，系統請求壓力達到了平時的10倍以上。

固然正常流程下，高峯期的時候，寫請求其實也是直接所有寫到MQ中間件集羣去的，因此哪怕你高峯期流量增長10倍也無所謂，MQ集羣是能夠自然抗高併發的。

可是當時不幸的是，在高峯期的時候，MQ中間件集羣忽然臨時故障，這也是一年遇不到幾回的。

這就致使這個系統忽然觸發了降級機制，而後就開始寫入數據到內存雙緩衝裏面去。

要知道，此時是高峯期啊，請求量是平時正常的10倍！所以10倍的請求壓力瞬間致使了一個問題的發生。

這個問題就是瞬時涌入的高併發請求一下將current緩衝區寫滿，而後兩個緩衝區交換，後臺線程開始刷新ready緩衝區的數據到磁盤文件裏去。

結果由於高峯期請求涌入過快，致使ready緩衝區的數據還沒來得及刷新到磁盤文件，此時current緩衝區又忽然寫滿了。。。

這就尷尬了，線上系統瞬間開始出現異常。。。

典型的表現就是，全部機器上部署的實例所有線程都卡死，處於wait的狀態。

4、定位問題，對症下藥

因而，這套系統開始在高峯期沒法響應任何請求。後來通過線上故障緊急排查、定位和搶修，才解決了這個問題。

其實說來解決方法也很簡單，咱們經過jvm dump出來快照進行分析，查看系統的線程具體是卡在哪一個環節，而後發現大量線程卡死在等待current緩衝區的地方。

這就很明顯知道緣由了，解決方法就是對線上系統擴容雙段緩衝的大小，從512kb擴容到一個緩衝區10mb。

這樣在線上高峯期的狀況下，也能夠穩穩的讓降級機制的雙緩衝機制流暢的運行，不會說瞬間高峯涌入的請求打滿兩塊緩衝區。

由於緩衝區越大，就可讓ready緩衝區被flush到磁盤文件的過程當中，current緩衝區沒那麼快被打滿。

可是這個線上故障反饋出來的一個教訓，就是對系統設計和開發的任何較爲複雜的機制，都必需要參照線上高峯期的最大流量來壓力測試。只有這樣，才能確保任何在系統上線的複雜機制能夠經得起線上高峯期的流量的考驗。

End

若有收穫，請幫忙轉發，您的鼓勵是做者最大的動力，謝謝！

一大波微服務、分佈式、高併發、高可用的原創系列文章正在路上

歡迎掃描下方二維碼，持續關注：

石杉的架構筆記（id:shishan100）

十餘年BAT架構經驗傾囊相授

**> **推薦閱讀：** > > 一、[拜託！面試請不要再問我Spring Cloud底層原理](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5be13b83f265da6116393fc7) > > 二、[【雙11狂歡的背後】微服務註冊中心如何承載大型系統的千萬級訪問？](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5be3f8dcf265da613a5382ca) > > 三、[【性能優化之道】每秒上萬併發下的Spring Cloud參數優化實戰](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5be83e166fb9a049a7115580) > > 四、[微服務架構如何保障雙11狂歡下的99.99%高可用](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5be99a68e51d4511a8090440) > > 五、[兄弟，用大白話告訴你小白都能聽懂的Hadoop架構原理](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5beaf02ce51d457e90196069) > > 六、[大規模集羣下Hadoop NameNode如何承載每秒上千次的高併發訪問](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5bec278c5188253e64332c76) > > 七、【[性能優化的祕密】Hadoop如何將TB級大文件的上傳性能優化上百倍](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5bed82a9e51d450f9461cfc7) > > [八、](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Fjuejin.im%2525252525252Fpost%2525252525252F5bf2c6b6e51d456693549af4)[拜託，面試請不要再問我TCC分佈式事務的實現原理坑爹呀！](https://juejin.cn/post/6844903716089233416) [](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Fjuejin.im%2525252525252Fpost%2525252525252F5bf2c6b6e51d456693549af4) > > 九、[【坑爹呀！】最終一致性分佈式事務如何保障實際生產中99.99%高可用？](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Fjuejin.im%2525252525252Fpost%2525252525252F5bf2c6b6e51d456693549af4) > > 十、[拜託，面試請不要再問我Redis分佈式鎖的實現原理！](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Fjuejin.im%2525252525252Fpost%2525252525252F5bf3f15851882526a643e207) > > **十一、****[【眼前一亮！】看Hadoop底層算法如何優雅的將大規模集羣性能提高10倍以上？](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Fjuejin.im%2525252525252Fpost%2525252525252F5bf5396f51882509a768067e)** > > **十二、****[億級流量系統架構之如何支撐百億級數據的存儲與計算](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Fjuejin.im%25252525252Fpost%25252525252F5bfab59fe51d4551584c7bcf)** > > 1三、[億級流量系統架構之如何設計高容錯分佈式計算系統](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Fjuejin.im%252525252Fpost%252525252F5bfbeeb9f265da61407e9679) > > 1四、[億級流量系統架構之如何設計承載百億流量的高性能架構](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Fjuejin.im%2525252Fpost%2525252F5bfd2df1e51d4574b133dd3a) > > 1五、[億級流量系統架構之如何設計每秒十萬查詢的高併發架構](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Fjuejin.im%25252Fpost%25252F5bfe771251882509a7681b3a) > > 1六、[億級流量系統架構之如何設計全鏈路99.99%高可用架構](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Fjuejin.im%252Fpost%252F5bffab686fb9a04a102f0022) > > 1七、[七張圖完全講清楚ZooKeeper分佈式鎖的實現原理](https://link.juejin.im/?target=https%3A%2F%2Fjuejin.im%2Fpost%2F5c01532ef265da61362232ed) > > 1八、[大白話聊聊Java併發面試問題之volatile究竟是什麼？](https://juejin.cn/post/6844903730303746061) > > 1九、[大白話聊聊Java併發面試問題之Java 8如何優化CAS性能？](https://juejin.cn/post/6844903731234865160) > > 20、[大白話聊聊Java併發面試問題之談談你對AQS的理解？](https://juejin.cn/post/6844903732061159437) > > 2一、[大白話聊聊Java併發面試問題之公平鎖與非公平鎖是啥？](https://juejin.cn/post/6844903732883226637) > > 2二、[大白話聊聊Java併發面試問題之微服務註冊中心的讀寫鎖優化](https://juejin.cn/post/6844903734267510798) > > 2三、[互聯網公司的面試官是如何360°無死角考察候選人的？（上篇）](https://juejin.cn/post/6844903734930046989) > > 2四、[互聯網公司面試官是如何360°無死角考察候選人的？（下篇）](https://juejin.cn/post/6844903735655661581) > > 2五、[Java進階面試系列之一：哥們，大家的系統架構中爲何要引入消息中間件？](https://juejin.cn/post/6844903736444207117) > > 2六、[【Java進階面試系列之二】：哥們，那你說說系統架構引入消息中間件有什麼缺點？](https://juejin.cn/post/6844903737123667975) > > 2七、[【行走的Offer收割機】記一位朋友斬獲BAT技術專家Offer的面試經歷](https://juejin.cn/post/6844903741213130765) > > 2八、[【Java進階面試系列之三】哥們，消息中間件在大家項目裏是如何落地的？](https://juejin.cn/post/6844903742114906125) > > 2九、[【Java進階面試系列之四】扎心！線上服務宕機時，如何保證數據100%不丟失？](https://juejin.cn/post/6844903742928601095) > > 30、[一次JVM FullGC的背後，竟隱藏着驚心動魄的線上生產事故！](https://juejin.cn/post/6844903743712935944)**