前幾天公司後端系統出現了故障,致使app多個功能沒法使用,查看日誌,發現日誌出現較多的redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool的異常信息,顯而易見,jedis/redis出現了問題。由於是connection的相關的問題,因此看了一下jedis和鏈接數相關的配置項,maxIdle和maxTotal都是200,jedis的封裝也在finally中釋放了connection,因此初步猜想問題發生在redis服務端redis
1.jedis機器-->ping-->redis機器,毫秒級的響應時間----網絡暢通後端
2.使用netstat -apn |grep redis-server鏈接數爲20多個--網絡鏈接數正常網絡
3.free -m內存使用率60%---(表面上)內存夠用app
4.df -h磁盤使用率15%---磁盤空間充足this
5.使用redis-cli,執行info命令,client部分:日誌
#Clientsserver
connected_clients:18進程
client_longest_output_list:0內存
client_biggest_input_buf:0get
blocked_clients:0
clients數量也正常
6.使用redis-cli,執行ping命令,異常信息出來了:
(error)MISCONF Redis is configured to save RDB snapshots, but is currently
not able to persist on disk. Commands that may modify the data set
are disabled. Please check Redis logs for details about the error.
而後查看redis日誌,出現了
WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
問題已經很清晰了,bgsave會fork一個子進程,由於vm.overcommit_memory = 0,因此申請的內存大小和父進程的同樣,因爲redis已經使用了60%的內存空間,因此fork失敗
解決辦法:
/etc/sysctl.conf 添加 vm.overcommit_memory=1
sysctl vm.overcommit_memory=1