redis報Cannot allocate memory錯誤

昨天16:27開始將dp的日誌使用ELK處理，當時redis使用內存的量不是很大，100M多點，結果今天早上到了一看xshell被關掉了，趕忙將各服務啓動起來，elasticsearch啓動沒有問題，logstash indexer端啓動時報錯，redis插件鏈接不到，建議查看下redis的日誌：

1 A plugin had an unrecoverable error. Will restart this plugin.
2   Plugin: <LogStash::Inputs::Redis host=>"10.132.24.166", type=>"redis-input", data_type=>"list", key=>"logstash", name=>"default">
3   Error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error. {:level=>:error}

覺得是redis掛掉了，查看端口，6379還在的，查看日誌：

redis 頻繁報錯日誌：

1 [7409] 12 Mar 09:57:11.069 * 1 changes in 900 seconds. Saving...
2 [7409] 12 Mar 09:57:11.070 # Can't save in background: fork: Cannot allocate memory
3 [7409] 12 Mar 09:57:17.086 * 1 changes in 900 seconds. Saving...
4 [7409] 12 Mar 09:57:17.086 # Can't save in background: fork: Cannot allocate memory
5 [7409] 12 Mar 09:57:23.100 * 1 changes in 900 seconds. Saving...

top後顯示VIRT：3758M，RES：3.5G

 

參考：http://segmentfault.net/blog/koy/1190000000767465

內核參數調整參考：https://www.kernel.org/doc/Documentation/vm/overcommit-accounting

https://www.kernel.org/doc/Documentation/sysctl/vm.txt

http://blog.sina.com.cn/s/blog_7429b9c801012evk.html