一次線上redis報障處理過程

昨天下午剛到公司，開發告訴我有個頁面打不開。後面通過排查排除代碼的問題，開始懷疑是否是redis出問題了，後面嘗試着重啓了一下redis，結果問題解決了。當時由於redis沒有配日誌文件，因此。。。

 

等到今天早上又出現一樣的問題了，不過依然沒有日誌，因此後面我在服務器裏配置了日誌，而後就不停對坐在我對面的開發說：馬丹，服務器怎麼還不出問題......

 

剛剛（2017-01-12 16：42）終於又出現這個問題了，我本着無比雞凍的眼神看了眼日誌

13718:M 12 Jan 14:21:30.150 # Background saving error
13718:M 12 Jan 14:21:36.064 * 1 changes in 900 seconds. Saving...
13718:M 12 Jan 14:21:36.064 * Background saving started by pid 18396
18396:C 12 Jan 14:21:36.065 # Failed opening .rdb for saving: Permission denied
13718:M 12 Jan 14:21:36.165 # Background saving error
13718:M 12 Jan 14:21:42.078 * 1 changes in 900 seconds. Saving...
13718:M 12 Jan 14:21:42.079 * Background saving started by pid 18398
18398:C 12 Jan 14:21:42.079 # Failed opening .rdb for saving: Permission denied
13718:M 12 Jan 14:21:42.179 # Background saving error
13718:M 12 Jan 14:21:48.092 * 1 changes in 900 seconds. Saving...
13718:M 12 Jan 14:21:48.093 * Background saving started by pid 18399
18399:C 12 Jan 14:21:48.093 # Failed opening .rdb for saving: Permission denied
13718:M 12 Jan 14:21:48.194 # Background saving error
13718:M 12 Jan 14:21:54.006 * 1 changes in 900 seconds. Saving...
13718:M 12 Jan 14:21:54.007 * Background saving started by pid 18400
18400:C 12 Jan 14:21:54.007 # Failed opening .rdb for saving: Permission denied

滿屏滿是permission denied。

 

個人猜測：.rdb是數據文件，說明數據文件沒有寫入權限。可是很奇怪我在redis目錄下沒有找到.rdb文件。而後我用find在全局查找一次。

[root@iZ2zeib1msf1e5n3hae56mZ ~]# find / -name *.rdb
/tmp/dump.rdb
/root/dump.rdb
/root/.ssh/dump.rdb

這裏是否是應該有個疑問？爲何redis的數據目錄不在redis目錄下面呢？因而上網百度，發現百度上面說的方法都治不了個人痛，着實不靠譜，也就沒有測試。

 

後來發現個問題，配置文件下

[root@iZ2zeib1msf1e5n3hae56mZ redis-3.0.7]# sed -n '192p' redis.conf 
dir ./

狀況一：

[root@test ~]# redis-server /data/redis-3.0.7/redis.conf &
[root@test ~]# ls
conf.n    dump.rdb  http:

狀況二：

[root@iZ2zeib1msf1e5n3hae56mZ redis-3.0.7]# ./src/redis-server /data/redis-3.0.7/redis.conf &
[root@iZ2zeib1msf1e5n3hae56mZ redis-3.0.7]# ls
00-RELEASENOTES  BUGS          COPYING  dump.rdb  Makefile   README      redis.log  runtest-cluster   sentinel.conf  tests
bin              CONTRIBUTING  deps     INSTALL   MANIFESTO  redis.conf  runtest    runtest-sentinel  src            utils

發現有什麼問題了吧？哈哈哈哈