Redis主節點內存佔用太高

0. 基本狀況

Redis採用集羣模式，560個主節點，主從比爲1:1，單臺機器上爲16個節點。info memory觀察到主節點A單個Redis內存used_memory_rss_human爲9.2G（設置的maxmemory_human爲6G）超過最大值設置。

1.查看客戶端鏈接狀況

[root@ A ~]# redis-cli -p 9001 info clients

# Clients

connected_clients:4705

client_longest_output_list:204928

client_biggest_input_buf:0

blocked_clients:0

經過client_longest_output_list發現有大量的輸出排隊，進一步觀察全部的客戶端均存在這個狀況。

 

在A機器上刷客戶端輸出list：

 

CLIENT_A:9001# Clients

client_longest_output_list:491335

CLIENT_A:9002# Clients

client_longest_output_list:520840

CLIENT_A:9003# Clients

client_longest_output_list:486498

CLIENT_A:9004# Clients

client_longest_output_list:485387

CLIENT_A:9005# Clients

client_longest_output_list:480000

CLIENT_A:9006# Clients

client_longest_output_list:537211

CLIENT_A:9007# Clients

client_longest_output_list:487594

CLIENT_A:9008# Clients

client_longest_output_list:490037

CLIENT_A:9009# Clients

client_longest_output_list:478734

CLIENT_A:9010# Clients

client_longest_output_list:524717

CLIENT_A:9011# Clients

client_longest_output_list:487200

CLIENT_A:9012# Clients

client_longest_output_list:491687

CLIENT_A:9013# Clients

client_longest_output_list:483900

CLIENT_A:9014# Clients

client_longest_output_list:524557

CLIENT_A:9015# Clients

client_longest_output_list:497967

CLIENT_A:9016# Clients

client_longest_output_list:484250

 

2.輸出內存佔用不爲0 的內容

 [root@ A redis-node]# redis-cli -p 9001 client list |grep -v "omem=0"

id=843 addr=client_B:53475 fd=2248 name= age=582784 idle=0 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=456078 omem=7446059139 events=rw cmd=replconf

 

經過cmd可知是主從同步佔用大量輸出內存

3.檢查統計值

 

有問題Client A的從client_B

[root@localhost 9001]# redis-cli -p 9001

127.0.0.1:9001> info stats

# Stats

total_connections_received:2

total_commands_processed:25

instantaneous_ops_per_sec:0

total_net_input_bytes:1280055129

total_net_output_bytes:36461

instantaneous_input_kbps:192.22

instantaneous_output_kbps:0.00

rejected_connections:0

 

正常的機器C的從Client D

[root@CLIENT D redis-node]# redis-cli -p 9001

127.0.0.1:9001> info Stats

# Stats

total_connections_received:10

total_commands_processed:23220212769

instantaneous_ops_per_sec:18281

total_net_input_bytes:3405817622262

total_net_output_bytes:59530419

instantaneous_input_kbps:2882.54

instantaneous_output_kbps:0.03

rejected_connections:0

輸入速度差了10倍，其餘配置均同樣，懷疑是網絡問題。

 

4.檢查網絡時延

 

機器A的從client_B

[root@A redis-node]# traceroute client_B

traceroute to client_B (client_B), 30 hops max, 60 byte packets

 1  *.*.*.254   0.752 ms  0.809 ms  0.900 ms

 2  *.*.*.133  2.164 ms  2.170 ms  2.157 ms

 3  * * *

 4  * * *

 5  *.*.*.2  163.542 ms  163.940 ms  164.181 ms

 6  client_B (client_B)  160.995 ms  156.202 ms  158.603 ms

 

Clinet C的從Client D

[root@SHJX-YSP-GKTZCQ-SEV-128-35 ~]# traceroute Client D

traceroute to Client D (Client D), 30 hops max, 60 byte packets

 1  *.*.*.254   0.732 ms  0.799 ms  0.897 ms

 2  *.*.*.133   3.147 ms  3.182 ms  3.168 ms

 3  * * *

 4  * * *

 5  *.*.*.2   5.537 ms  5.788 ms  5.961 ms

 6  Client D (Client D)  1.953 ms  1.960 ms  1.956 ms

 

測試機器A到機器D性能

[root@ 42 9001]#  traceroute Client D

traceroute to Client D (Client D), 30 hops max, 60 byte packets

 1  *.*.*.254 (10.180.128.254)  0.738 ms  0.800 ms  0.889 ms

 2  *.*.*.133 (10.180.176.133)  2.156 ms  2.190 ms  2.177 ms

 3  * * *

 4  * * *

 5  *.*.*.2   5.768 ms  5.942 ms  6.114 ms

 6  Client D (Client D)  1.940 ms  1.964 ms  1.948 ms

 

說明是機器A到client_B網絡有問題

 

5.解決辦法

更換從的機器到client_B到clinet E

 

啓動後機器A上9001 內存使用狀況

127.0.0.1:9001> info memory

# Memory

used_memory:2771776520

used_memory_human:2.58G

used_memory_rss:3403390976

used_memory_rss_human:3.17G

used_memory_peak:18154232496

used_memory_peak_human:16.91G

used_memory_peak_perc:15.27%

used_memory_overhead:758586508

used_memory_startup:8105584

used_memory_dataset:2013190012

used_memory_dataset_perc:72.84%

total_system_memory:540783263744

total_system_memory_human:503.64G

used_memory_lua:37888

used_memory_lua_human:37.00K

maxmemory:6442450944

maxmemory_human:6.00G

maxmemory_policy:volatile-ttl

mem_fragmentation_ratio:1.23

mem_allocator:jemalloc-4.0.3

active_defrag_running:0

lazyfree_pending_objects:0

127.0.0.1:9001>

6.解決後觀察

Client E內存同步狀況

 

127.0.0.1:9001> info stats

# Stats

total_connections_received:37

total_commands_processed:5318696

instantaneous_ops_per_sec:19310

total_net_input_bytes:1711079396

total_net_output_bytes:4954958

instantaneous_input_kbps:2967.88

instantaneous_output_kbps:70.17

rejected_connections:0

sync_full:0

 

修改後機器A上的9001 輸出

[root@A redis-node]# redis-cli -p 9001 info  clients

# Clients

connected_clients:4695

client_longest_output_list:0

client_biggest_input_buf:0

blocked_clients:

 

 

7.總結：

在redis設置一樣而內存使用不一樣時，應先查看info命令對比正常機器的輸出不一樣，優先關注Stats的輸出狀況以及本地的redis.log日誌