Redis3.x Sentinel集羣配置

redis高可用的三種常見的集羣方式：redis sentinel 、redis cluster（多主機+分佈式）、redis sharding。本文主機介紹redis sentinel的部署過程。

centos7部署redis的主機ip地址：

master：10.11.11.109

slave：10.11.11.110，10.11.11.111

3.一、Redis3.x Sentinel集羣配置

Redis服務啓動和配置文件端口的查詢

# ansible redmon -i /root/ans/ansible_inventory.txt -m systemd -a "name=redis enabled=yes state=started"

# ansible redmon -i /root/ans/ansible_inventory.txt -m shell -a "rpm -ql redis"

# ansible redmon -i /root/ans/ansible_inventory.txt -m shell -a "ps -ef | grep redis"

# ansible redmon -i /root/ans/ansible_inventory.txt -m shell -a "ss -tunlp | grep redis"

3.1.一、準備Redis配置文件：

# ansible redmon -i /root/ans/ansible_inventory.txt -m fetch -a "src=/etc/redis.conf dest=/root/ans/conf.d flat=yes" --limit=10.11.11.109

# ansible redmon -i /root/ans/ansible_inventory.txt -m fetch -a "src=/etc/redis-sentinel.conf dest=/root/ans/conf.d flat=yes" --limit=10.11.11.109

3.1.二、三臺機器主從節點的準備：

1）、master節點：10.11.11.109

修改後，包含默認的狀況以下：

# sed -e "s/#.*//g" redis.conf | awk '{if (length !=0) print $0}'

bind 10.11.11.109

protected-mode no

port 6379

tcp-backlog 511

timeout 0

tcp-keepalive 300

daemonize yes

supervised no

pidfile /var/run/redis/redis.pid

loglevel notice

logfile /var/log/redis/redis.log

databases 16

save 900 1

save 300 10

save 60 10000

stop-writes-on-bgsave-error yes

rdbcompression yes

rdbchecksum yes

dbfilename dump.rdb

dir /var/lib/redis

slave-serve-stale-data yes

slave-read-only yes

repl-diskless-sync no

repl-diskless-sync-delay 5

repl-disable-tcp-nodelay no

slave-priority 100

appendonly yes

appendfilename "appendonly.aof"

appendfsync everysec

no-appendfsync-on-rewrite no

auto-aof-rewrite-percentage 100

auto-aof-rewrite-min-size 64mb

aof-load-truncated yes

lua-time-limit 5000

slowlog-log-slower-than 10000

slowlog-max-len 128

latency-monitor-threshold 0

notify-keyspace-events ""

hash-max-ziplist-entries 512

hash-max-ziplist-value 64

list-max-ziplist-size -2

list-compress-depth 0

set-max-intset-entries 512

zset-max-ziplist-entries 128

zset-max-ziplist-value 64

hll-sparse-max-bytes 3000

activerehashing yes

client-output-buffer-limit normal 0 0 0

client-output-buffer-limit slave 256mb 64mb 60

client-output-buffer-limit pubsub 32mb 8mb 60

hz 10

aof-rewrite-incremental-fsync yes

下面是能夠或須要修改的選項：

僅僅修改這四項，爲的是方便管理，也能夠保持默認配置

daemonize    yes    使Redis以守護進程模式運行

pidfile    /var/run/redis_端口號.pid    設置Redis的PID文件位置

port    端口號6379    設置Redis監聽的端口號

dir     本身定義目錄    設置持久文件存放位置

logfile /var/log/redis/redis.log

dir /var/lib/redis

 

詳細狀況：

bind 10.11.11.109

protected-mode no

默認狀況下，Redis node和sentinel的protected-mode都是yes，在搭建集羣時，若想從遠程鏈接redis集羣，須要將redis.conf和sentinel.conf的protected-mode修改成no，若只修改redis node，從遠程鏈接sentinel後，依然是沒法正常使用的，且sentinel的配置文件中沒有protected-mode配置項，須要手工添加。

protected-mode在默認開啓的狀況下要是配置裏沒有指定bind和密碼。開啓該參數後，redis只會本地進行訪問，拒絕外部訪問

 

##啓用增量（Master禁用）

appendonly yes

appendfsync everysec

2）、slave節點：10.11.11.110-111

修改後，包含默認的狀況以下：

# sed -e "s/#.*//g" redis.conf.201-202 | awk '{if (length !=0) print $0}'

bind 10.11.11.110-111

protected-mode no

port 6379

tcp-backlog 511

timeout 0

tcp-keepalive 300

daemonize yes

supervised no

pidfile /var/run/redis/redis.pid

loglevel notice

logfile /var/log/redis/redis.log

databases 16

save 900 1

save 300 10

save 60 10000

stop-writes-on-bgsave-error yes

rdbcompression yes

rdbchecksum yes

dbfilename dump.rdb

dir /var/lib/redis

slaveof 10.11.7.205 6379

slave-serve-stale-data yes

slave-read-only yes

repl-diskless-sync no

repl-diskless-sync-delay 5

repl-disable-tcp-nodelay no

slave-priority 100

appendonly yes

appendfilename "appendonly.aof"

appendfsync everysec

no-appendfsync-on-rewrite no

auto-aof-rewrite-percentage 100

auto-aof-rewrite-min-size 64mb

aof-load-truncated yes

lua-time-limit 5000

slowlog-log-slower-than 10000

slowlog-max-len 128

latency-monitor-threshold 0

notify-keyspace-events ""

hash-max-ziplist-entries 512

hash-max-ziplist-value 64

list-max-ziplist-size -2

list-compress-depth 0

set-max-intset-entries 512

zset-max-ziplist-entries 128

zset-max-ziplist-value 64

hll-sparse-max-bytes 3000

activerehashing yes

client-output-buffer-limit normal 0 0 0

client-output-buffer-limit slave 256mb 64mb 60

client-output-buffer-limit pubsub 32mb 8mb 60

hz 10

aof-rewrite-incremental-fsync yes

詳細說明：

下面是能夠或須要修改的選項：

僅僅修改這四項，爲的是方便管理，也能夠保持默認配置

daemonize    yes    使Redis以守護進程模式運行

pidfile    /var/run/redis_端口號.pid    設置Redis的PID文件位置

port    端口號6379    設置Redis監聽的端口號

dir     本身定義目錄    設置持久文件存放位置

protected-mode no

默認狀況下，Redis node和sentinel的protected-mode都是yes，在搭建集羣時，若想從遠程鏈接redis集羣，須要將redis.conf和sentinel.conf的protected-mode修改成no，若只修改redis node，從遠程鏈接sentinel後，依然是沒法正常使用的，且sentinel的配置文件中沒有protected-mode配置項，須要手工添加。

protected-mode在默認開啓的狀況下要是配置裏沒有指定bind和密碼。開啓該參數後，redis只會本地進行訪問，拒絕外部訪問

logfile "/var/log/redis/redis.log"

 

bind 10.11.11.110-111

##啓用增量（Master禁用）

appendonly yes

appendfsync everysec

 

slave-priority 80                ---->110

slave-priority 50                --->111

添加爲從節點：

# slaveof <masterip> <masterport>

slaveof 192.168.0.247 6379

copy到指定的主從節點：

# ls

redis.conf  redis.conf.201  redis.conf.202  redis-sentinel.conf 

# ansible redmon -i /root/ans/ansible_inventory.txt -m copy -a "src=/root/ans/conf.d/redis.conf dest=/etc/redis.conf backup=yes" --limit=10.11.11.109

# ansible redmon -i /root/ans/ansible_inventory.txt -m copy -a "src=/root/ans/conf.d/redis.conf.201 dest=/etc/redis.conf backup=yes" --limit=10.11.11.110

# ansible redmon -i /root/ans/ansible_inventory.txt -m copy -a "src=/root/ans/conf.d/redis.conf.202 dest=/etc/redis.conf backup=yes" --limit=10.11.11.111

3）、啓動Redis主從集羣

先啓動主節點（master）：

# ansible redmon -i /root/ans/ansible_inventory.txt -m systemd -a "name=redis enabled=yes state=restarted" --limit=10.11.11.109

再啓動從節點：

# ansible redmon -i /root/ans/ansible_inventory.txt -m systemd -a "name=redis enabled=yes state=restarted" --limit=10.11.11.110,10.11.11.111

4）、防火牆放行

#  ansible redmon -i /root/ans/ansible_inventory.txt -m firewalld -a "zone=public state=enabled permanent=yes port=6379/tcp"

#  ansible redmon -i /root/ans/ansible_inventory.txt -m firewalld -a "zone=public state=enabled permanent=true port=6379/udp"

注意使其生效喔

#  ansible redmon -i /root/ans/ansible_inventory.txt -m shell -a "firewall-cmd --reload"

5）、查看redis主從狀態

# ansible redmon -i /root/ans/ansible_inventory.txt -m shell -a "redis-cli -h 10.11.11.109 info Replication" --limit=10.11.11.109

10.11.11.109 | SUCCESS | rc=0 >>

# Replication

role:master

connected_slaves:2

slave0:ip=10.11.11.111,port=6379,state=online,offset=57,lag=1

slave1:ip=10.11.11.110,port=6379,state=online,offset=57,lag=1

master_repl_offset:57

repl_backlog_active:1

repl_backlog_size:1048576

repl_backlog_first_byte_offset:2

repl_backlog_histlen:56

 

# ansible redmon -i /root/ans/ansible_inventory.txt -m shell -a "redis-cli -h 10.11.11.111 info Replication" --limit=10.11.11.111

10.11.11.111 | SUCCESS | rc=0 >>

# Replication

role:slave

master_host:10.11.11.109

master_port:6379

master_link_status:up

master_last_io_seconds_ago:2

master_sync_in_progress:0

slave_repl_offset:85

slave_priority:50

slave_read_only:1

connected_slaves:0

master_repl_offset:0

repl_backlog_active:0

repl_backlog_size:1048576

repl_backlog_first_byte_offset:0

repl_backlog_histlen:0

 

# ansible redmon -i /root/ans/ansible_inventory.txt -m shell -a "redis-cli -h 10.11.11.110 info Replication" --limit=10.11.11.110

10.11.11.110 | SUCCESS | rc=0 >>

# Replication

role:slave

master_host:10.11.11.109

master_port:6379

master_link_status:up

master_last_io_seconds_ago:6

master_sync_in_progress:0

slave_repl_offset:99

slave_priority:80

slave_read_only:1

connected_slaves:0

master_repl_offset:0

repl_backlog_active:0

repl_backlog_size:1048576

repl_backlog_first_byte_offset:0

repl_backlog_histlen:0

3.1.三、配置sentinel集羣

1）、主備主機的sentinel.conf配置以下：

master節點：

# sed -e "s/#.*//g" redis-sentinel.conf | awk '{if (length !=0) print $0}'

port 26379

dir /data/var/tmp

sentinel monitor mymaster 10.11.11.109 6379 2

sentinel down-after-milliseconds mymaster 30000

sentinel parallel-syncs mymaster 1

sentinel failover-timeout mymaster 180000

logfile /data/var/log/redis/sentinel.log

bind 10.11.11.109

protected-mode no

slave節點：

# sed -e "s/#.*//g" redis-sentinel.conf.110 | awk '{if (length !=0) print $0}'

port 26379

dir /data/var/tmp

sentinel monitor mymaster 10.11.11.109 6379 2

sentinel down-after-milliseconds mymaster 30000

sentinel parallel-syncs mymaster 1

sentinel failover-timeout mymaster 180000

logfile /data/var/log/redis/sentinel.log

bind 10.11.11.110

protected-mode no

# sed -e "s/#.*//g" redis-sentinel.conf.backup111 | awk '{if (length !=0) print $0}'

port 26379

dir /data/var/tmp

sentinel monitor mymaster 10.11.11.109 6379 2

sentinel down-after-milliseconds mymaster 30000

sentinel parallel-syncs mymaster 1

sentinel failover-timeout mymaster 180000

logfile /data/var/log/redis/sentinel.log

bind 10.11.11.111

protected-mode no

copy到指定的主從節點：

# ls

 redis-sentinel.conf.110  redis-sentinel.conf.backup111  redis-sentinel.conf.master

 

# ansible redmon -i /root/ans/ansible_inventory.txt -m copy -a "src=/root/ans/conf.d/redis-sentinel.conf.master dest=/etc/redis-sentinel.conf backup=yes" --limit=10.11.11.109

# ansible redmon -i /root/ans/ansible_inventory.txt -m copy -a "src=/root/ans/conf.d/redis-sentinel.conf.110 dest=/etc/redis-sentinel.conf backup=yes" --limit=10.11.11.110

# ansible redmon -i /root/ans/ansible_inventory.txt -m copy -a "src=/root/ans/conf.d/redis-sentinel.conf.backup111 dest=/etc/redis-sentinel.conf backup=yes" --limit=10.11.11.111

2）、防火牆配置

# ansible redmon -i /root/ans/ansible_inventory.txt -m firewalld -a "zone=public state=enabled permanent=yes port=26379/tcp"

# ansible redmon -i /root/ans/ansible_inventory.txt -m firewalld -a "zone=public state=enabled permanent=yes port=26379/udp"

# ansible redmon -i /root/ans/ansible_inventory.txt -m shell -a "firewall-cmd --reload"

3）、啓動sentinel集羣

注意點：

1)：首次啓動時，必須先啓動Master

2)：Sentinel 只在 server 端作主從切換，app端要本身開發(例如Jedis庫的SentinelJedis，可以監控Sentinel的狀態)

3)：若Master已經被斷定爲下線，Sentinel已經選擇了新的Master，也已經將old Master改爲Slave，可是尚未將其改爲new Master。若此時重啓old Master，則Redis集羣將處於無Master狀態，此時只能手動修改配置文件，而後從新啓動集羣

# ansible redmon -i /root/ans/ansible_inventory.txt -m systemd -a "name=redis-sentinel state=started enabled=yes" --limit=10.11.11.109

# ansible redmon -i /root/ans/ansible_inventory.txt -m systemd -a "name=redis-sentinel state=started enabled=yes" --limit=10.11.11.110

# ansible redmon -i /root/ans/ansible_inventory.txt -m systemd -a "name=redis-sentinel state=started enabled=yes" --limit=10.11.11.111

查詢狀態：

主從節點查看的結果是同樣的：

# ansible redmon -i /root/ans/ansible_inventory.txt -m shell -a "redis-cli -h 10.11.11.109 -p 26379 info Sentinel"

# ansible redmon -i /root/ans/ansible_inventory.txt -m shell -a "redis-cli -h 10.11.11.110 -p 26379 info Sentinel"

# ansible redmon -i /root/ans/ansible_inventory.txt -m shell -a "redis-cli -h 10.11.11.110 -p 26379 info Sentinel"

10.11.11.111 | SUCCESS | rc=0 >>

# Sentinel

sentinel_masters:1

sentinel_tilt:0

sentinel_running_scripts:0

sentinel_scripts_queue_length:0

sentinel_simulate_failure_flags:0

master0:name=mymaster,status=ok,address=10.11.11.109:6379,slaves=2,sentinels=3

 

10.11.11.109 | SUCCESS | rc=0 >>

# Sentinel

sentinel_masters:1

sentinel_tilt:0

sentinel_running_scripts:0

sentinel_scripts_queue_length:0

sentinel_simulate_failure_flags:0

master0:name=mymaster,status=ok,address=10.11.11.109:6379,slaves=2,sentinels=3

 

10.11.11.110 | SUCCESS | rc=0 >>

# Sentinel

sentinel_masters:1

sentinel_tilt:0

sentinel_running_scripts:0

sentinel_scripts_queue_length:0

sentinel_simulate_failure_flags:0

master0:name=mymaster,status=ok,address=10.11.11.109:6379,slaves=2,sentinels=3

4）、宕機演示

添加測試數據：

[root@redis247 ~]# redis-cli -h 192.168.0.8 -p 6379

192.168.0.9:6379> set lll 245

OK

192.168.0.8:6379> save

OK

 

[root@redis247 ~]# redis-cli -h 192.168.0.247 -p 26379 info Sentinel

# Sentinel

sentinel_masters:1

sentinel_tilt:0

sentinel_running_scripts:0

sentinel_scripts_queue_length:0

sentinel_simulate_failure_flags:0

master0:name=redismaster,status=ok,address=192.168.0.247:6379,slaves=2,sentinels=3

關掉redis-server服務：

[root@redis247 ~]# redis-cli -h 192.168.0.247 -p 6379 shutdown

[root@redis247 ~]# ss -tunlp | grep redis

tcp    LISTEN     0      128        192.168.0.247:26379                 *:*      users:(("redis-sentinel",1241,4))

狀態切換的結果：

[root@redis247 ~]# redis-cli -h 192.168.0.247 -p 26379 info Sentinel

# Sentinel

sentinel_masters:1

sentinel_tilt:0

sentinel_running_scripts:0

sentinel_scripts_queue_length:0

sentinel_simulate_failure_flags:0

master0:name=redismaster,status=ok,address=192.168.0.9:6379,slaves=2,sentinels=3

[root@redis247 ~]# redis-cli -h 192.168.0.9 -p 26381 info Sentinel

# Sentinel

sentinel_masters:1

sentinel_tilt:0

sentinel_running_scripts:0

sentinel_scripts_queue_length:0

sentinel_simulate_failure_flags:0

master0:name=redismaster,status=ok,address=192.168.0.9:6379,slaves=2,sentinels=3

再關閉sentinel：（至關於主機宕機）

[root@redis247 ~]# ss -tunlp | grep redis

tcp    LISTEN     0      128        192.168.0.247:26379                 *:*      users:(("redis-sentinel",1241,4))

[root@redis247 ~]# ps -aux | grep redis

Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ

root      1241  0.5  0.4 133532  2412 ?        Ssl  14:26   0:07 redis-sentinel 192.168.0.247:26379 [sentinel]

root      1272  0.0  0.1 103252   840 pts/0    S+   14:49   0:00 grep redis

[root@redis247 ~]# kill -9 1241

[root@redis247 ~]# ps -aux | grep redis

Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ

root      1274  0.0  0.1 103252   840 pts/0    S+   14:49   0:00 grep redis

[root@redis247 ~]# ss -tunlp | grep redis

狀態同上

 

從新啓動：

[root@redis247 ~]# /etc/init.d/redis restart

[root@redis247 ~]# redis-sentinel /usr/local/redis/sentinel.conf

[root@redis247 ~]# redis-cli -h 192.168.0.8 -p 26380 info Sentinel

# Sentinel

sentinel_masters:1

sentinel_tilt:0

sentinel_running_scripts:0

sentinel_scripts_queue_length:0

sentinel_simulate_failure_flags:0

master0:name=redismaster,status=ok,address=192.168.0.9:6379,slaves=2,sentinels=3

#注意：原來的主宕機從新啓動後，充當從的角色

# 若是不是主宕機，而是從宕機，那麼不會發生切換行爲，只會把宕機的那臺從集羣中剔除。

# 已宕機的機器，若是再次加入集羣，只要它成爲了當前主的從機，則Sentinel會自動發現，並將其加入集羣成員。

 

再次查看測試數據：

[root@redis247 ~]# redis-cli -h 192.168.0.9 -p 6379

192.168.0.9:6379> get lll

"245"