高可用Redis(九)：Redis Sentinel

時間 2019-12-09

標籤可用 redis sentinel 欄目 Redis 简体版

原文原文鏈接

1.主從複製高可用的問題

主從複製高可用的做用python

1.爲master提供備份，當master宕機時，slave有完整的備份數據
2.對master實現分流，實現讀寫分離

可是主從架構有一個問題mysql

1.若是master宕機，故障轉移須要手動完成或者由別的工具來完成，從slave中選擇一個slave作爲新的mastergit

寫能力和存儲能力受限
只能在一個節點是寫入數據
全部數據只能保存在一個節點上

上圖模擬了主從複製架構中一主兩從狀況下，master宕機，則slave從master同步數據也斷開，此時client向master寫入數據會失敗，讀寫分離時讀取數據正常，但不能更新數據redis

master出現故障以後，手動進行故障轉移步驟算法

1.選擇一個slave，執行slave no one命令使之成爲一個master

2.對其他的slave執行slaveof new master命令，將這些slave指定爲新的master的slave

3.client會對新的master進行寫入數據，從slave中讀取原來的數據

上面的操做過程須要手動完成，或者編寫腳本，由腳原本執行這個過程，可是這個過程是有問題的：sql

怎麼判斷一個Redis節點是有問題的，怎麼通知client對新master進行寫入操做
怎麼保證一個完整的事務實現過程

上面的過程就可使用Redis Sentinel來實現api

2.Redis Sentinel架構說明

Redis Sentinel的功能：對Redis節點進行監控，故障判斷，故障轉移，故障通知架構

對於Redis的主從架構，Redis Sentinel會運行幾個sentinel進程dom

sentinel進程不操做數據，而是對Redis故障進行判斷和轉移socket

同時多個sentinel運行，即便一個sentinel進程運行異常，還有別的sentinel繼續運行，能夠保證對故障節點判斷的準確性，同時保證Redis的高可用

對於redis-cli來講，Redis cli不會再記錄Redis的IP和端口，而是從sentinel獲取Redis信息，而後進行鏈接Redis節點，進行數據寫入和讀取操做

多個Redis Sentinel對全部的master和slave進行監控，會實時記錄master和slave的地址信息

Redis Sentinel故障轉移步驟：

1.當某個master發生故障，多個sentinel會監控到這個異常，這些sentinel會按照必定規則從多個slave中選中一個作爲新的master,並通知別的slave重新的master中同步數據
2.當某個slave轉換爲新的master，sentinel會記錄新的master的地址信息和slave的地址信息，通知Redis cli
3.Redis cli接收到新的master和slave的信息，就會向新的master寫入數據，從slave中讀取數據
4.等到原來的master重啓以後，會變成新的master的slave,並重新的master同步數據

在上面的步驟裏，sentinel實現了Redis的故障自動發現，自動轉移和自動通知

說明：一套Redis sentinel集合能夠通知master-name作爲標識同時監控多套主從架構

3.Redis Sentinel安裝配置

3.1 環境說明

實驗在兩臺虛擬機上完成，IP地址分別爲：192.168.81.100和192.168.81.101

在兩臺虛擬機上運行4個redis-server,其中

192.168.81.100的6379端口爲master節點
192.168.81.100的6380端口爲slave節點
192.168.81.101的6379端口和6380端口爲slave節點

在192.168.81.101的26379，26380，26381端口開啓三個sentinel進行監控

3.2 在192.168.81.100虛擬機上配置主從節點

[root@localhost ~]# cd /etc/                            # 進入/etc目錄
[root@localhost ~]# systemctl stop redis                # 關閉系統中運行的redis
[root@localhost etc]# cp redis.conf redis_6379.conf     # 複製redis配置文件，以端口區分，方便後面進行配置
[root@localhost etc]# cp redis.conf redis_6380.conf     # 複製redis配置文件，以端口區分，方便後面進行配置
[root@localhost etc]# vi redis_6379.conf                # 編輯redis-server配置文件，修改下面幾行
    bind 0.0.0.0                                        # 修改bing選項，才能從系統外鏈接redis
    protected-mode yes                                  # 開啓保存模式
    port 6379                                           # 指定redis運行的端口
    daemonize yes                                       # 以守護進程啓動redis
    pidfile "/var/run/redis_6379.pid"                   # 指定redis運行時pid保存路徑
    logfile "/var/log/redis/redis_6379.log"             # 指定redis運行時日誌保存路徑
    dir /var/lib/redis_6379                             # 指定redis運行時數據文件保存路徑
    
[root@localhost etc]# vi redis_6380.conf                # 修改redis-server,修改下面幾行
    bind 0.0.0.0
    port 6380                                           # 指定redis運行的端口
    daemonize yes
    pidfile "/var/run/redis_6380.pid"
    logfile "/var/log/redis/redis_6380.log"
    dir /var/lib/redis_6380
    slaveof 192.168.81.100 6379                         # 指定redis-server爲192.168.81.100:6379的slave
    
[root@localhost etc]# redis-server /etc/redis_6379.conf     # 指定配置文件運行redis-server
[root@localhost etc]# redis-server /etc/redis_6380.conf     # 指定配置文件運行redis-server
[root@localhost etc]# ps aux | grep redis-server            # 查看redis-server是否運行
root       2548  0.3  1.7 155192 17720 ?        Ssl  23:14   0:00 redis-server 0.0.0.0:6379
root       2562  1.3  1.7 155192 17596 ?        Ssl  23:15   0:00 redis-server 0.0.0.0:6380
root       2567  0.0  0.0 112648   960 pts/3    R+   23:15   0:00 grep --color=auto redis-server
[root@localhost etc]# redis-cli -p 6380 info replication    # 進入6380端口運行redis客戶端，並執行'info replication'命令
# Replication
role:slave                              # 角色爲slave
master_host:192.168.81.100              # master爲192.168.81.100
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:1919
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

3.3 在192.168.81.101虛擬機上配置從節點

[root@mysql ~]# cd /etc/                                # 操做同192.168.81.100相同
[root@mysql ~]# systemctl stop redis
[root@mysql etc]# cp redis.conf redis_6379.conf
[root@mysql etc]# cp redis.conf redis_6380.conf
[root@mysql etc]# vi redis_6379.conf
    bind 0.0.0.0
    protected-mode yes
    port 6379
    daemonize yes
    pidfile "/var/run/redis_6379.pid"
    logfile "/var/log/redis/redis_6379.log"
    dir /var/lib/redis_6379
    slaveof 192.168.81.100 6379                         # 指定redis-server爲192.168.81.100:6379的slave
    
[root@mysql etc]# vi redis_6380.conf    
    bind 0.0.0.0
    port 6380
    daemonize yes
    pidfile "/var/run/redis_6380.pid"
    logfile "/var/log/redis/redis_6380.log"
    dir /var/lib/redis_6380
    slaveof 192.168.81.100 6379                         # 指定redis-server爲192.168.81.100:6379的slave
    
[root@mysql etc]# redis-server /etc/redis_6379.conf     # 指定配置文件運行redis-server
[root@mysql etc]# redis-server /etc/redis_6380.conf     # 指定配置文件運行redis-server
[root@mysql ~]# ps aux | grep redis-server              # 查看redis-server是否運行
root       2178  0.2  0.8 155204 17728 ?        Ssl  15:10   0:02 redis-server 0.0.0.0:6379
root       2184  0.2  0.8 155204 17724 ?        Ssl  15:10   0:02 redis-server 0.0.0.0:6380
root       2411  0.0  0.0 112664   972 pts/2    R+   15:29   0:00 grep --color=auto redis-server
[root@mysql ~]# redis-cli -p 6379 info replication
# Replication
role:slave                              # 角色爲slave
master_host:192.168.81.100              # master爲192.168.81.100
master_port:6379
master_link_status:up
master_last_io_seconds_ago:6
master_sync_in_progress:0
slave_repl_offset:1961
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
[root@mysql ~]# redis-cli -p 6380 info replication   
# Replication
role:slave                              # 角色爲slave
master_host:192.168.81.100              # master爲192.168.81.100
master_port:6379
master_link_status:up
master_last_io_seconds_ago:2
master_sync_in_progress:0
slave_repl_offset:1975
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

3.4 在192.168.81.101虛擬機上配置並運行sentinel

[root@mysql etc]# cp redis-sentinel.conf sentinel_26379.conf    # 複製sentinel配置文件，方便區分
[root@mysql etc]# cp redis-sentinel.conf sentinel_26380.conf    # 複製sentinel配置文件，方便區分
[root@mysql etc]# cp redis-sentinel.conf sentinel_26381.conf    # 複製sentinel配置文件，方便區分
[root@mysql etc]# vi sentinel_26379.conf                        # 修改sentinel配置文件,修改下面幾行
    daemonize yes                                               # 以守護進程方式啓動
    port 26379                                                  # 指定端口
    protected-mode no                                           # 關閉保護模式
    sentinel monitor mymaster 192.168.81.100 6379 2             # 設置sentinel監控信息
    logfile /var/log/redis/sentinel_26379.log                   # 設置日誌文件保存路徑
[root@mysql etc]# vi sentinel_26380.conf 
    daemonize yes
    port 26380  
    protected-mode no  
    sentinel monitor mymaster 192.168.81.100 6379 2
    logfile /var/log/redis/sentinel_26380.log
[root@mysql etc]# vi sentinel_26381.conf 
    protected-mode no  
    port 26381
    daemonize yes  
    sentinel monitor mymaster 192.168.81.100 6379 2
    logfile /var/log/redis/sentinel_26381.log
[root@mysql etc]# redis-sentinel /etc/sentinel_26379.conf       # 指定配置文件，啓動Redis Sentinel
[root@mysql etc]# redis-sentinel /etc/sentinel_26380.conf       # 指定配置文件，啓動Redis Sentinel  
[root@mysql etc]# redis-sentinel /etc/sentinel_26381.conf       # 指定配置文件，啓動Redis Sentinel
[root@mysql etc]# ps aux | grep sentinel                        # 查看Redis Sentinel是否運行
root       2709  0.9  0.2 142916  5464 ?        Ssl  15:49   0:00 redis-sentinel *:26379 [sentinel]
root       2713  1.1  0.2 142916  5472 ?        Ssl  15:49   0:00 redis-sentinel *:26380 [sentinel]
root       2717  2.0  0.2 142916  5476 ?        Rsl  15:49   0:00 redis-sentinel *:26381 [sentinel]
root       2721  0.0  0.0 112664   964 pts/2    R+   15:49   0:00 grep --color=auto sentinel
[root@mysql ~]# redis-cli -p 26379
127.0.0.1:26379> ping                                           # 執行ping操做
PONG
127.0.0.1:26379> info sentinel                                  # 查看全部sentinel的信息
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.81.100:6379,slaves=3,sentinels=3                   # 被監控的Redis主從架構命名爲mymaster，被監控Redis節點的master爲192.168.81.100L6379,有三個slave,同時有3個sentinel運行
127.0.0.1:26379> exit
[root@mysql ~]# grep -v '^#' /etc/sentinel_26379.conf | grep -v '^$'     #查看sentinel_26379配置文件，去除註釋和空行，Redis Sentinel向配置文件中添加了幾行內容
port 26379                                                  # sentinel運行的端口
dir "/tmp"
sentinel myid 9611958fc3e8b7c2be43385e44be88f87d725a77
sentinel monitor mymaster 192.168.81.100 6379 2             # sentinel監控的Redis節點名爲mymaster,master地址爲192.168.81.100:6379,quorem設置爲2
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
logfile "/var/log/redis/sentinel_26379.log"
daemonize yes
sentinel known-slave mymaster 192.168.81.101 6379           # sentinel探測到的slave
sentinel known-slave mymaster 192.168.81.100 6380           # sentinel探測到的slave
sentinel known-slave mymaster 192.168.81.101 6380           # sentinel探測到的slave
sentinel known-sentinel mymaster 192.168.81.101 26380 17ca0cb82becb58bd24e5a87ee3b6e8e9a49caf1                    # Redis Sentinel深測到的別的運行的sentinel
sentinel known-sentinel mymaster 127.0.0.1 26381 fb9342f3007e2abff165f5c33de1d48cf089f062                    # Redis Sentinel深測到的別的運行的sentinel
sentinel current-epoch 0
[root@mysql ~]# grep -v '^#' /etc/sentinel_26380.conf | grep -v '^$'  
port 26380
dir "/tmp"
sentinel myid 17ca0cb82becb58bd24e5a87ee3b6e8e9a49caf1
sentinel monitor mymaster 192.168.81.100 6379 2
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
logfile "/var/log/redis/sentinel_26380.log"
daemonize yes
sentinel known-slave mymaster 192.168.81.101 6379
sentinel known-slave mymaster 192.168.81.101 6380
sentinel known-slave mymaster 192.168.81.100 6380
sentinel known-sentinel mymaster 127.0.0.1 26381 fb9342f3007e2abff165f5c33de1d48cf089f062
sentinel known-sentinel mymaster 192.168.81.101 26379 9611958fc3e8b7c2be43385e44be88f87d725a77
sentinel current-epoch 0
[root@mysql ~]# grep -v '^#' /etc/sentinel_26381.conf | grep -v '^$' 
port 26381
daemonize yes
dir "/tmp"
sentinel myid fb9342f3007e2abff165f5c33de1d48cf089f062
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
logfile "/var/log/redis/sentinel_26381.log"
sentinel known-sentinel mymaster 127.0.0.1 26380 17ca0cb82becb58bd24e5a87ee3b6e8e9a49caf1
sentinel known-sentinel mymaster 192.168.81.101 26379 9611958fc3e8b7c2be43385e44be88f87d725a77
sentinel current-epoch 0

至此，3個sentinel已經正常運行了

3.5 python客戶端持續經過sentinel向Redis寫入數據，讀取數據

import random
import time
from redis.sentinel import Sentinel

sentinel = Sentinel([
    ('192.168.81.101',26379),
    ('192.168.81.101',26380),
    ('192.168.81.101',26381),
],socket_timeout=0.1)               # 傳入Redis Sentinel集合

while True:
    try:
        master = sentinel.discover_master('mymaster')
        print('current master IP:',master)          # 打印當前master的IP地址和端口
        val = random.randint(0,10000)               # 獲取10000之內隨機整數
        key = 'k%d' % val
        m = sentinel.master_for('mymaster', socket_timeout=0.5)
        m.set(key,'val%d' % val)                    # 經過sentinel向master節點寫入數據
        v = m.get(key)                              # 經過sentinel讀取數據
        print('{0} value is {1}'.format(key,v))
        time.sleep(1)
    except Exception as e:
        print("get no val:",e)

運行上面的代碼：

current master IP: ('192.168.81.100', 6379)
k6081 value is b'val6081'
current master IP: ('192.168.81.100', 6379)
k1778 value is b'val1778'
current master IP: ('192.168.81.100', 6379)
k4927 value is b'val4927'
current master IP: ('192.168.81.100', 6379)
k4074 value is b'val4074'
current master IP: ('192.168.81.100', 6379)
k1138 value is b'val1138'
current master IP: ('192.168.81.100', 6379)
k862 value is b'val862'
current master IP: ('192.168.81.100', 6379)
k4854 value is b'val4854'
current master IP: ('192.168.81.100', 6379)
k9233 value is b'val9233'
current master IP: ('192.168.81.100', 6379)
k6844 value is b'val6844'
current master IP: ('192.168.81.100', 6379)
k8089 value is b'val8089'

3.6 在192.168.81.100虛擬機上模擬master故障

[root@localhost etc]# redis-cli -p 6379                 # 鏈接6379端口鏈接redis-server
127.0.0.1:6379> info server                             # 執行命令查看當前redis-server的進程ID
# Server
redis_version:3.2.10
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:c8b45a0ec7dc67c6
redis_mode:standalone
os:Linux 3.10.0-514.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.5
process_id:2548                                         # 192.168.81.100:6379這個master節點的進程ID爲2548
run_id:8ae71ba92660697d52bfb74b99fb15ee82a7cf84
tcp_port:6379
uptime_in_seconds:6552
uptime_in_days:0
hz:10
lru_clock:12896355
executable:/etc/redis-server
config_file:/etc/redis_6379.conf
127.0.0.1:6379> exit
[root@localhost etc]# kill -9 2548                      # 在系統中kill掉2548這個進程
[root@localhost etc]# ps aux | grep redis-server        # 查看redis-server運行的進程，2548已經被終止
root       2562  0.2  1.7 155192 17756 ?        Ssl  Oct15   0:19 redis-server 0.0.0.0:6380
root       2651  0.0  0.0 112648   960 pts/3    S+   01:19   0:00 grep --color=auto redis-server

3.7 查看python代碼執行結果

current master IP: ('192.168.81.100', 6379)
1539597332.9189174 k6187 value is b'val6187'
current master IP: ('192.168.81.100', 6379)
1539597333.9236474 k1462 value is b'val1462'
current master IP: ('192.168.81.100', 6379)
get no val: Timeout connecting to server
current master IP: ('192.168.81.100', 6379)
get no val: Timeout connecting to server
current master IP: ('192.168.81.100', 6379)
get no val: Timeout connecting to server
...中間報錯信息省略
get no val: No master found for 'mymaster'
get no val: No master found for 'mymaster'
get no val: DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a bind address or an authentication password. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside.
get no val: No master found for 'mymaster'
...中間報錯信息省略
get no val: DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a bind address or an authentication password. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside.
current master IP: ('192.168.81.101', 6379)
1539597365.971147 k3568 value is b'val3568'
current master IP: ('192.168.81.101', 6379)
1539597366.974567 k7745 value is b'val7745'
current master IP: ('192.168.81.101', 6379)
1539597367.9783657 k6281 value is b'val6281'
current master IP: ('192.168.81.101', 6379)

能夠看到初始master是192.168.81.100:6379，在系統命令提示符中kill掉6379端口的redis-server後，python程序從1539597333.9236474秒後一直拋出異常，直到1539597365.971147秒，python程序經過sentinel獲取到新的master爲192.168.81.101:6379。

1539597365.971147減去1539597333.9236474剛好就是sentinel配置文件中sentinel down-after-milliseconds mymaster選項設定的30000毫秒，也就是30秒

3.8 查看Redis Sentinel的日誌

在192.168.81.101虛擬機上查看/var/log/redis/sentinel_26379.log日誌，下面的部分日誌就是sentinel在切換master的記錄

4067:X 15 Oct 17:55:40.303 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:42.119 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:42.334 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:44.208 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:44.349 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:46.266 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:46.365 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:48.419 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:48.481 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:50.429 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:50.506 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:52.548 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:52.571 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:54.592 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:54.598 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:56.655 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:56.662 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:58.734 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:58.886 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:00.798 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:00.972 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:02.884 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:03.042 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:04.917 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:05.099 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:07.045 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:07.112 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:09.145 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:09.324 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:11.218 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:11.345 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:12.211 # +sdown master mymaster 192.168.81.100 6379
4067:X 15 Oct 17:56:12.345 # +new-epoch 8
4067:X 15 Oct 17:56:12.346 # +vote-for-leader fb9342f3007e2abff165f5c33de1d48cf089f062 8
4067:X 15 Oct 17:56:13.267 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:13.269 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:13.289 # +odown master mymaster 192.168.81.100 6379 #quorum 2/2
4067:X 15 Oct 17:56:13.289 # Next failover delay: I will not start a failover before Mon Oct 15 18:02:13 2018
4067:X 15 Oct 17:56:13.440 # +config-update-from sentinel fb9342f3007e2abff165f5c33de1d48cf089f062 192.168.81.101 26381 @ mymaster 192.168.81.100 6379
4067:X 15 Oct 17:56:13.441 # +switch-master mymaster 192.168.81.100 6379 192.168.81.101 6379
4067:X 15 Oct 17:56:13.441 * +slave slave 192.168.81.101:6380 192.168.81.101 6380 @ mymaster 192.168.81.101 6379
4067:X 15 Oct 17:56:13.441 * +slave slave 192.168.81.100:6380 192.168.81.100 6380 @ mymaster 192.168.81.101 6379
4067:X 15 Oct 17:56:13.441 * +slave slave 192.168.81.100:6379 192.168.81.100 6379 @ mymaster 192.168.81.101 6379
4067:X 15 Oct 17:56:15.586 * +sentinel-address-switch master mymaster 192.168.81.101 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:15.604 * +sentinel-address-switch master mymaster 192.168.81.101 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:15.689 * +sentinel-address-switch master mymaster 192.168.81.101 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:15.693 * +sentinel-address-switch master mymaster 192.168.81.101 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062

從上面的日誌能夠看出，在17:56:13.441時間點sentinel把master從182.168.81.100:6379切換到192.168.81.101:6379

4.Redis Sentinel客戶端鏈接(python版)

Redis Sentinel的高可用指的是服務端的高可用，對於Redis服務端的master宕機，sentinel能夠對故障實現自動發現，自動轉移，自動通知。這個過程客戶端是感知不到的

Redis高可用即依賴於服務端的高可用，又依賴於客戶端的高可用

經過分析Redis Sentinel的請求響應流程，能夠知道客戶端實現高可用步驟：

1.客戶端遍歷sentinel節點集合，獲取一個可用的sentinel節點，同時獲取masterName
2.在可用的sentinel節點上執行sentinel的API，獲取master的地址和端口
3.在sentinel內部，sentinel會按照必定頻率在master或者slave上執行info命令，獲取對應節點的信息
4.客戶端獲取到master的地址和端口，會執行role命令或者role replication命令，對master進行驗證
5.當master出現故障，sentinel按照算法從slave中選出一個作爲新的master，同時把其他的slave作爲新的master的slave
6.sentinel維護一個頻道，sentinel在這個頻道中發佈消息，指出新master的IP和端口
7.客戶端訂閱這個頻道，接收消息後就知道新master的IP和端口，向新master鏈接進行數據處理

原理圖以下

python客戶端接入Redis Sentinel須要兩個參數：sentinel地址集合，masterName

須要注意的是Redis節點的配置文件中的protected-mode必須設置爲yes，不然鏈接會失敗

from redis.sentinel import Sentinel

sentinel = Sentinel([
    ('192.168.81.101',26379),
    ('192.168.81.101',26380),
    ('192.168.81.101',26381),
],socket_timeout=0.1)

master = sentinel.discover_master('mymaster')
print(master)       # ('192.168.81.100', 6379)
slave = sentinel.discover_slaves('mymaster')
print(slave)        # [('192.168.81.100', 6380), ('192.168.81.101', 6379), ('192.168.81.101', 6380)]

5.Redis Sentinel實現原理

5.1 Redis Sentinel內部的三個定時任務

Redis Sentinel內部有三個定時任務來對redid節點進行故障判斷和轉移

1.每10秒每一個sentinel對master和slave執行info命令，以發現slave節點和確認主從關係

sentinel在master節點執行info replication命令，從命令執行結果中解析出slave節點

2.每2秒每一個sentinel經過master節點的channel交換信息(發佈訂閱)

master節點上有一個發佈訂閱的channel頻道：__sentinel__:hello，用於全部sentinel之間進行信息交換

一個sentinel發佈消息，消息包含當前sentinel節點的信息，對其餘sentinel節點的判斷以及當前sentinel對master節點和slave節點的一些判斷

其餘sentinel均可以接收到這條消息

新加入sentinel節點時，sentinel節點之間能夠相互感知，以達到信息交互的功能

3.每1秒每一個sentinel對其餘sentinel節點和Redis節點執行ping操做

每一個sentinel均可以知道其餘sentinel節點，當監控的master發生故障時，方便進行判斷和新master的挑選，這個定時任務是master進行故障斷定的依據

5.2 主觀下線和客觀下線

主觀下線：每一個sentinel節點對Redis節點失敗的'偏見'

在redis-sentinel配置文件中，有下面這種配置

sentinel monitor <master-name> <ip> <redis-port> <quorum>
sentinel down-after-milliseconds <master-name> <timeout>

一個sentinel集合能夠同時監控多個master,slave的節點

sentinel對多個master，slave節點進行區分的標識就是master-name,ip和port是master節點的IP地址和端口,quorum是master客觀下線以後sentinel進行判斷的節點數

sentinel對master進行主觀下線判斷的時間，單們爲毫秒

每一個sentinel每秒對master和slave執行ping操做，當sentinel對master或slave在timeout定義的毫秒時間內沒有回覆，則sentinel會認爲這個節點已經被主觀下線了

在前面的例子中對sentinel的配置是

sentinel monitor mymaster 192.168.81.100 6379 2
sentinel down-after-milliseconds mymaster 30000

解釋：

sentinel集合監控名爲mymaster的master,slave節點
被監控的master節點的IP地址是192.168.81.100，端口爲6379，
sentinel會在`__sentinel__:hello`頻道中交流對master節點的見解，若是sentinel節點都對master節點ping失敗'達成共識'，sentinel個數超過quorum的個數，sentinel集合則會認爲master節點客觀下線
當兩個sentinel對master節點執行ping操做，在30000毫秒(30秒)時間內沒有獲得回覆，則認爲節點已經被主觀下線

quorum建議設置爲：(sentinel節點數 / 2) + 1，能夠根據應用場景進行設定

5.3 sentinel領導者選舉

要點：

只須要一個sentinel節點就能夠完成故障轉移
經過`sentinel is-master-down-by-addr`命令來完成sentinel交換對master節點的失敗斷定和新master的選舉

完成sentinel領導者選舉步驟：

1.每一個作主觀下線的sentinel節點向其餘sentinel節點發送命令，要求將本身設置爲領導者
2.收到命令的sentinel節點若是沒有贊成贊成其餘sentinel節點發送的命令，那麼將贊成該請求，不然拒絕
3.若是該sentinel節點發現本身的票數已經超過sentinel集合半數且超過quorum，將成爲領導者
4.若是此過程當中有多個sentinel節點成爲領導者，那麼將等待一段時間從新進行選舉

5.4 故障轉移(由sentinel領導者節點完成)

故障轉移步驟：

1.從slave節點中選出一個合適的節點做爲新的master節點
2.對選出的slave節點執行`slaveof no one`命令，使成爲新的master節點
3.向剩餘的slave節點發送命令，讓slave節點成爲新master節點的slave節點，而後重新master節點同步數據
    數據同步規則和parallel-syncs參數有關
    如一個一主三從架構中，master故障，sentinel領導者從3個slave中選出一個做爲新的master節點，剩餘的兩個slave節點會成爲新master節點的slave，重新master節點同步同步數據
    master節點只須要生成一次RDB文件
        若是parallel-syncs參數設置爲1，則剩餘兩個slave節點會按順序重新master節點拷貝數據，一個slave切點拷貝完成，另一個slave纔會重新master節點拷貝數據
        若是parallel-syncs參數設置爲2，則兩個slave節點會同時從master節點進行數據拷貝，這無疑會加入新master的開銷
4.sentinel領導者會把原來的master節點設置爲slave節點，並保持對其'關注'，當原來的master節點恢復後，sentinel會使其去複製新master節點的數據

5.5 slave節點的選擇

slave節點選擇規則

1.選擇slave-priority(slave節點優先級)最高的slave節點，若是存在則返回，不存在則繼續
2.選擇複製偏移量(offset)最大的slave節點,offset最大說明對master的數據複製的最完整，若是存在則返回，不存在則繼續
3.選擇run_id最小的slave節點，run_id最小說明slave節點啓動最先

6.總結：

Redis Sentinel是Redis的高可用實現方案：故障發現，故障自動轉移，配置中心，客戶端通知
Redis Sentinel是Redis 2.8版本開始才正式生產可用，以前版本不可用於生產
盡能夠在不一樣物理機上部署Redis Sentinel全部節點，可是最好一個局域網內
Redis Sentinel中sentinel節點個數應該大於等於3，且最好爲奇數，能夠保證判斷的公平
Redis Sentinel中的數據節點與普通數據節點沒有區別
客戶端初始化時鏈接的是Sentinel節點集合，不是具體的Redis節點，可是Sentinel只是配置中心不是代理
Redis Sentinel經過三個定時任務實現了Sentinel節點對於master，slave，其他sentinel節點的監控
Redis Sentinel在對節點作失敗斷定時分爲主觀下線和客觀下線
看懂Redis Sentinel故障轉移日誌對於Redis Sentinel以及問題排查很是有幫助
Redis Sentinel實現讀寫分離高可用能夠依賴Redis Sentinel節點的消息通知，獲取Redis數據節點的狀態變化

相關標籤/搜索

redis+sentinel

redis+redis

redis

docker+redis+sentinel

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。