docker 實現redis集羣搭建

時間  2019-12-14
標籤 docker 實現 redis 集羣 搭建 欄目 Docker 简体版
原文   原文鏈接

摘要：接觸docker以來，彷佛養成了一種習慣，安裝什麼應用軟件都想往docker方向作，今天就想來嘗試下使用docker搭建redis集羣。html

首先，咱們須要理論知識：Redis Cluster是Redis的分佈式解決方案，它解決了redis單機中心化的問題，分佈式數據庫——首要解決把整個數據集按照分區規則映射到多個節點的問題。node

這邊就須要知道分區規則——哈希分區規則。Redis Cluster 採用哈希分區規則中的虛擬槽分區。全部的鍵根據哈希函數映射到0 ～ 16383，計算公式：slot = CRC16(key)&16383。每個節點負責維護一部分槽以及槽所映射的鍵值數據。redis

1、建立redis docker基礎鏡像

  1. 下載redis安裝包，使用版本爲：4.0.1
    [root@etcd1 tmp]# mkdir docker_redis_cluster
[root@etcd1 tmp]# cd docker_redis_cluster/
[root@etcd2 docker_redis_cluster]# wget http://download.redis.io/releases/redis-4.0.1.tar.gz
  2. 解壓編譯redis
    [root@etcd1 docker_redis_cluster]# tar zxvf redis-4.0.1.tar.gz
[root@etcd1 docker_redis_cluster]# cd redis-4.0.1/
[root@etcd1 redis-4.0.1]# make
  3. 修改redis配置
    [root@etcd3 redis-4.0.1]# vi /tmp/docker_redis_cluster/redis-4.0.1/redis.conf

    　　
    修改bind ip地址docker

    # ~~~ WARNING ~~~ If the computer running Redis is directly exposed to the
# internet, binding to all the interfaces is dangerous and will expose the
# instance to everybody on the internet. So by default we uncomment the
# following bind directive, that will force Redis to listen only into
# the IPv4 lookback interface address (this means Redis will be able to
# accept connections only from clients running into the same computer it
# is running).
#
# IF YOU ARE SURE YOU WANT YOUR INSTANCE TO LISTEN TO ALL THE INTERFACES
# JUST COMMENT THE FOLLOWING LINE.
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#bind 127.0.0.1
bind 0.0.0.0

    　　

    將守護進程yes改爲no數據庫

    # By default Redis does not run as a daemon. Use 'yes' if you need it.
# Note that Redis will write a pid file in /var/run/redis.pid when daemonized.
daemonize no

    　　
    將密碼項註釋去掉，添加新密碼json

    # Warning: since Redis is pretty fast an outside user can try up to
# 150k passwords per second against a good box. This means that you should
# use a very strong password otherwise it will be very easy to break.
#
# requirepass foobared

    　　修改成centos

    # Warning: since Redis is pretty fast an outside user can try up to
# 150k passwords per second against a good box. This means that you should
# use a very strong password otherwise it will be very easy to break.
#
requirepass 123456


    由於配置了密碼，因此，配置中另一處主從鏈接也須要配置密碼bash

    # If the master is password protected (using the "requirepass" configuration
# directive below) it is possible to tell the slave to authenticate before
# starting the replication synchronization process, otherwise the master will
# refuse the slave request.
#
# masterauth <master-password>

    　　修改成網絡

    # If the master is password protected (using the "requirepass" configuration
# directive below) it is possible to tell the slave to authenticate before
# starting the replication synchronization process, otherwise the master will
# refuse the slave request.
#
# masterauth <master-password>
masterauth 123456

    　　

    　　

    設置日誌路徑app

    # Specify the log file name. Also the empty string can be used to force
# Redis to log on the standard output. Note that if you use standard
# output for logging but daemonize, logs will be sent to /dev/null
logfile "/var/log/redis/redis-server.log"

    　　
    配置集羣相關信息，去掉配置項前面的註釋

    # Normal Redis instances can't be part of a Redis Cluster; only nodes that are
# started as cluster nodes can. In order to start a Redis instance as a
# cluster node enable the cluster support uncommenting the following:
#
cluster-enabled yes

# Every cluster node has a cluster configuration file. This file is not
# intended to be edited by hand. It is created and updated by Redis nodes.
# Every Redis Cluster node requires a different cluster configuration file.
# Make sure that instances running in the same system do not have
# overlapping cluster configuration file names.
#
cluster-config-file nodes-6379.conf

# Cluster node timeout is the amount of milliseconds a node must be unreachable
# for it to be considered in failure state.
# Most other internal time limits are multiple of the node timeout.
#
cluster-node-timeout 15000
  4. 鏡像製做
    [root@etcd3 docker_redis_cluster]# cd /tmp/docker_redis_cluster
[root@etcd3 docker_redis_cluster]# vi Dockerfile 
# Redis
# Version 4.0.1

FROM Centos:7
    
ENV REDIS_HOME /usr/local
    
ADD redis-4.0.1.tar.gz / # 本地的redis源碼包複製到鏡像的根路徑下，ADD命令會在複製事後自動解包。被複制的對象必須處於Dockerfile同一路徑，且ADD後面必須使用相對路徑
RUN mkdir -p $REDIS_HOME/redis # 建立安裝目錄
ADD redis-4.0.1/redis.conf $REDIS_HOME/redis/  # 將一開始編譯產生並修改後的配置複製到安裝目錄

RUN yum -y update  # 更新yum源
RUN yum install -y gcc make # 安裝編譯須要的工具

WORKDIR /redis-4.0.1
RUN make
RUN mv /redis-4.0.1/src/redis-server  $REDIS_HOME/redis/   # 編譯後，容器中只須要可執行文件redis-server

WORKDIR /
RUN rm -rf /redis-4.0.1          # 刪除解壓文件

RUN yum remove -y gcc make   # 安裝編譯完成以後，能夠刪除多餘的gcc跟make

VOLUME ["/var/log/redis"]  # 添加數據卷

EXPOSE 6379   # 暴露6379端口，也能夠暴露多個端口，這裏不須要如此

    　　
    PS.當前鏡像非可執行鏡像，因此沒有包含ENTRYPOINT和CMD指令

  5. 構建鏡像
    # 切換中國源
[root@etcd3 docker_redis_cluster]# vi /etc/docker/daemon.json
{
  "registry-mirrors": ["https://registry.docker-cn.com"]
}

# 編譯
[root@etcd3 docker_redis_cluster]# docker build -t hakimdstx/cluster-redis .
...

Complete!
 ---> 546cb1d34f35
Removing intermediate container 6b6556c5f28d
Step 14/15 : VOLUME /var/log/redis
 ---> Running in 05a6642e4046
 ---> e7e2fb8676b2
Removing intermediate container 05a6642e4046
Step 15/15 : EXPOSE 6379
 ---> Running in 5d7abe1709e2
 ---> 2d1322475f79
Removing intermediate container 5d7abe1709e2
Successfully built 2d1322475f79

    　　
    鏡像製做完成，製做中間可能會報： Public key for glibc-headers-2.17-222.el7.x86_64.rpm is not installed 錯誤，這時候須要在鏡像配置中添加一句命令：

    ...
RUN rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
RUN yum -y update  # 更新yum源
RUN yum install -y gcc make # 安裝編譯須要的工具

    查看鏡像：
    [root@etcd3 docker_redis_cluster]# docker images
REPOSITORY                                  TAG                 IMAGE ID            CREATED             SIZE
hakimdstx/cluster-redis                     4.0.1               1fca5a08a4c7        14 seconds ago      435 MB
centos                                      7                   49f7960eb7e4        2 days ago          200 MB

    　

    以上，redis 基礎鏡像就製做完成了

2、製做redis節點鏡像

  1. 基於此前製做的redis基礎鏡像建立一個redis節點鏡像
    [root@etcd3 tmp]# mkdir docker_redis_nodes
[root@etcd3 tmp]# cd docker_redis_nodes
[root@etcd3 docker_redis_nodes]# vi Dockerfile
# Redis Node
# Version 4.0.1
    
FROM hakimdstx/cluster-redis:4.0.1

# MAINTAINER_INFO
MAINTAINER hakim 1194842583@qq.com

ENTRYPOINT ["/usr/local/redis/redis-server", "/usr/local/redis/redis.conf"]

    　　

  2. 構建redis節點鏡像
    [root@etcd3 docker_redis_nodes]# docker build -t hakimdstx/nodes-redis:4.0.1 .       
Sending build context to Docker daemon 2.048 kB
Step 1/3 : FROM hakimdstx/cluster-redis:4.0.1
 ---> 1fca5a08a4c7
Step 2/3 : MAINTAINER hakim 1194842583@qq.com
 ---> Running in cc6e07eb2c36
 ---> 55769d3bfacb
Removing intermediate container cc6e07eb2c36
Step 3/3 : ENTRYPOINT /usr/local/redis/redis-server /usr/local/redis/redis.conf
 ---> Running in f5dedf88f6f6
 ---> da64da483559
Removing intermediate container f5dedf88f6f6
Successfully built da64da483559

    　　

  3. 查看鏡像
    [root@etcd3 docker_redis_nodes]# docker images
REPOSITORY                                  TAG                 IMAGE ID            CREATED             SIZE
hakimdstx/nodes-redis                       4.0.1               da64da483559        51 seconds ago      435 MB
hakimdstx/cluster-redis                     4.0.1               1fca5a08a4c7        9 minutes ago       435 MB
centos                                      7                   49f7960eb7e4        2 days ago          200 MB

    　　

3、運行redis集羣

  1. 運行redis容器
    [root@etcd3 docker_redis_nodes]# docker run -d --name redis-6379 -p 6379:6379 hakimdstx/nodes-redis:4.0.1   
1673a7d859ea83257d5bf14d82ebf717fb31405c185ce96a05f597d8f855aa7d
[root@etcd3 docker_redis_nodes]# docker run -d --name redis-6380 -p 6380:6379 hakimdstx/nodes-redis:4.0.1    
df6ebce6f12a6f3620d5a29adcfbfa7024e906c3af48f21fa7e1fa524a361362
[root@etcd3 docker_redis_nodes]# docker run -d --name redis-6381 -p 6381:6379 hakimdstx/nodes-redis:4.0.1   
396e174a1d9235228b3c5f0266785a12fb1ea49efc7ac755c9e7590e17aa1a79
[root@etcd3 docker_redis_nodes]# docker run -d --name redis-6382 -p 6382:6379 hakimdstx/nodes-redis:4.0.1 
d9a71dd3f969094205ffa7596c4a04255575cdd3acca2d47fe8ef7171a3be528
[root@etcd3 docker_redis_nodes]# docker run -d --name redis-6383 -p 6383:6379 hakimdstx/nodes-redis:4.0.1 
73e4f843d8cb28595456e21b04f97d18ce1cdf8dc56d1150844ba258a3781933
[root@etcd3 docker_redis_nodes]# docker run -d --name redis-6384 -p 6384:6379 hakimdstx/nodes-redis:4.0.1 
10c62aafa4dac47220daf5bf3cec84406f086d5261599b54ec6c56bb7da97d6d
  2. 查看容器信息
    [root@etcd3 redis]# docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS                    NAMES
10c62aafa4da        hakimdstx/nodes-redis:4.0.1   "/usr/local/redis/..."   3 seconds ago       Up 2 seconds        0.0.0.0:6384->6379/tcp   redis-6384
73e4f843d8cb        hakimdstx/nodes-redis:4.0.1   "/usr/local/redis/..."   12 seconds ago      Up 10 seconds       0.0.0.0:6383->6379/tcp   redis-6383
d9a71dd3f969        hakimdstx/nodes-redis:4.0.1   "/usr/local/redis/..."   20 seconds ago      Up 18 seconds       0.0.0.0:6382->6379/tcp   redis-6382
396e174a1d92        hakimdstx/nodes-redis:4.0.1   "/usr/local/redis/..."   3 days ago          Up 3 days           0.0.0.0:6381->6379/tcp   redis-6381
df6ebce6f12a        hakimdstx/nodes-redis:4.0.1   "/usr/local/redis/..."   3 days ago          Up 3 days           0.0.0.0:6380->6379/tcp   redis-6380
1673a7d859ea        hakimdstx/nodes-redis:4.0.1   "/usr/local/redis/..."   3 days ago          Up 3 days           0.0.0.0:6379->6379/tcp   redis-6379

    　　

  3. 運行 redis 集羣容器
    1. 經過遠程鏈接，查看redis  info replication 信息 
      [root@etcd2 ~]#  redis-cli -h 192.168.10.52 -p 6379
192.168.10.52:6379> info replication
NOAUTH Authentication required.
192.168.10.52:6379> auth 123456
OK
192.168.10.52:6379> info replication
# Replication
role:master
connected_slaves:0
master_replid:2f0a7b50aed699fa50a79f3f7f9751a070c50ee9
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
192.168.10.52:6379>
# 其他基本信息同上

      　　能夠看到，客戶鏈接以後，由於以前設置了密碼，因此須要先輸入密碼認證，不然就沒法經過。以上信息，咱們知道全部的redis都是master角色 role:master ，這顯然不是咱們所但願的。

    2. 在配置以前咱們須要查看全部容器當前的IP地址
      [root@etcd3 redis]# docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS                    NAMES
10c62aafa4da        hakimdstx/nodes-redis:4.0.1   "/usr/local/redis/..."   3 seconds ago       Up 2 seconds        0.0.0.0:6384->6379/tcp   redis-6384
73e4f843d8cb        hakimdstx/nodes-redis:4.0.1   "/usr/local/redis/..."   12 seconds ago      Up 10 seconds       0.0.0.0:6383->6379/tcp   redis-6383
d9a71dd3f969        hakimdstx/nodes-redis:4.0.1   "/usr/local/redis/..."   20 seconds ago      Up 18 seconds       0.0.0.0:6382->6379/tcp   redis-6382
396e174a1d92        hakimdstx/nodes-redis:4.0.1   "/usr/local/redis/..."   3 days ago          Up 3 days           0.0.0.0:6381->6379/tcp   redis-6381
df6ebce6f12a        hakimdstx/nodes-redis:4.0.1   "/usr/local/redis/..."   3 days ago          Up 3 days           0.0.0.0:6380->6379/tcp   redis-6380
1673a7d859ea        hakimdstx/nodes-redis:4.0.1   "/usr/local/redis/..."   3 days ago          Up 3 days           0.0.0.0:6379->6379/tcp   redis-6379
[root@etcd3 redis]# 
[root@etcd3 redis]# docker inspect 10c62aafa4da 73e4f843d8cb d9a71dd3f969 396e174a1d92 df6ebce6f12a 1673a7d859ea | grep IPA
            "SecondaryIPAddresses": null,
            "IPAddress": "172.17.0.7",
                    "IPAMConfig": null,
                    "IPAddress": "172.17.0.7",
            "SecondaryIPAddresses": null,
            "IPAddress": "172.17.0.6",
                    "IPAMConfig": null,
                    "IPAddress": "172.17.0.6",
            "SecondaryIPAddresses": null,
            "IPAddress": "172.17.0.5",
                    "IPAMConfig": null,
                    "IPAddress": "172.17.0.5",
            "SecondaryIPAddresses": null,
            "IPAddress": "172.17.0.4",
                    "IPAMConfig": null,
                    "IPAddress": "172.17.0.4",
            "SecondaryIPAddresses": null,
            "IPAddress": "172.17.0.3",
                    "IPAMConfig": null,
                    "IPAddress": "172.17.0.3",
            "SecondaryIPAddresses": null,
            "IPAddress": "172.17.0.2",
                    "IPAMConfig": null,
                    "IPAddress": "172.17.0.2",

      能夠知道：  redis-6379：172.17.0.2，redis-6380：172.17.0.3，redis-6381：172.17.0.4，redis-6382：172.17.0.5，redis-6383：172.17.0.6，redis-6384：172.17.0.7  

    3. 配置redis
    4. ert
  4. Redis Cluster 的集羣感知操做
    //集羣(cluster)  
CLUSTER INFO 打印集羣的信息  
CLUSTER NODES 列出集羣當前已知的全部節點（node），以及這些節點的相關信息。   
  
//節點(node)  
CLUSTER MEET <ip> <port> 將 ip 和 port 所指定的節點添加到集羣當中，讓它成爲集羣的一份子。  
CLUSTER FORGET <node_id> 從集羣中移除 node_id 指定的節點。  
CLUSTER REPLICATE <node_id> 將當前節點設置爲 node_id 指定的節點的從節點。  
CLUSTER SAVECONFIG 將節點的配置文件保存到硬盤裏面。   
  
//槽(slot)  
CLUSTER ADDSLOTS <slot> [slot ...] 將一個或多個槽（slot）指派（assign）給當前節點。  
CLUSTER DELSLOTS <slot> [slot ...] 移除一個或多個槽對當前節點的指派。  
CLUSTER FLUSHSLOTS 移除指派給當前節點的全部槽，讓當前節點變成一個沒有指派任何槽的節點。  
CLUSTER SETSLOT <slot> NODE <node_id> 將槽 slot 指派給 node_id 指定的節點，若是槽已經指派給另外一個節點，那麼先讓另外一個節點刪除該槽>，而後再進行指派。  
CLUSTER SETSLOT <slot> MIGRATING <node_id> 將本節點的槽 slot 遷移到 node_id 指定的節點中。  
CLUSTER SETSLOT <slot> IMPORTING <node_id> 從 node_id 指定的節點中導入槽 slot 到本節點。  
CLUSTER SETSLOT <slot> STABLE 取消對槽 slot 的導入（import）或者遷移（migrate）。   
  
//鍵 (key)  
CLUSTER KEYSLOT <key> 計算鍵 key 應該被放置在哪一個槽上。  
CLUSTER COUNTKEYSINSLOT <slot> 返回槽 slot 目前包含的鍵值對數量。  
CLUSTER GETKEYSINSLOT <slot> <count> 返回 count 個 slot 槽中的鍵。

    　　
    redis 集羣感知：節點握手——是指一批運行在集羣模式的節點經過Gossip協議彼此通訊，達到感知對方的過程。
     

    192.168.10.52:6379> CLUSTER MEET 172.17.0.3 6379
OK
192.168.10.52:6379> CLUSTER MEET 172.17.0.4 6379
OK
192.168.10.52:6379> CLUSTER MEET 172.17.0.5 6379
OK
192.168.10.52:6379> CLUSTER MEET 172.17.0.6 6379
OK
192.168.10.52:6379> CLUSTER MEET 172.17.0.7 6379
OK
192.168.10.52:6379>  CLUSTER NODES
54cb5c2eb8e5f5aed2d2f7843f75a9284ef6785c 172.17.0.3:6379@16379 master - 0 1528697195600 1 connected
f45f9109f2297a83b1ac36f9e1db5e70bbc174ab 172.17.0.4:6379@16379 master - 0 1528697195600 0 connected
ae86224a3bc29c4854719c83979cb7506f37787a 172.17.0.7:6379@16379 master - 0 1528697195600 5 connected
98aebcfe42d8aaa8a3375e4a16707107dc9da683 172.17.0.6:6379@16379 master - 0 1528697194000 4 connected
0bbdc4176884ef0e3bb9b2e7d03d91b0e7e11f44 172.17.0.5:6379@16379 master - 0 1528697194995 3 connected
760e4d0039c5ac13d04aa4791c9e6dc28544d7c7 172.17.0.2:6379@16379 myself,master - 0 1528697195000 2 connected

    　　
    當前已經使這六個節點組成集羣，可是如今還沒法工做，由於集羣節點尚未分配槽（slot）。

    1. 分配槽信息
      查看172.17.0.2:6379 的槽個數
      192.168.10.52:6379> CLUSTER INFO
cluster_state:fail
cluster_slots_assigned:0    # 被分配槽的個數爲0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:0
cluster_current_epoch:5
cluster_my_epoch:2
cluster_stats_messages_ping_sent:260418
cluster_stats_messages_pong_sent:260087
cluster_stats_messages_meet_sent:10
cluster_stats_messages_sent:520515
cluster_stats_messages_ping_received:260086
cluster_stats_messages_pong_received:260328
cluster_stats_messages_meet_received:1
cluster_stats_messages_received:520415

      　　
      上面看到集羣狀態是失敗的，緣由是槽位沒有分配，並且須要一次性把16384個槽位徹底分配了，集羣纔可用。

    2. 分配槽位

      分配槽位： CLUSTER ADDSLOTS  槽位，一個槽位只能分配一個節點，16384個槽位必須分配完，不一樣節點不能衝突。
      因此經過腳本進行分配 addslots.sh：
      #!/bin/bash
# node1 192.168.10.52   172.17.0.2
n=0
for ((i=n;i<=5461;i++))
do
   /usr/local/bin/redis-cli -h 192.168.10.52 -p 6379 -a 123456  CLUSTER ADDSLOTS $i
done


# node2 192.168.10.52    172.17.0.3
n=5462
for ((i=n;i<=10922;i++))
do
   /usr/local/bin/redis-cli -h 192.168.10.52 -p 6380 -a 123456 CLUSTER ADDSLOTS $i
done


# node3 192.168.10.52    172.17.0.4
n=10923
for ((i=n;i<=16383;i++))
do
   /usr/local/bin/redis-cli -h 192.168.10.52 -p 6381 -a 123456 CLUSTER ADDSLOTS $i
done

      　　
      其中， -a 123456  表示須要輸入的密碼。

      192.168.10.52:6379> CLUSTER INFO
cluster_state:fail　　　　       # 集羣狀態爲失敗
cluster_slots_assigned:16101    # 沒有徹底分配結束
cluster_slots_ok:16101
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:5
cluster_my_epoch:2
cluster_stats_messages_ping_sent:266756
cluster_stats_messages_pong_sent:266528
cluster_stats_messages_meet_sent:10
cluster_stats_messages_sent:533294
cluster_stats_messages_ping_received:266527
cluster_stats_messages_pong_received:266666
cluster_stats_messages_meet_received:1
cluster_stats_messages_received:533194
      
192.168.10.52:6379> CLUSTER INFO
cluster_state:ok                   # 集羣狀態爲成功
cluster_slots_assigned:16384       # 已經所有分配完成
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:5
cluster_my_epoch:2
cluster_stats_messages_ping_sent:266757
cluster_stats_messages_pong_sent:266531
cluster_stats_messages_meet_sent:10
cluster_stats_messages_sent:533298
cluster_stats_messages_ping_received:266530
cluster_stats_messages_pong_received:266667
cluster_stats_messages_meet_received:1
cluster_stats_messages_received:533198

      　　

      綜上可知，當所有槽位分配完成以後，集羣仍是可行的，若是咱們手欠，移除一個槽位，那麼集羣就立馬那不行了，本身去試試吧 ——CLUSTER DELSLOTS 0 。

  5. 如何變成高可用性
    以上咱們已經搭建了一套完整的可運行的redis cluster，可是每一個節點都是單點，這樣子可能出現，一個節點掛掉，整個集羣由於槽位分配不徹底而崩潰，所以，咱們須要爲每一個節點配置副本備用節點。
    前面咱們已經提早建立了6個備用節點，搭建集羣花了三個，所以還有剩下三個直接能夠用來作備用副本。
    192.168.10.52:6379> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6   # 總共6個節點
cluster_size:3          # 集羣爲 3 個節點
cluster_current_epoch:5
cluster_my_epoch:2
cluster_stats_messages_ping_sent:270127
cluster_stats_messages_pong_sent:269893
cluster_stats_messages_meet_sent:10
cluster_stats_messages_sent:540030
cluster_stats_messages_ping_received:269892
cluster_stats_messages_pong_received:270037
cluster_stats_messages_meet_received:1
cluster_stats_messages_received:539930

    　　
    查看全部節點的id

    192.168.10.52:6379> CLUSTER NODES
54cb5c2eb8e5f5aed2d2f7843f75a9284ef6785c 172.17.0.3:6379@16379 master - 0 1528704114535 1 connected 5462-10922
f45f9109f2297a83b1ac36f9e1db5e70bbc174ab 172.17.0.4:6379@16379 master - 0 1528704114000 0 connected 10923-16383
ae86224a3bc29c4854719c83979cb7506f37787a 172.17.0.7:6379@16379 master - 0 1528704114023 5 connected
98aebcfe42d8aaa8a3375e4a16707107dc9da683 172.17.0.6:6379@16379 master - 0 1528704115544 4 connected
0bbdc4176884ef0e3bb9b2e7d03d91b0e7e11f44 172.17.0.5:6379@16379 master - 0 1528704114836 3 connected
760e4d0039c5ac13d04aa4791c9e6dc28544d7c7 172.17.0.2:6379@16379 myself,master - 0 1528704115000 2 connected 0-5461

    　　
    編寫腳本，添加副本節點

    [root@etcd2 tmp]# vi addSlaveNodes.sh 
#!/bin/bash

/usr/local/bin/redis-cli -h 192.168.10.52 -p 6382 -a 123456 CLUSTER REPLICATE 760e4d0039c5ac13d04aa4791c9e6dc28544d7c7

/usr/local/bin/redis-cli -h 192.168.10.52 -p 6383 -a 123456 CLUSTER REPLICATE 54cb5c2eb8e5f5aed2d2f7843f75a9284ef6785c

/usr/local/bin/redis-cli -h 192.168.10.52 -p 6384 -a 123456 CLUSTER REPLICATE f45f9109f2297a83b1ac36f9e1db5e70bbc174ab

    　　
    注意：一、做爲備用的節點，必須是未分配槽位的，否者會操做失敗 (error) ERR To set a master the node must be empty and without assigned slots 。
               二、須要從須要添加的節點上面執行操做，CLUSTER REPLICATE [node_id]  ，使當前節點成爲 node_id 的副本節點。
               三、添加從節點（集羣複製）： 複製的原理和單機的Redis複製原理同樣，區別是：集羣下的從節點也須要運行在cluster模式下，要先添加到集羣裏面，再作複製。
    查看全部節點信息：

    192.168.10.52:6379> CLUSTER NODES
54cb5c2eb8e5f5aed2d2f7843f75a9284ef6785c 172.17.0.3:6379@16379 master - 0 1528705604149 1 connected 5462-10922
f45f9109f2297a83b1ac36f9e1db5e70bbc174ab 172.17.0.4:6379@16379 master - 0 1528705603545 0 connected 10923-16383
ae86224a3bc29c4854719c83979cb7506f37787a 172.17.0.7:6379@16379 slave f45f9109f2297a83b1ac36f9e1db5e70bbc174ab 0 1528705603144 5 connected
98aebcfe42d8aaa8a3375e4a16707107dc9da683 172.17.0.6:6379@16379 slave 54cb5c2eb8e5f5aed2d2f7843f75a9284ef6785c 0 1528705603000 4 connected
0bbdc4176884ef0e3bb9b2e7d03d91b0e7e11f44 172.17.0.5:6379@16379 slave 760e4d0039c5ac13d04aa4791c9e6dc28544d7c7 0 1528705603000 3 connected
760e4d0039c5ac13d04aa4791c9e6dc28544d7c7 172.17.0.2:6379@16379 myself,master - 0 1528705602000 2 connected 0-5461

    　　
    能夠看到咱們如今實現了三主三從的一個高可用集羣。

  6. 高可用測試——故障轉移
    查看當前運行狀態：
    192.168.10.52:6379> CLUSTER NODES
54cb5c2eb8e5f5aed2d2f7843f75a9284ef6785c 172.17.0.3:6379@16379 master - 0 1528705604149 1 connected 5462-10922
f45f9109f2297a83b1ac36f9e1db5e70bbc174ab 172.17.0.4:6379@16379 master - 0 1528705603545 0 connected 10923-16383
ae86224a3bc29c4854719c83979cb7506f37787a 172.17.0.7:6379@16379 slave f45f9109f2297a83b1ac36f9e1db5e70bbc174ab 0 1528705603144 5 connected
98aebcfe42d8aaa8a3375e4a16707107dc9da683 172.17.0.6:6379@16379 slave 54cb5c2eb8e5f5aed2d2f7843f75a9284ef6785c 0 1528705603000 4 connected
0bbdc4176884ef0e3bb9b2e7d03d91b0e7e11f44 172.17.0.5:6379@16379 slave 760e4d0039c5ac13d04aa4791c9e6dc28544d7c7 0 1528705603000 3 connected
760e4d0039c5ac13d04aa4791c9e6dc28544d7c7 172.17.0.2:6379@16379 myself,master - 0 1528705602000 2 connected 0-5461

    　　以上，運行正常

    嘗試關閉一個master，選擇端口爲6380的容器，停掉以後：

    192.168.10.52:6379> CLUSTER NODES
54cb5c2eb8e5f5aed2d2f7843f75a9284ef6785c 172.17.0.3:6379@16379 master,fail - 1528706408935 1528706408000 1 connected 5462-10922
f45f9109f2297a83b1ac36f9e1db5e70bbc174ab 172.17.0.4:6379@16379 master - 0 1528706463000 0 connected 10923-16383
ae86224a3bc29c4854719c83979cb7506f37787a 172.17.0.7:6379@16379 slave f45f9109f2297a83b1ac36f9e1db5e70bbc174ab 0 1528706462980 5 connected
98aebcfe42d8aaa8a3375e4a16707107dc9da683 172.17.0.6:6379@16379 slave 54cb5c2eb8e5f5aed2d2f7843f75a9284ef6785c 0 1528706463000 4 connected
0bbdc4176884ef0e3bb9b2e7d03d91b0e7e11f44 172.17.0.5:6379@16379 slave 760e4d0039c5ac13d04aa4791c9e6dc28544d7c7 0 1528706463985 3 connected
760e4d0039c5ac13d04aa4791c9e6dc28544d7c7 172.17.0.2:6379@16379 myself,master - 0 1528706462000 2 connected 0-5461
192.168.10.52:6379> 
192.168.10.52:6379> CLUSTER INFO
cluster_state:fail
cluster_slots_assigned:16384
cluster_slots_ok:10923
cluster_slots_pfail:0
cluster_slots_fail:5461
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:5
cluster_my_epoch:2
cluster_stats_messages_ping_sent:275112
cluster_stats_messages_pong_sent:274819
cluster_stats_messages_meet_sent:10
cluster_stats_messages_fail_sent:5
cluster_stats_messages_sent:549946
cluster_stats_messages_ping_received:274818
cluster_stats_messages_pong_received:275004
cluster_stats_messages_meet_received:1
cluster_stats_messages_fail_received:1
cluster_stats_messages_received:549824

    　　以上，發現整個集羣都失敗了，從節點沒有自動升級爲主節點，怎麼回事？？
    重啓停掉的容器，經排查日誌信息 [root@df6ebce6f12a /]# tail -f /var/log/redis/redis-server.log  ：

    1:S 11 Jun 09:57:46.712 # Cluster state changed: ok
1:S 11 Jun 09:57:46.718 * (Non critical) Master does not understand REPLCONF listening-port: -NOAUTH Authentication required.
1:S 11 Jun 09:57:46.718 * (Non critical) Master does not understand REPLCONF capa: -NOAUTH Authentication required.
1:S 11 Jun 09:57:46.719 * Partial resynchronization not possible (no cached master)
1:S 11 Jun 09:57:46.719 # Unexpected reply to PSYNC from master: -NOAUTH Authentication required.
1:S 11 Jun 09:57:46.719 * Retrying with SYNC...
1:S 11 Jun 09:57:46.719 # MASTER aborted replication with an error: NOAUTH Authentication required.
1:S 11 Jun 09:57:46.782 * Connecting to MASTER 172.17.0.6:6379
1:S 11 Jun 09:57:46.782 * MASTER <-> SLAVE sync started
1:S 11 Jun 09:57:46.782 * Non blocking connect for SYNC fired the event.


    能夠看到，主從之間訪問須要auth，以前忘記了配置 redis.conf  中的 # masterauth <master-password> ，因此致使主從之間沒法通信。修改配置以後，自動故障轉移正常。


    有時候須要實施人工故障轉移：

    登陸6380端口的從節點：6383，執行 CLUSTER FAILOVER 命令：
    192.168.10.52:6383> CLUSTER  FAILOVER
(error) ERR Master is down or failed, please use CLUSTER FAILOVER FORCE

    　　
    發現由於master已經down了，因此咱們須要執行強制轉移

    192.168.10.52:6383> CLUSTER FAILOVER FORCE
OK

    　　
    查看當前 cluster node 狀況：

    192.168.10.52:6383>  CLUSTER NODES
0bbdc4176884ef0e3bb9b2e7d03d91b0e7e11f44 172.17.0.5:6379@16379 slave 760e4d0039c5ac13d04aa4791c9e6dc28544d7c7 0 1528707535332 3 connected
ae86224a3bc29c4854719c83979cb7506f37787a 172.17.0.7:6379@16379 slave f45f9109f2297a83b1ac36f9e1db5e70bbc174ab 0 1528707534829 5 connected
f45f9109f2297a83b1ac36f9e1db5e70bbc174ab 172.17.0.4:6379@16379 master - 0 1528707534527 0 connected 10923-16383
98aebcfe42d8aaa8a3375e4a16707107dc9da683 172.17.0.6:6379@16379 myself,master - 0 1528707535000 6 connected 5462-10922
760e4d0039c5ac13d04aa4791c9e6dc28544d7c7 172.17.0.2:6379@16379 master - 0 1528707535834 2 connected 0-5461
54cb5c2eb8e5f5aed2d2f7843f75a9284ef6785c 172.17.0.3:6379@16379 master,fail - 1528707472833 1528707472000 1 connected

    　　
    從節點已經升級爲master節點。這時候，咱們嘗試重啓了，6380節點的redis（實際上是從新啓動停掉的容器）：

    192.168.10.52:6383>  CLUSTER NODES
0bbdc4176884ef0e3bb9b2e7d03d91b0e7e11f44 172.17.0.5:6379@16379 slave 760e4d0039c5ac13d04aa4791c9e6dc28544d7c7 0 1528707556044 3 connected
ae86224a3bc29c4854719c83979cb7506f37787a 172.17.0.7:6379@16379 slave f45f9109f2297a83b1ac36f9e1db5e70bbc174ab 0 1528707555000 5 connected
f45f9109f2297a83b1ac36f9e1db5e70bbc174ab 172.17.0.4:6379@16379 master - 0 1528707556000 0 connected 10923-16383
98aebcfe42d8aaa8a3375e4a16707107dc9da683 172.17.0.6:6379@16379 myself,master - 0 1528707556000 6 connected 5462-10922
760e4d0039c5ac13d04aa4791c9e6dc28544d7c7 172.17.0.2:6379@16379 master - 0 1528707556000 2 connected 0-5461
54cb5c2eb8e5f5aed2d2f7843f75a9284ef6785c 172.17.0.3:6379@16379 slave 98aebcfe42d8aaa8a3375e4a16707107dc9da683 0 1528707556547 6 connected

    　　

    咱們發現，6380節點反而變成了 6383節點的從節點。

    如今集羣應該是完整的了，因此，集羣狀態應該已經恢復了，咱們查看下：
    192.168.10.52:6383> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:6
cluster_stats_messages_ping_sent:19419
cluster_stats_messages_pong_sent:19443
cluster_stats_messages_meet_sent:1
cluster_stats_messages_auth-req_sent:5
cluster_stats_messages_update_sent:1
cluster_stats_messages_sent:38869
cluster_stats_messages_ping_received:19433
cluster_stats_messages_pong_received:19187
cluster_stats_messages_meet_received:5
cluster_stats_messages_fail_received:4
cluster_stats_messages_auth-ack_received:2
cluster_stats_messages_received:38631

    　　

    OK，沒有問題。

  7. 集羣訪問
    客戶端在初始化的時候只須要知道一個節點的地址便可，客戶端會先嚐試向這個節點執行命令，好比  get key ，若是key所在的slot恰好在該節點上，則可以直接執行成功。若是slot不在該節點，則節點會返回MOVED錯誤，同時把該slot對應的節點告訴客戶端，客戶端能夠去該節點執行命令
     
    192.168.10.52:6383> get hello
(error) MOVED 866 172.17.0.2:6379

192.168.10.52:6379> set number 20004
(error) MOVED 7743 172.17.0.3:6379

    　　另外，redis集羣版只使用db0，select命令雖然可以支持select 0。其餘的db都會返回錯誤。

    192.168.10.52:6383> select 0
OK
192.168.10.52:6383> select 1
(error) ERR SELECT is not allowed in cluster mode

     

 

  1. 近期，有網友詢問docker redis集羣鏈接報錯的問題，具體報錯以下：

    　

     

    初步認爲是，node節點沒有所有添加進去，添加以後，依然有上述問題。想到是跨主機訪問，應該是路由尋址不了致使的。當初寫上述教程的時候，docker是以默認的網絡模式bridge模式運行的，畢竟當初是以學習整理文檔爲主，主要是單機訪問。可是，實際應用化場景中，可能是公網跨主機訪問，問題明朗了，想着集羣這東西最好仍是設置成共享主機公網ip比較好，因而解決以下：
    1. 在docker運行時，執行網絡模式爲：host。
    2. 端口衝突解決，畢竟host模式下，容器會佔用宿主機的端口，因而，咱們就從配置下手，在宿主機上生成配置redis-60001.conf,redis-60002.conf,redis-60003.conf...，有多少端口建多少個文件，最終運行一個容器，掛載一個配置到容器中用於覆蓋主機中的配置。
      最終的運行方式以下： 
       docker run -d --name redis-6380 --net host -v /tmp/redis.conf:/usr/local/redis/redis.conf  hakimdstx/nodes-redis:4.0.1


      至此，網絡問題獲得解決。
      PS.生產環境須要注意防火牆問題，否則也是會報錯的。　　

以上

引用：

     一、Redis Cluster部署、管理和測試

     二、Docker下redis的主從、持久化配置

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程