Redis cluster集羣：原理及搭建

1.爲何使用redis？

redis是一種典型的no-sql 即非關係數據庫 像python的字典同樣 存儲key-value鍵值對 工做在memory中

因此很適合用來充當整個互聯網架構中各級之間的cache 好比lvs的4層轉發層 nginx的7層代理層

尤爲是lnmp架構應用層如php-fpm或者是Tomcat到mysql之間 作一個cache 以減輕db的壓力

由於有至關一部分的數據 只是簡單的key-value對應關係，並且在實際的業務中經常在短期內迅速變更 

若是用關係數據庫mysql之類存儲 會大大增長對db的訪問 致使db的負擔很重 由於全部的require中的大部分最後都要匯聚到db

因此若是想要業務穩定 那麼解決db的壓力 就是關鍵 因此如今大部分的解決方案就是在db層之上的各級使用多級的no-sql 

像memcache redis 等 來爲db提供緩衝  

2.爲何使用redis-cluster？

爲了在大流量訪問下提供穩定的業務，集羣化是存儲的必然形態

將來的發展趨勢確定是雲計算和大數據的緊密結合 只有分佈式架構能知足要求

若是沒有集羣化 何來的分佈式？

3.順帶一提總結一波今天的redis原理之數據持久化

雖然redis這種no-sql通常都是做爲cache來服務  可是若是徹底沒有數據可持久化的方法 那麼顯得有些單薄

就像memcache 因爲這種no-sql是工做在memory的 那麼因爲memory的實體是ram

因此若是重啓或者宕機 memory中的數據就全沒了 數據的一致性的不到保障 

可是 redis不一樣 redis有相對的數據持久化的方案 由兩種方式構成 aof & rdb

aof就像關係數據庫中的binlog同樣  把每一次寫操做以追加的形式記錄在其中以文件的形式刷到磁盤裏 

而且可使用不一樣的fsync策略     無fsync,每秒fsync,每次寫的時候fsync.

使用默認的每秒fsync策略,Redis的性能依然很好(fsync是由後臺線程進行處理的,主線程會盡力處理客戶端請求)

一旦出現故障，最多丟失1秒的數據.

可是缺點也隨之而來 那就是aof文件的大小會隨着時間線性增加 一段時間以後 就會變得很大 

若是要在一端以AOF的形式來恢復數據 那麼因爲AOF文件的巨大致積 可能會讓進程如同假死同樣 十分的慢

rdb則是一種快照機制 

redis工做在內存中 rdb就是每隔一段時間 對內存中的數據作一次快照  保存在rdb文件中 

並且redis的主從同步能夠實現異步 也是因爲rdb的機制 他在作快照時會fork出一個子進程 由子進程來作快照

父進程徹底處理請求 絕不影響 很適合數據的備份 

可是問題是 若是數據量很大的話 rdb它要保存一個完整的數據集 是一個大的工做 若是時間間隔設置的過短 

那麼嚴重影響redis的性能 可是按照常規設置的話 如5分鐘一次 那麼若是宕機或者重啓 就會基於上次作rdb的時間

從而丟失分鐘級的數據

point：在redis4.0的新特性中 採用了aof-rdb的混合方案來保障數據的持久性 可是官方的說法是還不成熟

是一個長期的工做  因此有待觀察吧 

4.redis集羣實現方案： 

關於redis的集羣化方案 目前有三種 

（1）Twitter開發的twemproxy

（2）豌豆莢開發的codis

（3）redis官方的redis-cluster

簡介：twemproxy架構簡單 就是用proxy對後端redis server進行代理 可是因爲代理層的消耗性能很低 並且一般涉及多個key的操做都是不支持的 並且自己不支持動態擴容和透明的數據遷移 並且也失去維護 Twitter內部已經不使用了

          redis-cluster是三個裏性能最強大的 由於他使用去中心化的思想 使用hash slot方式 將16348個hash slot 覆蓋到全部節點上 對於存儲的每一個key值 使用CRC16(KEY)&16348=slot 獲得他對應的hash slot 並在訪問key時就去找他的hash slot在哪個節點上 而後由當前訪問節點從實際被分配了這個hash slot的節點去取數據 節點之間使用輕量協議通訊 減小帶寬佔用 性能很高 自動實現負載均衡與高可用 自動實現failover  而且支持動態擴展 官方已經玩到能夠1000個節點 實現的複雜度低 總之我的比較喜歡這個架構 由於他的去中心化思想免去了proxy的消耗 是全新的思路

           可是它也有一些不足 例如官方沒有提供圖形化管理工具 運維體驗差 全手工數據遷移 而且本身對本身自己的redis命令支持也不徹底等 可是這些問題 我以爲不能掩蓋他關鍵的新思想所帶來的的優點 隨着官方的推動 這些問題應該都能在必定時間內獲得解決 那麼這時候去中心化思想帶來的高性能就會表現出他巨大的優點 

          codis使用的也是proxy思路 可是作的比較好 是這兩種之間的一箇中間級 並且支持redis命令是最多的 有圖形化GUI管理和監控工具 運維友好 這個過段時間會詳細另外寫出來原理 工做機制和搭建實現

 

 

Redis集羣

基本介紹

Redis 集羣是一個能夠在多個 Redis 節點之間進行數據共享的設施installation。

Redis 集羣不支持那些須要同時處理多個鍵的 Redis 命令， 由於執行這些命令須要在多個 Redis 節點之間移動數據， 而且在高負載的狀況下， 這些命令將下降Redis集羣的性能， 並致使不可預測的行爲。

Redis 集羣經過分區partition來提供必定程度的可用性availability： 即便集羣中有一部分節點失效或者沒法進行通信， 集羣也能夠繼續處理命令請求。

Redis集羣提供瞭如下兩個好處：

• 將數據自動切分split到多個節點的能力。
• 當集羣中的一部分節點失效或者沒法進行通信時， 仍然能夠繼續處理命令請求的能力。

集羣原理

redis-cluster架構圖

1. 全部的redis節點彼此互聯(PING-PONG機制),內部使用二進制協議優化傳輸速度和帶寬。

2. 節點的fail是經過集羣中超過半數的節點檢測失效時才生效。

3. 客戶端與redis節點直連,不須要中間proxy層.客戶端不須要鏈接集羣全部節點,鏈接集羣中任何一個可用節點便可。

4. redis-cluster把全部的物理節點映射到[0-16383]slot上,cluster 負責維護node<->slot<->value

Redis集羣中內置了 16384 個哈希槽，當須要在 Redis 集羣中放置一個 key-value 時，redis 先對key 使用 crc16 算法算出一個結果，而後把結果對 16384 求餘數，這樣每一個 key 都會對應一個編號在 0-16383 之間的哈希槽，redis 會根據節點數量大體均等的將哈希槽映射到不一樣的節點

redis-cluster投票:容錯

1. 投票過程是集羣中全部master參與,若是半數以上master節點與master節點通訊超時(cluster-node-timeout),認爲當前master節點掛掉.

2. 何時整個集羣不可用(cluster_state:fail)?

   • 若是集羣任意master掛掉,且當前master沒有slave.集羣進入fail狀態,也能夠理解成集羣的slot映射[0-16383]不完整時進入fail狀態. 
     
     • redis-3.0.0.rc1加入cluster-require-full-coverage參數,默認關閉,打開集羣兼容部分失敗.
   • 若是集羣超過半數以上master掛掉，不管是否有slave，集羣進入fail狀態.

Redis集羣搭建

Redis單機版的安裝見博客《redis入門——安裝篇》，安裝好以後，將redis複製成6份，注意要將.rdb和.aof後綴的文件刪除，若是有的話。

Ruby環境

使用

1.  
   yum -y install ruby
2.  
   yum -y install rubygems

安裝ruby環境。 
網上下載redis-3.0.0.gem，執行gem install redis-3.0.0.gem安裝。

redis配置文件修改

如今已經準備好了，6份乾淨的redis，以下所示

1.  
   [root @localhost redis-cluster]# pwd
2.  
   /usr/ local/redis/redis-cluster
3.  
   [root @localhost redis-cluster]# ll
4.  
   total 72
5.  
   drwxr-xr- x 2 root root 4096 Nov 2 00:17 redis1
6.  
   drwxr-xr- x 2 root root 4096 Nov 2 00:25 redis2
7.  
   drwxr-xr- x 2 root root 4096 Nov 2 00:25 redis3
8.  
   drwxr-xr- x 2 root root 4096 Nov 2 00:25 redis4
9.  
   drwxr-xr- x 2 root root 4096 Nov 2 00:25 redis5
10.  
    drwxr-xr- x 2 root root 4096 Nov 2 00:25 redis6
11.  
    -rwxr-xr- x 1 root root 48141 Nov 2 00:16 redis-trib.rb
12.  
    [root @localhost redis-cluster]#

PS：注意，這裏已經將redis源文件src目錄下的redis-trib.rb文件拷貝過來了。 
redis-trib.rb這個文件是redis集羣的管理文件，ruby腳本。

將六個節點的redis.conf配置文件按照以下進行修改

1.  
   ################################ GENERAL #####################################
2.  
    
3.  
   # By default Redis does not run as a daemon. Use 'yes' if you need it.
4.  
   # Note that Redis will write a pid file in /var/run/redis.pid when daemonized.
5.  
   daemonize yes
6.  
    
7.  
   # Accept connections on the specified port, default is 6379.
8.  
   # If port 0 is specified Redis will not listen on a TCP socket.
9.  
   port *
10.  
     
11.  
    ################################ REDIS CLUSTER ###############################
12.  
    #
13.  
    # ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
14.  
    # WARNING EXPERIMENTAL: Redis Cluster is considered to be stable code, however
15.  
    # in order to mark it as "mature" we need to wait for a non trivial percentage
16.  
    # of users to deploy it in production.
17.  
    # ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
18.  
    #
19.  
    # Normal Redis instances can't be part of a Redis Cluster; only nodes that are
20.  
    # started as cluster nodes can. In order to start a Redis instance as a
21.  
    # cluster node enable the cluster support uncommenting the following:
22.  
    #
23.  
    cluster-enabled yes

PS：端口號若是是同一臺主機的話，必須不一樣。不一樣主機能夠相同。

PS：我這裏是使用一臺主機，因此我將六個節點的端口號修改成7001-7006

編寫集羣啓動腳本和中止腳本

啓動腳本start-all.sh

1.  
   cd redis1
2.  
   ./redis-server redis.conf
3.  
   cd ..
4.  
   cd redis2
5.  
   ./redis-server redis.conf
6.  
   cd ..
7.  
   cd redis3
8.  
   ./redis-server redis.conf
9.  
   cd ..
10.  
    cd redis4
11.  
    ./redis-server redis.conf
12.  
    cd ..
13.  
    cd redis5
14.  
    ./redis-server redis.conf
15.  
    cd ..
16.  
    cd redis6
17.  
    ./redis-server redis.conf
18.  
    cd ..

中止腳本stop-all.sh

1.  
   ./redis1/redis-cli -p 7001 shutdown
2.  
   ./redis1/redis-cli -p 7002 shutdown
3.  
   ./redis1/redis-cli -p 7003 shutdown
4.  
   ./redis1/redis-cli -p 7004 shutdown
5.  
   ./redis1/redis-cli -p 7005 shutdown
6.  
   ./redis1/redis-cli -p 7006 shutdown

PS：兩個腳本都放在以下所屬目錄

1.  
   [root @localhost redis-cluster]# pwd
2.  
   /usr/ local/redis/redis-cluster
3.  
   [root @localhost redis-cluster]# ll
4.  
   total 80
5.  
   drwxr-xr- x 2 root root 4096 Nov 2 00:52 redis1
6.  
   drwxr-xr- x 2 root root 4096 Nov 2 00:51 redis2
7.  
   drwxr-xr- x 2 root root 4096 Nov 2 00:53 redis3
8.  
   drwxr-xr- x 2 root root 4096 Nov 2 00:53 redis4
9.  
   drwxr-xr- x 2 root root 4096 Nov 2 00:53 redis5
10.  
    drwxr-xr- x 2 root root 4096 Nov 2 00:53 redis6
11.  
    -rwxr-xr- x 1 root root 48141 Nov 2 00:16 redis-trib.rb
12.  
    -rw-r--r-- 1 root root 252 Nov 2 00:55 start-all.sh
13.  
    -rw-r--r-- 1 root root 216 Nov 2 00:57 stop-all.sh
14.  
    [root @localhost redis-cluster]#

修改權限

[root@localhost redis-cluster]# chmod -u+x start-all.sh stop-all.sh 

啓動節點

1.  
   [root @localhost redis-cluster]# ./start-all.sh
2.  
   [root @localhost redis-cluster]# ps aux | grep redis
3.  
   root 2924 0.8 0.1 33932 2048 ? Ssl Nov01 3:53 ./redis-server *:6379 [cluster]
4.  
   root 11924 0.0 0.1 33936 1948 ? Ssl 01:01 0:00 ./redis-server *:7001 [cluster]
5.  
   root 11928 0.0 0.1 33936 1952 ? Ssl 01:01 0:00 ./redis-server *:7002 [cluster]
6.  
   root 11932 0.0 0.1 33936 1948 ? Ssl 01:01 0:00 ./redis-server *:7003 [cluster]
7.  
   root 11936 0.0 0.1 33936 1952 ? Ssl 01:01 0:00 ./redis-server *:7004 [cluster]
8.  
   root 11940 0.0 0.1 33936 1952 ? Ssl 01:01 0:00 ./redis-server *:7005 [cluster]
9.  
   root 11944 0.0 0.1 33936 1948 ? Ssl 01:01 0:00 ./redis-server *:7006 [cluster]
10.  
    root 11948 0.0 0.0 4360 748 pts/2 S+ 01:01 0:00 grep redis
11.  
    [root @localhost redis-cluster]#

執行建立集羣命令

1.  
   [root @localhost redis-cluster]# pwd
2.  
   /usr/ local/redis/redis-cluster
3.  
   [root @localhost redis-cluster]# ./redis-trib.rb create --replicas 1 192.168.37.131:7001 192.168.37.131:7002 192.168.37.131:7003 192.168.37.131:7004 192.168.37.131:7005 192.168.37.131:7006

成功啓動信息

1.  
   >>> Creating cluster
2.  
   Connecting to node 192.168.37.131:7001: OK
3.  
   Connecting to node 192.168.37.131:7002: OK
4.  
   Connecting to node 192.168.37.131:7003: OK
5.  
   Connecting to node 192.168.37.131:7004: OK
6.  
   Connecting to node 192.168.37.131:7005: OK
7.  
   Connecting to node 192.168.37.131:7006: OK
8.  
   >>> Performing hash slots allocation on 6 nodes...
9.  
   Using 3 masters:
10.  
    192.168.37.131:7001
11.  
    192.168.37.131:7002
12.  
    192.168.37.131:7003
13.  
    Adding replica 192.168.37.131:7004 to 192.168.37.131:7001
14.  
    Adding replica 192.168.37.131:7005 to 192.168.37.131:7002
15.  
    Adding replica 192.168.37.131:7006 to 192.168.37.131:7003
16.  
    M: 8b153503b52f83634e04b0077f32ef629ad91ee6 192.168.37.131:7001
17.  
    slots: 0-5460 (5461 slots) master
18.  
    M: f89799066dd8ecaaa1430559be4ce9c8c87055d8 192.168.37.131:7002
19.  
    slots: 5461-10922 (5462 slots) master
20.  
    M: 53d698ad56b09f89cfef34850213e2d0a44154dd 192.168.37.131:7003
21.  
    slots: 10923-16383 (5461 slots) master
22.  
    S: e73204399d08c14def1f71d0c5377cbc757dc4b8 192.168.37.131:7004
23.  
    replicates 8b153503b52f83634e04b0077f32ef629ad91ee6
24.  
    S: 1d5dcc8d1ccb6bce55efc3e3aadc690dc77808d8 192.168.37.131:7005
25.  
    replicates f89799066dd8ecaaa1430559be4ce9c8c87055d8
26.  
    S: e9458233cb85bd897ff694003e6d8a834eba2b44 192.168.37.131:7006
27.  
    replicates 53d698ad56b09f89cfef34850213e2d0a44154dd
28.  
    Can I set the above configuration? (type 'yes' to accept): y
29.  
    *** Aborting...
30.  
    [root@localhost redis-cluster] # ./redis-trib.rb create --replicas 1 192.168.37.131:7001 192.168.37.131:7002 192.168.37.131:7003 192.168.37.131:7004 192.168.37.131:7005 192.168.37.131:7006
31.  
    >>> Creating cluster
32.  
    Connecting to node 192.168.37.131:7001: OK
33.  
    Connecting to node 192.168.37.131:7002: OK
34.  
    Connecting to node 192.168.37.131:7003: OK
35.  
    Connecting to node 192.168.37.131:7004: OK
36.  
    Connecting to node 192.168.37.131:7005: OK
37.  
    Connecting to node 192.168.37.131:7006: OK
38.  
    >>> Performing hash slots allocation on 6 nodes...
39.  
    Using 3 masters:
40.  
    192.168.37.131:7001
41.  
    192.168.37.131:7002
42.  
    192.168.37.131:7003
43.  
    Adding replica 192.168.37.131:7004 to 192.168.37.131:7001
44.  
    Adding replica 192.168.37.131:7005 to 192.168.37.131:7002
45.  
    Adding replica 192.168.37.131:7006 to 192.168.37.131:7003
46.  
    M: 8b153503b52f83634e04b0077f32ef629ad91ee6 192.168.37.131:7001
47.  
    slots: 0-5460 (5461 slots) master
48.  
    M: f89799066dd8ecaaa1430559be4ce9c8c87055d8 192.168.37.131:7002
49.  
    slots: 5461-10922 (5462 slots) master
50.  
    M: 53d698ad56b09f89cfef34850213e2d0a44154dd 192.168.37.131:7003
51.  
    slots: 10923-16383 (5461 slots) master
52.  
    S: e73204399d08c14def1f71d0c5377cbc757dc4b8 192.168.37.131:7004
53.  
    replicates 8b153503b52f83634e04b0077f32ef629ad91ee6
54.  
    S: 1d5dcc8d1ccb6bce55efc3e3aadc690dc77808d8 192.168.37.131:7005
55.  
    replicates f89799066dd8ecaaa1430559be4ce9c8c87055d8
56.  
    S: e9458233cb85bd897ff694003e6d8a834eba2b44 192.168.37.131:7006
57.  
    replicates 53d698ad56b09f89cfef34850213e2d0a44154dd
58.  
    Can I set the above configuration? (type 'yes' to accept): yes
59.  
    >>> Nodes configuration updated
60.  
    >>> Assign a different config epoch to each node
61.  
    >>> Sending CLUSTER MEET messages to join the cluster
62.  
    Waiting for the cluster to join.....
63.  
    >>> Performing Cluster Check ( using node 192.168.37.131:7001)
64.  
    M: 8b153503b52f83634e04b0077f32ef629ad91ee6 192.168.37.131:7001
65.  
    slots: 0-5460 (5461 slots) master
66.  
    M: f89799066dd8ecaaa1430559be4ce9c8c87055d8 192.168.37.131:7002
67.  
    slots: 5461-10922 (5462 slots) master
68.  
    M: 53d698ad56b09f89cfef34850213e2d0a44154dd 192.168.37.131:7003
69.  
    slots: 10923-16383 (5461 slots) master
70.  
    M: e73204399d08c14def1f71d0c5377cbc757dc4b8 192.168.37.131:7004
71.  
    slots: ( 0 slots) master
72.  
    replicates 8b153503b52f83634e04b0077f32ef629ad91ee6
73.  
    M: 1d5dcc8d1ccb6bce55efc3e3aadc690dc77808d8 192.168.37.131:7005
74.  
    slots: ( 0 slots) master
75.  
    replicates f89799066dd8ecaaa1430559be4ce9c8c87055d8
76.  
    M: e9458233cb85bd897ff694003e6d8a834eba2b44 192.168.37.131:7006
77.  
    slots: ( 0 slots) master
78.  
    replicates 53d698ad56b09f89cfef34850213e2d0a44154dd
79.  
    [OK] All nodes agree about slots configuration.
80.  
    >>> Check for open slots...
81.  
    >>> Check slots coverage...
82.  
    [OK] All 16384 slots covered.
83.  
    [root@localhost redis-cluster] #

異常

1.  
   >>> Creating cluster
2.  
   Connecting to node 192.168.37.131:7001: OK
3.  
   [ERR] Node 192.168.37.131:7001 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

解決方法是刪除生成的配置文件nodes.conf，若是不行則說明如今建立的結點包括了舊集羣的結點信息，須要刪除redis的持久化文件後再重啓redis，好比：appendonly.aof、dump.rdb

首先，使用stop-all.sh中止服務

1.  
   [root @localhost redis-cluster]# ./stop-all.sh
2.  
   [root @localhost redis-cluster]# ps aux | grep redis
3.  
   root 2924 0.8 0.1 33932 2048 ? Ssl Nov01 3:54 ./redis-server *:6379 [cluster]
4.  
   root 11969 0.0 0.0 4360 744 pts/2 S+ 01:10 0:00 grep redis
5.  
   [root @localhost redis-cluster]#

而後每一個節點中的appendonly.aof、dump.rdb 、nodes.conf。

1.  
   [root @localhost redis-cluster]# rm -f redis*/dump.rdb
2.  
   [root @localhost redis-cluster]# rm -f redis*/appendonly.aof
3.  
   [root @localhost redis-cluster]# rm -f redis*/nodes.conf

而後使用腳本start-all.sh啓動，再啓動集羣管理服務。

1.  
   [root @localhost redis-cluster]# ./start-all.sh
2.  
   [root @localhost redis-cluster]# ps aux | grep redis
3.  
   root 2924 0.8 0.1 33932 2048 ? Ssl Nov01 3:54 ./redis-server *:6379 [cluster]
4.  
   root 11980 0.0 0.1 33936 1952 ? Ssl 01:12 0:00 ./redis-server *:7001 [cluster]
5.  
   root 11982 0.0 0.1 33936 1952 ? Ssl 01:12 0:00 ./redis-server *:7002 [cluster]
6.  
   root 11984 0.0 0.1 33936 1952 ? Ssl 01:12 0:00 ./redis-server *:7003 [cluster]
7.  
   root 11986 0.0 0.1 33936 1952 ? Ssl 01:12 0:00 ./redis-server *:7004 [cluster]
8.  
   root 11988 0.0 0.1 33936 1948 ? Ssl 01:12 0:00 ./redis-server *:7005 [cluster]
9.  
   root 11990 0.0 0.1 33936 1948 ? Ssl 01:12 0:00 ./redis-server *:7006 [cluster]
10.  
    root 12004 0.0 0.0 4360 748 pts/2 S+ 01:12 0:00 grep redis
11.  
    [root @localhost redis-cluster]# ./redis-trib.rb create --replicas 1 192.168.37.131:7001 192.168.37.131:7002 192.168.37.131:7003 192.168.37.131:7004 192.168.37.131:7005 192.168.37.131:7006
12.  
    [root @localhost redis-cluster]#

Redis集羣節點的操做

查詢集羣信息

集羣建立成功登錄任意redis結點查詢集羣中的節點狀況。

客戶端以集羣方式登錄：

1.  
   [root @localhost redis-cluster]# ./redis1/redis-cli -c -h 192.168.37.131 -p 7001
2.  
   192.168.37.131:7001>

說明： 
./redis1/redis-cli -c -h 192.168.37.131 -p 7001 ，其中-c表示以集羣方式鏈接redis，-h指定ip地址，-p指定端口號

cluster nodes 查詢集羣結點信息

1.  
   192 .168.37.131:7001> cluster nodes
2.  
   8 e1186475f87c928e8a146d3804d9a2697246ad0 192.168.37.131:7003 master - 0 1478085160899 3 connected 10923-16383
3.  
   80 de7003738f74134a3403fc939fed253b7774f2 192.168.37.131:7004 master - 0 1478085156858 8 connected 0-5460
4.  
   652 caf5daf7971135679951324eba7b50e99251a 192.168.37.131:7005 slave 3adba62fdcc331ce231ca580cd2c8701e047bc6d 0 1478085158876 9 connected
5.  
   1 cd6482fd7038d78ad556b52b0cb9e2590ad5598 192.168.37.131:7001 myself,slave 80de7003738f74134a3403fc939fed253b7774f2 0 0 1 connected
6.  
   3 adba62fdcc331ce231ca580cd2c8701e047bc6d 192.168.37.131:7002 master - 0 1478085159889 9 connected 5461-10922
7.  
   cf23ca6d78cba3d3924065e1f7a394f6c51d4b28 192.168.37.131:7006 slave 8e1186475f87c928e8a146d3804d9a2697246ad0 0 1478085157867 6 connected
8.  
   192 .168.37.131:7001>

cluster info 查詢集羣狀態信息

1.  
   192 .168.37.131:7001> cluster info
2.  
   cluster_state:ok
3.  
   cluster_slots_assigned:16384
4.  
   cluster_slots_ok:16384
5.  
   cluster_slots_pfail:0
6.  
   cluster_slots_fail:0
7.  
   cluster_known_nodes:6
8.  
   cluster_size:3
9.  
   cluster_current_epoch:9
10.  
    cluster_my_epoch:8
11.  
    cluster_stats_messages_sent:12727
12.  
    cluster_stats_messages_received:10820
13.  
    192 .168.37.131:7001>

添加主節點

集羣建立成功後能夠向集羣中添加節點，下面是添加一個master主節點。

首先，準備一個乾淨的redis節點。按上面集羣版修改redis配置文件。開啓該redis節點。

查看redis進程

1.  
   [root @localhost redis-cluster]# !ps
2.  
   ps aux | grep redis
3.  
   root 2924 0.6 0.1 33932 1800 ? Ssl Nov01 4:08 ./redis-server *:6379 [cluster]
4.  
   root 11980 0.0 0.2 33936 2216 ? Ssl 01:12 0:09 ./redis-server *:7001 [cluster]
5.  
   root 11982 0.0 0.2 33936 2244 ? Ssl 01:12 0:09 ./redis-server *:7002 [cluster]
6.  
   root 11984 0.0 0.2 33936 2220 ? Ssl 01:12 0:09 ./redis-server *:7003 [cluster]
7.  
   root 11986 0.0 0.2 33936 2216 ? Ssl 01:12 0:09 ./redis-server *:7004 [cluster]
8.  
   root 11988 0.0 0.2 33936 2228 ? Ssl 01:12 0:09 ./redis-server *:7005 [cluster]
9.  
   root 11990 0.0 0.2 33936 2212 ? Ssl 01:12 0:09 ./redis-server *:7006 [cluster]
10.  
    root 13913 0.0 0.1 33936 1952 ? Ssl 04:21 0:00 ./redis7/redis-server *:7007 [cluster]
11.  
    root 13917 0.0 0.0 4360 728 pts/3 S+ 04:21 0:00 grep redis
12.  
    [root @localhost redis-cluster]#

執行下邊命令：

1.  
   [root@localhost redis-cluster] # ./redis-trib.rb add-node 192.168.37.131:7007 192.168.37.131:7001
2.  
   >>> Adding node 192.168.37.131:7007 to cluster 192.168.37.131:7001
3.  
   Connecting to node 192.168.37.131:7001: OK
4.  
   Connecting to node 192.168.37.131:7003: OK
5.  
   Connecting to node 192.168.37.131:7004: OK
6.  
   Connecting to node 192.168.37.131:7005: OK
7.  
   Connecting to node 192.168.37.131:7002: OK
8.  
   Connecting to node 192.168.37.131:7006: OK
9.  
   >>> Performing Cluster Check ( using node 192.168.37.131:7001)
10.  
    S: 1cd6482fd7038d78ad556b52b0cb9e2590ad5598 192.168.37.131:7001
11.  
    slots: ( 0 slots) slave
12.  
    replicates 80de7003738f74134a3403fc939fed253b7774f2
13.  
    M: 8e1186475f87c928e8a146d3804d9a2697246ad0 192.168.37.131:7003
14.  
    slots: 10923-16383 (5461 slots) master
15.  
    1 additional replica(s)
16.  
    M: 80de7003738f74134a3403fc939fed253b7774f2 192.168.37.131:7004
17.  
    slots: 0-5460 (5461 slots) master
18.  
    1 additional replica(s)
19.  
    S: 652caf5daf7971135679951324eba7b50e99251a 192.168.37.131:7005
20.  
    slots: ( 0 slots) slave
21.  
    replicates 3adba62fdcc331ce231ca580cd2c8701e047bc6d
22.  
    M: 3adba62fdcc331ce231ca580cd2c8701e047bc6d 192.168.37.131:7002
23.  
    slots: 5461-10922 (5462 slots) master
24.  
    1 additional replica(s)
25.  
    S: cf23ca6d78cba3d3924065e1f7a394f6c51d4b28 192.168.37.131:7006
26.  
    slots: ( 0 slots) slave
27.  
    replicates 8e1186475f87c928e8a146d3804d9a2697246ad0
28.  
    [OK] All nodes agree about slots configuration.
29.  
    >>> Check for open slots...
30.  
    >>> Check slots coverage...
31.  
    [OK] All 16384 slots covered.
32.  
    Connecting to node 192.168.37.131:7007: OK
33.  
    >>> Send CLUSTER MEET to node 192.168.37.131:7007 to make it join the cluster.
34.  
    [OK] New node added correctly.
35.  
    [root@localhost redis-cluster] #

查看集羣結點發現7007已添加到集羣中：

1.  
   [root@localhost redis-cluster] # ./redis1/redis-cli -c -h 192.168.37.131 -p 7001
2.  
   192.168.37.131:7001> cluster nodes
3.  
   8e1186475f87c928e8a146d3804d9a2697246ad0 192.168.37.131:7003 master - 0 1478085870038 3 connected 10923-16383
4.  
   80de7003738f74134a3403fc939fed253b7774f2 192.168.37.131:7004 master - 0 1478085868020 8 connected 0-5460
5.  
   652caf5daf7971135679951324eba7b50e99251a 192.168.37.131:7005 slave 3adba62fdcc331ce231ca580cd2c8701e047bc6d 0 1478085874075 9 connected
6.  
   1cd6482fd7038d78ad556b52b0cb9e2590ad5598 192.168.37.131:7001 myself,slave 80de7003738f74134a3403fc939fed253b7774f2 0 0 1 connected
7.  
   3adba62fdcc331ce231ca580cd2c8701e047bc6d 192.168.37.131:7002 master - 0 1478085873064 9 connected 5461-10922
8.  
   cf23ca6d78cba3d3924065e1f7a394f6c51d4b28 192.168.37.131:7006 slave 8e1186475f87c928e8a146d3804d9a2697246ad0 0 1478085875086 6 connected
9.  
   5d6c61ecff23bff3b0fb01a86c66d882f2d402a0 192.168.37.131:7007 master - 0 1478085872056 0 connected
10.  
    192.168.37.131:7001>

hash槽從新分配

添加完主節點須要對主節點進行hash槽分配這樣該主節才能夠存儲數據。 
redis集羣有16384個槽，集羣中的每一個master結點分配一些槽，經過查看集羣結點能夠看到槽佔用狀況。

給剛添加的7007結點分配槽：

第一步：鏈接上集羣 
./redis-trib.rb reshard 192.168.37.131:7001（鏈接集羣中任意一個可用結點都行）

1.  
   [root@localhost redis-cluster] # ./redis-trib.rb reshard 192.168.37.131:7001
2.  
   Connecting to node 192.168.37.131:7001: OK
3.  
   Connecting to node 192.168.37.131:7003: OK
4.  
   Connecting to node 192.168.37.131:7004: OK
5.  
   Connecting to node 192.168.37.131:7005: OK
6.  
   Connecting to node 192.168.37.131:7002: OK
7.  
   Connecting to node 192.168.37.131:7006: OK
8.  
   Connecting to node 192.168.37.131:7007: OK
9.  
   >>> Performing Cluster Check ( using node 192.168.37.131:7001)
10.  
    S: 1cd6482fd7038d78ad556b52b0cb9e2590ad5598 192.168.37.131:7001
11.  
    slots: ( 0 slots) slave
12.  
    replicates 80de7003738f74134a3403fc939fed253b7774f2
13.  
    M: 8e1186475f87c928e8a146d3804d9a2697246ad0 192.168.37.131:7003
14.  
    slots: 10923-16383 (5461 slots) master
15.  
    1 additional replica(s)
16.  
    M: 80de7003738f74134a3403fc939fed253b7774f2 192.168.37.131:7004
17.  
    slots: 0-5460 (5461 slots) master
18.  
    1 additional replica(s)
19.  
    S: 652caf5daf7971135679951324eba7b50e99251a 192.168.37.131:7005
20.  
    slots: ( 0 slots) slave
21.  
    replicates 3adba62fdcc331ce231ca580cd2c8701e047bc6d
22.  
    M: 3adba62fdcc331ce231ca580cd2c8701e047bc6d 192.168.37.131:7002
23.  
    slots: 5461-10922 (5462 slots) master
24.  
    1 additional replica(s)
25.  
    S: cf23ca6d78cba3d3924065e1f7a394f6c51d4b28 192.168.37.131:7006
26.  
    slots: ( 0 slots) slave
27.  
    replicates 8e1186475f87c928e8a146d3804d9a2697246ad0
28.  
    M: 5d6c61ecff23bff3b0fb01a86c66d882f2d402a0 192.168.37.131:7007
29.  
    slots: ( 0 slots) master
30.  
    0 additional replica(s)
31.  
    [OK] All nodes agree about slots configuration.
32.  
    >>> Check for open slots...
33.  
    >>> Check slots coverage...
34.  
    [OK] All 16384 slots covered.
35.  
    How many slots do you want to move (from 1 to 16384)?

第二步：輸入要分配的槽數量

1.  
   How many slots do you want to move (from 1 to 16384)? 1000
2.  
   What is the receiving node ID?

輸入1000表示要分配1000個槽

第三步：輸入接收槽的結點id

這裏準備給7007分配槽，經過cluster nodes查看7007結點id爲5d6c61ecff23bff3b0fb01a86c66d882f2d402a0

輸入：5d6c61ecff23bff3b0fb01a86c66d882f2d402a0

1.  
   What is the receiving node ID? 5d6c61ecff23bff3b0fb01a86c66d882f2d402a0
2.  
   Please enter all the source node IDs.
3.  
   Type 'all' to use all the nodes as source nodes for the hash slots.
4.  
   Type 'done' once you entered all the source nodes IDs.
5.  
   Source node #1:

第四步：輸入源結點id 
輸入源結點id，槽將從源結點中拿，分配後的槽在源結點中就不存在了。 
輸入all表示從全部源結點中獲取槽。 
輸入done取消分配。

這裏輸入all

1.  
   Source node #1:all
2.  
    
3.  
   省略了不少
4.  
    
5.  
   Moving slot 11253 from 8e1186475f87c928e8a146d3804d9a2697246ad0
6.  
   Moving slot 11254 from 8e1186475f87c928e8a146d3804d9a2697246ad0
7.  
   Moving slot 11255 from 8e1186475f87c928e8a146d3804d9a2697246ad0
8.  
   Do you want to proceed with the proposed reshard plan (yes/no)?

第五步：輸入yes開始移動槽到目標結點id

1.  
   省略了不少
2.  
    
3.  
   Moving slot 11253 from 192.168.37.131:7003 to 192.168.37.131:7007:
4.  
   Moving slot 11254 from 192.168.37.131:7003 to 192.168.37.131:7007:
5.  
   Moving slot 11255 from 192.168.37.131:7003 to 192.168.37.131:7007:
6.  
   [root@localhost redis-cluster]#

第六步：查看結點信息

1.  
   [root@localhost redis-cluster] # ./redis1/redis-cli -c -h 192.168.37.131 -p 7001
2.  
   192.168.37.131:7001> cluster nodes
3.  
   8e1186475f87c928e8a146d3804d9a2697246ad0 192.168.37.131:7003 master - 0 1478086754466 3 connected 11256-16383
4.  
   80de7003738f74134a3403fc939fed253b7774f2 192.168.37.131:7004 master - 0 1478086758509 8 connected 333-5460
5.  
   652caf5daf7971135679951324eba7b50e99251a 192.168.37.131:7005 slave 3adba62fdcc331ce231ca580cd2c8701e047bc6d 0 1478086756490 9 connected
6.  
   1cd6482fd7038d78ad556b52b0cb9e2590ad5598 192.168.37.131:7001 myself,slave 80de7003738f74134a3403fc939fed253b7774f2 0 0 1 connected
7.  
   3adba62fdcc331ce231ca580cd2c8701e047bc6d 192.168.37.131:7002 master - 0 1478086757500 9 connected 5795-10922
8.  
   cf23ca6d78cba3d3924065e1f7a394f6c51d4b28 192.168.37.131:7006 slave 8e1186475f87c928e8a146d3804d9a2697246ad0 0 1478086755477 6 connected
9.  
   5d6c61ecff23bff3b0fb01a86c66d882f2d402a0 192.168.37.131:7007 master - 0 1478086759518 10 connected 0-332 5461-5794 10923-11255
10.  
    192.168.37.131:7001>

查看結點信息，能夠發現7007結點已經從三個主節點中獲取了slot。

添加從節點

集羣建立成功後能夠向集羣中添加節點，下面是添加一個slave從節點。 
添加7008從結點，將7008做爲7007的從結點。

首先，準備一個乾淨的redis節點。按上面集羣版修改redis配置文件。開啓該redis節點。

查看redis進程

1.  
   [root @localhost redis-cluster]# ps aux | grep redis
2.  
   root 2924 0.5 0.1 33932 1800 ? Ssl Nov01 4:12 ./redis-server *:6379 [cluster]
3.  
   root 11980 0.0 0.2 33936 2308 ? Ssl 01:12 0:13 ./redis-server *:7001 [cluster]
4.  
   root 11982 0.0 0.2 33936 2288 ? Ssl 01:12 0:13 ./redis-server *:7002 [cluster]
5.  
   root 11984 0.0 0.2 33936 2236 ? Ssl 01:12 0:13 ./redis-server *:7003 [cluster]
6.  
   root 11986 0.0 0.2 33936 2288 ? Ssl 01:12 0:13 ./redis-server *:7004 [cluster]
7.  
   root 11988 0.0 0.2 33936 2248 ? Ssl 01:12 0:13 ./redis-server *:7005 [cluster]
8.  
   root 11990 0.0 0.2 33936 2244 ? Ssl 01:12 0:13 ./redis-server *:7006 [cluster]
9.  
   root 13913 0.1 0.2 33936 2092 ? Ssl 04:21 0:04 ./redis7/redis-server *:7007 [cluster]
10.  
    root 14000 0.1 0.1 33936 1948 ? Ssl 05:24 0:00 ./redis-server *:7008 [cluster]
11.  
    root 14006 0.0 0.0 4360 732 pts/3 S+ 05:24 0:00 grep redis
12.  
    [root @localhost redis-cluster]#

命令格式爲：

./redis-trib.rb add-node --slave --master-id 主節點id 添加節點的ip和端口 集羣中已存在節點ip和端口

執行以下命令：

1.  
   [root@localhost redis-cluster] # ./redis-trib.rb add-node --slave --master-id 5d6c61ecff23bff3b0fb01a86c66d882f2d402a0 192.168.37.131:7008 192.168.37.131:7001
2.  
   >>> Adding node 192.168.37.131:7008 to cluster 192.168.37.131:7001
3.  
   Connecting to node 192.168.37.131:7001: OK
4.  
   Connecting to node 192.168.37.131:7003: OK
5.  
   Connecting to node 192.168.37.131:7004: OK
6.  
   Connecting to node 192.168.37.131:7005: OK
7.  
   Connecting to node 192.168.37.131:7002: OK
8.  
   Connecting to node 192.168.37.131:7006: OK
9.  
   Connecting to node 192.168.37.131:7007: OK
10.  
    >>> Performing Cluster Check ( using node 192.168.37.131:7001)
11.  
    S: 1cd6482fd7038d78ad556b52b0cb9e2590ad5598 192.168.37.131:7001
12.  
    slots: ( 0 slots) slave
13.  
    replicates 80de7003738f74134a3403fc939fed253b7774f2
14.  
    M: 8e1186475f87c928e8a146d3804d9a2697246ad0 192.168.37.131:7003
15.  
    slots: 11256-16383 (5128 slots) master
16.  
    1 additional replica(s)
17.  
    M: 80de7003738f74134a3403fc939fed253b7774f2 192.168.37.131:7004
18.  
    slots: 333-5460 (5128 slots) master
19.  
    1 additional replica(s)
20.  
    S: 652caf5daf7971135679951324eba7b50e99251a 192.168.37.131:7005
21.  
    slots: ( 0 slots) slave
22.  
    replicates 3adba62fdcc331ce231ca580cd2c8701e047bc6d
23.  
    M: 3adba62fdcc331ce231ca580cd2c8701e047bc6d 192.168.37.131:7002
24.  
    slots: 5795-10922 (5128 slots) master
25.  
    1 additional replica(s)
26.  
    S: cf23ca6d78cba3d3924065e1f7a394f6c51d4b28 192.168.37.131:7006
27.  
    slots: ( 0 slots) slave
28.  
    replicates 8e1186475f87c928e8a146d3804d9a2697246ad0
29.  
    M: 5d6c61ecff23bff3b0fb01a86c66d882f2d402a0 192.168.37.131:7007
30.  
    slots: 0-332,5461-5794,10923-11255 (1000 slots) master
31.  
    0 additional replica(s)
32.  
    [OK] All nodes agree about slots configuration.
33.  
    >>> Check for open slots...
34.  
    >>> Check slots coverage...
35.  
    [OK] All 16384 slots covered.
36.  
    Connecting to node 192.168.37.131:7008: OK
37.  
    >>> Send CLUSTER MEET to node 192.168.37.131:7008 to make it join the cluster.
38.  
    Waiting for the cluster to join.
39.  
    >>> Configure node as replica of 192.168.37.131:7007.
40.  
    [OK] New node added correctly.
41.  
    [root@localhost redis-cluster] #

5d6c61ecff23bff3b0fb01a86c66d882f2d402a0是7007結點的id，能夠經過cluster nodes查看。

查看集羣中的結點，剛添加的7008爲7007的從節點：

1.  
   [root@localhost redis-cluster] # ./redis1/redis-cli -p 7001 -c 127.0.0.1:7001> cluster nodes
2.  
   8e1186475f87c928e8a146d3804d9a2697246ad0 192.168.37.131:7003 master - 0 1478089964769 3 connected 11256-16383
3.  
   80de7003738f74134a3403fc939fed253b7774f2 192.168.37.131:7004 master - 0 1478089966584 8 connected 333-5460
4.  
   652caf5daf7971135679951324eba7b50e99251a 192.168.37.131:7005 slave 3adba62fdcc331ce231ca580cd2c8701e047bc6d 0 1478089963748 9 connected
5.  
   1cd6482fd7038d78ad556b52b0cb9e2590ad5598 192.168.37.131:7001 myself,slave 80de7003738f74134a3403fc939fed253b7774f2 0 0 1 connected
6.  
   3adba62fdcc331ce231ca580cd2c8701e047bc6d 192.168.37.131:7002 master - 0 1478089966787 9 connected 5795-10922
7.  
   cf23ca6d78cba3d3924065e1f7a394f6c51d4b28 192.168.37.131:7006 slave 8e1186475f87c928e8a146d3804d9a2697246ad0 0 1478089962729 6 connected
8.  
   5c97e8eab019c40ea3df4925c7400fe7df1846bb 192.168.37.131:7008 slave 5d6c61ecff23bff3b0fb01a86c66d882f2d402a0 0 1478089961713 10 connected
9.  
   5d6c61ecff23bff3b0fb01a86c66d882f2d402a0 192.168.37.131:7007 master - 0 1478089965777 10 connected 0-332 5461-5794 10923-11255
10.  
    127.0.0.1:7001>

刪除結點

./redis-trib.rb del-node 192.168.37.131:7007 5d6c61ecff23bff3b0fb01a86c66d882f2d402a0

刪除已經佔有hash槽的結點會失敗，報錯以下：

[ERR] Node 192.168.37.131:7007 is not empty! Reshard data away and try again.

須要將該結點佔用的hash槽分配出去，請參考《hash槽從新分配》這段內容。

客戶端對Redis集羣的使用方法

使用redis命令行客戶端鏈接

1.  
   [root@localhost redis-cluster] # ./redis1/redis-cli -p 7001 -c
2.  
   127.0.0.1:7001> get a
3.  
   -> Redirected to slot [15495] located at 192.168.37.131:7003
4.  
   ( nil)
5.  
   192.168.37.131:7003>

必定要加-c參數,節點之間就能夠互相跳轉

使用jedis鏈接

1.  
   package com.pc.jedis.test;
2.  
    
3.  
   import java.util.HashSet;
4.  
   import java.util.Set;
5.  
    
6.  
   import redis.clients.jedis.HostAndPort;
7.  
   import redis.clients.jedis.JedisCluster;
8.  
    
9.  
   /**
10.  
    * Jedis集羣測試
11.  
    *
12.  
    * @author Switch
13.  
    * @data 2017年2月11日
14.  
    * @version V1.0
15.  
    */
16.  
    public class JedisClusterTest {
17.  
    public static void main(String[] args) {
18.  
    // 建立並填充節點信息
19.  
    Set<HostAndPort> nodes = new HashSet<>();
20.  
    nodes.add( new HostAndPort("192.168.37.131", 7001));
21.  
    nodes.add( new HostAndPort("192.168.37.131", 7002));
22.  
    nodes.add( new HostAndPort("192.168.37.131", 7003));
23.  
    nodes.add( new HostAndPort("192.168.37.131", 7004));
24.  
    nodes.add( new HostAndPort("192.168.37.131", 7005));
25.  
    nodes.add( new HostAndPort("192.168.37.131", 7006))
26. // 建立JedisCluster對象
27. JedisCluster jedisCluster = new JedisCluster(nodes);
28.  
     
29.  
    // 使用jedisCluster操做redis
30. String key = "jedisCluster";
31. String setResult = jedisCluster.set(key, "hello redis!");
32. System.out.println(setResult);
33.  
34. String getResult = jedisCluster.get(key);
35.  
    System.out.println(getResult);
36.  
37. // 關閉jedisCluster（程序執行完後才能關閉，內部封裝了鏈接池）
38.  
    jedisCluster.close();
39.  
    }
40.  
    }

——參考《Redis官方文檔》