redis cluster搭建

時間 2019-11-20

標籤 redis cluster 搭建欄目 Redis 简体版

原文原文鏈接

服務器信息：node

redis版本 3.2.0redis

ruby版本 1.8.7.374ruby

10.100.0.29 redis node 1服務器

10.100.0.45 redis node 2app

10.100.0.46 redis node 3負載均衡

10.100.0.47 redis node 4less

10.100.0.49 redis node 5tcp

redis cluster 節點配置

protected-mode no
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 0
daemonize yes
supervised no
pidfile /var/run/redis.pid
loglevel notice
logfile "/var/log/rediscluster.log"
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir ./
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
cluster-enabled yes           #開啓cluster支持
cluster-config-file nodes.conf    
cluster-node-timeout 5000

1. 安裝redis

# make PREFIX=/data/app/redis-3.2.0 && make install PREFIX=/data/app/redis-3.2.0測試

# cp -a utils /data/app/redis-3.2.03d

# cp -a src /data/app/redis-3.2.0/

# mkdir -p /data/app/redis-3.2.0/conf/

將redis配置文件保存爲/data/app/redis-3.2.0/conf/rediscluster.conf 一臺機器啓動了2個實例，另外一個實例使用6380端口

2. 安裝ruby redis模塊

# yum install redis rubygems -y && gem install redis

3. 啓動各節點，查看是否啓動正常

# cd /data/app/redis-3.2.0

# bin/redis-server conf/rediscluster.conf

4. 建立節點

# src/redis-trib.rb create --replicas 1 10.100.0.29:6379 10.100.0.29:6380 10.100.0.45:6379 10.100.0.45:6380 10.100.0.46:6379 10.100.0.46:6380

>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
10.100.0.29:6379
10.100.0.46:6379
10.100.0.45:6379
Adding replica 10.100.0.46:6380 to 10.100.0.29:6379
Adding replica 10.100.0.29:6380 to 10.100.0.46:6379
Adding replica 10.100.0.45:6380 to 10.100.0.45:6379
M: 484c437d11e330023eb29268c6f00ebc5af409fd 10.100.0.29:6379
   slots:0-5460 (5461 slots) master
S: ee8ca5b7303392831ef753e3c07ccb4c086d82fe 10.100.0.29:6380
   replicates 3df73cb44e750cd676e3996c97309e148b628abf
M: 12952b61c854a54112897a5bea35b8ddc6f04390 10.100.0.45:6379
   slots:10923-16383 (5461 slots) master
S: 5501efd20bfcad76e193f3b00d0aeba3acdb0715 10.100.0.45:6380
   replicates 12952b61c854a54112897a5bea35b8ddc6f04390
M: 3df73cb44e750cd676e3996c97309e148b628abf 10.100.0.46:6379
   slots:5461-10922 (5462 slots) master
S: 9ae373556ba70cd0fc24fdcba95c0fee3fd0201e 10.100.0.46:6380
   replicates 484c437d11e330023eb29268c6f00ebc5af409fd
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 10.100.0.29:6379)
M: 484c437d11e330023eb29268c6f00ebc5af409fd 10.100.0.29:6379
   slots:0-5460 (5461 slots) master
M: ee8ca5b7303392831ef753e3c07ccb4c086d82fe 10.100.0.29:6380
   slots: (0 slots) master
   replicates 3df73cb44e750cd676e3996c97309e148b628abf
M: 12952b61c854a54112897a5bea35b8ddc6f04390 10.100.0.45:6379
   slots:10923-16383 (5461 slots) master
M: 5501efd20bfcad76e193f3b00d0aeba3acdb0715 10.100.0.45:6380
   slots: (0 slots) master
   replicates 12952b61c854a54112897a5bea35b8ddc6f04390
M: 3df73cb44e750cd676e3996c97309e148b628abf 10.100.0.46:6379
   slots:5461-10922 (5462 slots) master
M: 9ae373556ba70cd0fc24fdcba95c0fee3fd0201e 10.100.0.46:6380
   slots: (0 slots) master
   replicates 484c437d11e330023eb29268c6f00ebc5af409fd
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

5. 查看cluster狀態

# src/redis-trib.rb info 10.100.0.29:6379

10.100.0.29:6379 (484c437d...) -> 0 keys | 5461 slots | 1 slaves.
10.100.0.46:6379 (3df73cb4...) -> 0 keys | 5462 slots | 1 slaves.
10.100.0.45:6379 (12952b61...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.

# src/redis-trib.rb check 10.100.0.29:6379

>>> Performing Cluster Check (using node 10.100.0.29:6379)
M: 484c437d11e330023eb29268c6f00ebc5af409fd 10.100.0.29:6379
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: ee8ca5b7303392831ef753e3c07ccb4c086d82fe 10.100.0.29:6380
   slots: (0 slots) slave
   replicates 3df73cb44e750cd676e3996c97309e148b628abf
S: 5501efd20bfcad76e193f3b00d0aeba3acdb0715 10.100.0.45:6380
   slots: (0 slots) slave
   replicates 12952b61c854a54112897a5bea35b8ddc6f04390
M: 3df73cb44e750cd676e3996c97309e148b628abf 10.100.0.46:6379
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 9ae373556ba70cd0fc24fdcba95c0fee3fd0201e 10.100.0.46:6380
   slots: (0 slots) slave
   replicates 484c437d11e330023eb29268c6f00ebc5af409fd
M: 12952b61c854a54112897a5bea35b8ddc6f04390 10.100.0.45:6379
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

6. 測試

一、master停機漂移

中止10.100.0.45:6379,查看集羣狀態

通過查看，在通過一段時間後10.100.0.45:6380代替10.100.0.45:6379成爲master

[root@localhost redis-3.2.0]# src/redis-trib.rb info 10.100.0.29:6379
[ERR] Sorry, can't connect to node 10.100.0.45:6379
*** WARNING: 10.100.0.45:6380 claims to be slave of unknown node ID 12952b61c854a54112897a5bea35b8ddc6f04390.
10.100.0.29:6379 (484c437d...) -> 4 keys | 5461 slots | 1 slaves.
10.100.0.46:6379 (3df73cb4...) -> 1 keys | 5462 slots | 1 slaves.
[OK] 5 keys in 2 masters.
0.00 keys per slot on average.
[root@localhost redis-3.2.0]# src/redis-trib.rb info 10.100.0.29:6379
10.100.0.29:6379 (484c437d...) -> 4 keys | 5461 slots | 1 slaves.
10.100.0.45:6380 (5501efd2...) -> 5 keys | 5461 slots | 0 slaves.
10.100.0.46:6379 (3df73cb4...) -> 1 keys | 5462 slots | 1 slaves.
[OK] 10 keys in 3 masters.
0.00 keys per slot on average.

二、啓動宕機的10.100.0.45:6379

原先宕機的redis實例成爲了slave

[root@localhost redis-3.2.0]# src/redis-trib.rb check 10.100.0.29:6379
>>> Performing Cluster Check (using node 10.100.0.29:6379)
M: 484c437d11e330023eb29268c6f00ebc5af409fd 10.100.0.29:6379
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: ee8ca5b7303392831ef753e3c07ccb4c086d82fe 10.100.0.29:6380
   slots: (0 slots) slave
   replicates 3df73cb44e750cd676e3996c97309e148b628abf
M: 5501efd20bfcad76e193f3b00d0aeba3acdb0715 10.100.0.45:6380
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
M: 3df73cb44e750cd676e3996c97309e148b628abf 10.100.0.46:6379
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 9ae373556ba70cd0fc24fdcba95c0fee3fd0201e 10.100.0.46:6380
   slots: (0 slots) slave
   replicates 484c437d11e330023eb29268c6f00ebc5af409fd
S: 12952b61c854a54112897a5bea35b8ddc6f04390 10.100.0.45:6379
   slots: (0 slots) slave
   replicates 5501efd20bfcad76e193f3b00d0aeba3acdb0715
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@localhost redis-3.2.0]# src/redis-trib.rb info 10.100.0.29:6379
10.100.0.29:6379 (484c437d...) -> 4 keys | 5461 slots | 1 slaves.
10.100.0.45:6380 (5501efd2...) -> 5 keys | 5461 slots | 1 slaves.
10.100.0.46:6379 (3df73cb4...) -> 1 keys | 5462 slots | 1 slaves.
[OK] 10 keys in 3 masters.
0.00 keys per slot on average.

redis cluster擴展

將node 4 ，5上的4個實例加入集羣

# src/redis-trib.rb add-node 10.100.0.47:6379 10.100.0.45:6379 #待加入的節點後跟已有節點

>>> Adding node 10.100.0.47:6379 to cluster 10.100.0.45:6379
>>> Performing Cluster Check (using node 10.100.0.45:6379)
S: 12952b61c854a54112897a5bea35b8ddc6f04390 10.100.0.45:6379
   slots: (0 slots) slave
   replicates 5501efd20bfcad76e193f3b00d0aeba3acdb0715
M: 484c437d11e330023eb29268c6f00ebc5af409fd 10.100.0.29:6379
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
M: 3df73cb44e750cd676e3996c97309e148b628abf 10.100.0.46:6379
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 9ae373556ba70cd0fc24fdcba95c0fee3fd0201e 10.100.0.46:6380
   slots: (0 slots) slave
   replicates 484c437d11e330023eb29268c6f00ebc5af409fd
M: 5501efd20bfcad76e193f3b00d0aeba3acdb0715 10.100.0.45:6380
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
S: ee8ca5b7303392831ef753e3c07ccb4c086d82fe 10.100.0.29:6380
   slots: (0 slots) slave
   replicates 3df73cb44e750cd676e3996c97309e148b628abf
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 10.100.0.47:6379 to make it join the cluster.
[OK] New node added correctly.

同理加入10.100.0.47:6380後查看集羣狀態

# src/redis-trib.rb check 10.100.0.29:6379

>>> Performing Cluster Check (using node 10.100.0.29:6379)
M: 484c437d11e330023eb29268c6f00ebc5af409fd 10.100.0.29:6379
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: ee8ca5b7303392831ef753e3c07ccb4c086d82fe 10.100.0.29:6380
   slots: (0 slots) slave
   replicates 3df73cb44e750cd676e3996c97309e148b628abf
M: 5ebaf1831f7c8e4c1293c87632968312f371d96a 10.100.0.47:6380
   slots: (0 slots) master
   0 additional replica(s)
M: 76d6d37d6c2c33adacc45cbfbf49e134ec4b1dc5 10.100.0.47:6379
   slots: (0 slots) master
   0 additional replica(s)
M: 5501efd20bfcad76e193f3b00d0aeba3acdb0715 10.100.0.45:6380
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
M: 3df73cb44e750cd676e3996c97309e148b628abf 10.100.0.46:6379
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 9ae373556ba70cd0fc24fdcba95c0fee3fd0201e 10.100.0.46:6380
   slots: (0 slots) slave
   replicates 484c437d11e330023eb29268c6f00ebc5af409fd
S: 12952b61c854a54112897a5bea35b8ddc6f04390 10.100.0.45:6379
   slots: (0 slots) slave
   replicates 5501efd20bfcad76e193f3b00d0aeba3acdb0715
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

發現2個節點都爲master，另外加入的2個節點須要爲slave

使用redis-trib.rb add-node --slave --master-id 5ebaf1831f7c8e4c1293c87632968312f371d96a 10.100.0.49:6379 10.100.0.45:6379 # 指定成爲某個master的slave

# src/redis-trib.rb add-node --slave --master-id 5ebaf1831f7c8e4c1293c87632968312f371d96a 10.100.0.49:6379 10.100.0.45:6379

# src/redis-trib.rb add-node --slave --master-id 76d6d37d6c2c33adacc45cbfbf49e134ec4b1dc5 10.100.0.49:6380 10.100.0.45:6379

再次檢查cluster狀態

# src/redis-trib.rb info 10.100.0.29:6379

10.100.0.29:6379 (484c437d...) -> 4 keys | 5461 slots | 1 slaves.
10.100.0.45:6380 (5501efd2...) -> 5 keys | 5461 slots | 1 slaves.
10.100.0.47:6379 (76d6d37d...) -> 0 keys | 0 slots | 1 slaves.
10.100.0.47:6380 (5ebaf183...) -> 0 keys | 0 slots | 1 slaves.
10.100.0.46:6379 (3df73cb4...) -> 1 keys | 5462 slots | 1 slaves.
[OK] 10 keys in 5 masters.
0.00 keys per slot on average.

redis cluster負載均衡

自動rebalance

# src/redis-trib.rb rebalance 10.100.0.29:6379

額負載平衡的時候會報錯，不讓移動

>>> Performing Cluster Check (using node 10.100.0.29:6379)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Rebalancing across 3 nodes. Total weight = 3
Moving 1003 slots from 10.100.0.45:6380 to 10.100.0.29:6379
###########################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################
Moving 994 slots from 10.100.0.45:6380 to 10.100.0.46:6379
##################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################

手動移動

# src/redis-trib.rb reshard 10.100.0.29:6379

>>> Performing Cluster Check (using node 10.100.0.29:6379)
M: 484c437d11e330023eb29268c6f00ebc5af409fd 10.100.0.29:6379
   slots:999-5460 (4462 slots) master
   1 additional replica(s)
S: ee8ca5b7303392831ef753e3c07ccb4c086d82fe 10.100.0.29:6380
   slots: (0 slots) slave
   replicates 3df73cb44e750cd676e3996c97309e148b628abf
S: 0ee117b8c72529f9b7038c69cfaa1c60cd7f952a 10.100.0.49:6380
   slots: (0 slots) slave
   replicates 76d6d37d6c2c33adacc45cbfbf49e134ec4b1dc5
M: 5501efd20bfcad76e193f3b00d0aeba3acdb0715 10.100.0.45:6380
   slots:11922-16383 (4462 slots) master
   1 additional replica(s)
S: 12952b61c854a54112897a5bea35b8ddc6f04390 10.100.0.45:6379
   slots: (0 slots) slave
   replicates 5501efd20bfcad76e193f3b00d0aeba3acdb0715
S: 9ae373556ba70cd0fc24fdcba95c0fee3fd0201e 10.100.0.46:6380
   slots: (0 slots) slave
   replicates 484c437d11e330023eb29268c6f00ebc5af409fd
S: 69a64bd82f5037fdad55d25c0b224845f297d521 10.100.0.49:6379
   slots: (0 slots) slave
   replicates 5ebaf1831f7c8e4c1293c87632968312f371d96a
M: 76d6d37d6c2c33adacc45cbfbf49e134ec4b1dc5 10.100.0.47:6379
   slots:0-998,5461-6461,10923-11921 (2999 slots) master
   1 additional replica(s)
M: 5ebaf1831f7c8e4c1293c87632968312f371d96a 10.100.0.47:6380
   slots: (0 slots) master
   1 additional replica(s)
M: 3df73cb44e750cd676e3996c97309e148b628abf 10.100.0.46:6379
   slots:6462-10922 (4461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 10      #移動多少個slot,可是實際上不會移動那麼多slot
What is the receiving node ID? 5ebaf1831f7c8e4c1293c87632968312f371d96a   #移動到哪一個node
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1:all        #選擇源node

Ready to move 10 slots.
  Source nodes:
    M: 484c437d11e330023eb29268c6f00ebc5af409fd 10.100.0.29:6379
   slots:999-5460 (4462 slots) master
   1 additional replica(s)
    M: 5501efd20bfcad76e193f3b00d0aeba3acdb0715 10.100.0.45:6380
   slots:11922-16383 (4462 slots) master
   1 additional replica(s)
    M: 76d6d37d6c2c33adacc45cbfbf49e134ec4b1dc5 10.100.0.47:6379
   slots:0-998,5461-6461,10923-11921 (2999 slots) master
   1 additional replica(s)
    M: 3df73cb44e750cd676e3996c97309e148b628abf 10.100.0.46:6379
   slots:6462-10922 (4461 slots) master
   1 additional replica(s)
  Destination node:
    M: 5ebaf1831f7c8e4c1293c87632968312f371d96a 10.100.0.47:6380
   slots: (0 slots) master
   1 additional replica(s)
  Resharding plan:
    Moving slot 999 from 484c437d11e330023eb29268c6f00ebc5af409fd
    Moving slot 1000 from 484c437d11e330023eb29268c6f00ebc5af409fd
    Moving slot 1001 from 484c437d11e330023eb29268c6f00ebc5af409fd
    Moving slot 11922 from 5501efd20bfcad76e193f3b00d0aeba3acdb0715
    Moving slot 11923 from 5501efd20bfcad76e193f3b00d0aeba3acdb0715
    Moving slot 6462 from 3df73cb44e750cd676e3996c97309e148b628abf
    Moving slot 6463 from 3df73cb44e750cd676e3996c97309e148b628abf
    Moving slot 0 from 76d6d37d6c2c33adacc45cbfbf49e134ec4b1dc5
Do you want to proceed with the proposed reshard plan (yes/no)? yes    #yes開始move slot
......

移動後查看cluster狀態

...
M: 5ebaf1831f7c8e4c1293c87632968312f371d96a 10.100.0.47:6380
   slots:0,999-1001,6462-6463,11922-11923 (8 slots) master
   1 additional replica(s)
...

redis cluster node刪除

刪除node 4，5上的4個redis node

使用src/redis-trib.rb reshard 10.100.0.29:6379命令手動移動須要刪除的node到指定的node，確保node爲空

[root@localhost redis-3.2.0]# src/redis-trib.rb del-node 10.100.0.47:6379 76d6d37d6c2c33adacc45cbfbf49e134ec4b1dc5
>>> Removing node 76d6d37d6c2c33adacc45cbfbf49e134ec4b1dc5 from cluster 10.100.0.47:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[root@localhost redis-3.2.0]# src/redis-trib.rb del-node 10.100.0.47:6380 5ebaf1831f7c8e4c1293c87632968312f371d96a
>>> Removing node 5ebaf1831f7c8e4c1293c87632968312f371d96a from cluster 10.100.0.47:6380
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[root@localhost redis-3.2.0]# src/redis-trib.rb del-node 10.100.0.49:6379 69a64bd82f5037fdad55d25c0b224845f297d521
>>> Removing node 69a64bd82f5037fdad55d25c0b224845f297d521 from cluster 10.100.0.49:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[root@localhost redis-3.2.0]# src/redis-trib.rb del-node 10.100.0.49:6380 0ee117b8c72529f9b7038c69cfaa1c60cd7f952a
>>> Removing node 0ee117b8c72529f9b7038c69cfaa1c60cd7f952a from cluster 10.100.0.49:6380
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[root@localhost redis-3.2.0]# src/redis-trib.rb info 10.100.0.29:6379
10.100.0.29:6379 (484c437d...) -> 4 keys | 4459 slots | 1 slaves.
10.100.0.45:6380 (5501efd2...) -> 5 keys | 7458 slots | 1 slaves.
10.100.0.46:6379 (3df73cb4...) -> 1 keys | 4467 slots | 1 slaves.
[OK] 10 keys in 3 masters.
0.00 keys per slot on average.
[root@localhost redis-3.2.0]# src/redis-trib.rb check 10.100.0.29:6379
>>> Performing Cluster Check (using node 10.100.0.29:6379)
M: 484c437d11e330023eb29268c6f00ebc5af409fd 10.100.0.29:6379
   slots:1002-5460 (4459 slots) master
   1 additional replica(s)
S: ee8ca5b7303392831ef753e3c07ccb4c086d82fe 10.100.0.29:6380
   slots: (0 slots) slave
   replicates 3df73cb44e750cd676e3996c97309e148b628abf
M: 5501efd20bfcad76e193f3b00d0aeba3acdb0715 10.100.0.45:6380
   slots:1-998,5461-6461,10923-11921,11924-16383 (7458 slots) master
   1 additional replica(s)
S: 12952b61c854a54112897a5bea35b8ddc6f04390 10.100.0.45:6379
   slots: (0 slots) slave
   replicates 5501efd20bfcad76e193f3b00d0aeba3acdb0715
S: 9ae373556ba70cd0fc24fdcba95c0fee3fd0201e 10.100.0.46:6380
   slots: (0 slots) slave
   replicates 484c437d11e330023eb29268c6f00ebc5af409fd
M: 3df73cb44e750cd676e3996c97309e148b628abf 10.100.0.46:6379
   slots:0,999-1001,6462-10922,11922-11923 (4467 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。