Linux Redis集羣搭建與集羣客戶端實現

時間 2019-11-24

原文原文鏈接

硬件環境html

本文適用的硬件環境以下node

Linux版本：CentOS release 6.7 (Final)
Redis版本：3.2.1

Redis已經成功安裝，安裝路徑爲/home/idata/yangfan/local/redis-3.2.1。python

咱們要在單臺機器上搭建Redis集羣，方式是經過不一樣的TCP端口啓動多個實例，而後組成集羣。git

一、啓動Redis多個實例github

咱們在Redis安裝目錄下建立目錄cluster，並編寫7000.conf~7005.conf 6個配置文件，這6個配置文件用來啓動6個實例，後面將使用這6個實例組成集羣。redis

以7000.conf爲例，配置文件須要填寫以下幾項。算法

port  7000                                        //端口7000,7002,7003        
bind 10.93.84.53                                     //默認ip爲127.0.0.1 須要改成其餘節點機器可訪問的ip 不然建立集羣時沒法訪問對應的端口，沒法建立集羣
daemonize    yes                               //redis後臺運行
pidfile  ./redis_7000.pid          //pidfile文件對應7000,7001,7002
cluster-enabled  yes                           //開啓集羣  把註釋#去掉
cluster-config-file  nodes_7000.conf   //集羣的配置  配置文件首次啓動自動生成 7000,7001,7002
cluster-node-timeout  15000                //請求超時  默認15秒，可自行設置
appendonly  yes                           //aof日誌開啓  有須要就開啓，它會每次寫操做都記錄一條日誌

分別啓動6個實例shell

./bin/redis-server cluster/conf/7000.conf
./bin/redis-server cluster/conf/7001.conf 
./bin/redis-server cluster/conf/7002.conf 
./bin/redis-server cluster/conf/7003.conf 
./bin/redis-server cluster/conf/7004.conf 
./bin/redis-server cluster/conf/7005.conf

啓動成功後，看一下進程windows

# ps -ef | grep redis | grep cluster
idata    15711 22329  0 18:40 pts/10   00:00:00 ./bin/redis-server 10.93.84.53:7000 [cluster] idata 15740 22329 0 18:40 pts/10 00:00:00 ./bin/redis-server 10.93.84.53:7001 [cluster] idata 15810 22329 0 18:40 pts/10 00:00:00 ./bin/redis-server 10.93.84.53:7002 [cluster] idata 17023 22329 0 18:42 pts/10 00:00:00 ./bin/redis-server 10.93.84.53:7003 [cluster] idata 17030 22329 0 18:42 pts/10 00:00:00 ./bin/redis-server 10.93.84.53:7004 [cluster] idata 17035 22329 0 18:42 pts/10 00:00:00 ./bin/redis-server 10.93.84.53:7005 [cluster]

至此，ip=10.93.84.53機器上建立了6個實例，端口號爲port=7000~7005。centos

Redis 3.0以上的集羣方式是經過Redis安裝目錄下的bin/redis-trib.rb腳本搭建。

這個腳本是用Ruby編寫的，嘗試運行，若是打印以下，你能夠跳過本文的第二部分。

idata@qa-f1502-xg01.xg01:~/yangfan/local/redis-3.2.1/bin$ ruby redis-trib.rb 
Usage: redis-trib <command> <options> <arguments ...>

  create          host1:port1 ... hostN:portN
                  --replicas <arg>
  check           host:port
  info            host:port
  fix             host:port
                  --timeout <arg>
  reshard         host:port
                  --from <arg>
                  --to <arg>
                  --slots <arg>
                  --yes
                  --timeout <arg>
                  --pipeline <arg>
  rebalance       host:port
                  --weight <arg>
                  --auto-weights
                  --use-empty-masters
                  --timeout <arg>
                  --simulate
                  --pipeline <arg>
                  --threshold <arg>
  add-node        new_host:new_port existing_host:existing_port
                  --slave
                  --master-id <arg>
  del-node        host:port node_id
  set-timeout     host:port milliseconds
  call            host:port command arg arg .. arg
  import          host:port
                  --from <arg>
                  --copy
                  --replace
  help            (show this help)

For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.

若是執行失敗，那麼不幸的是你的機器沒有Ruby運行的環境，那麼你須要安裝Ruby。進入第二部分。

二、安裝ruby

下面的過程都是在root權限下完成的。

1）yum安裝ruby和依賴的包。

# yum -y install ruby ruby-devel rubygems rpm-build

通常來講，這一步是能正常完成的。

2）使用gem這個命令來安裝redis接口

gem是ruby的一個工具包

# gem install redis

安裝過程出錯

鬱悶，看樣子要升級ruby版本。

3）升級Ruby的版本

安裝rvm，我不知道這是個什麼東西，可是感受像是Ruby的一個包管理器。

# curl -L get.rvm.io | bash -s stable

WTF，又出問題了

氣急敗壞的照着他說的作

# gpg2 --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
gpg: keyring `/root/.gnupg/secring.gpg' created
gpg: requesting key D39DC0E3 from hkp server keys.gnupg.net
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key D39DC0E3: public key "Michal Papis (RVM signing) <mpapis@gmail.com>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

而後從新下載rvm安裝，成功了。

# curl -L get.rvm.io | bash -s stable 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 24090  100 24090    0     0  18023      0  0:00:01  0:00:01 --:--:--  129k
Downloading https://github.com/rvm/rvm/archive/1.29.3.tar.gz
Downloading https://github.com/rvm/rvm/releases/download/1.29.3/1.29.3.tar.gz.asc
gpg: Signature made Mon 11 Sep 2017 04:59:21 AM CST using RSA key ID BF04FF17
gpg: Good signature from "Michal Papis (RVM signing) <mpapis@gmail.com>"
gpg:                 aka "Michal Papis <michal.papis@toptal.com>"
gpg:                 aka "[jpeg image of size 5015]"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 409B 6B17 96C2 7546 2A17  0311 3804 BB82 D39D C0E3
     Subkey fingerprint: 62C9 E5F4 DA30 0D94 AC36  166B E206 C29F BF04 FF17
GPG verified '/usr/local/rvm/archives/rvm-1.29.3.tgz'
Creating group 'rvm'

Installing RVM to /usr/local/rvm/
Installation of RVM in /usr/local/rvm/ is almost complete:

  * First you need to add all users that will be using rvm to 'rvm' group,
    and logout - login again, anyone using rvm will be operating with `umask u=rwx,g=rwx,o=rx`.

  * To start using RVM you need to run `source /etc/profile.d/rvm.sh`
    in all your open shell windows, in rare cases you need to reopen all shell windows.

接着，source環境，讓rvm可用。

# source /usr/local/rvm/scripts/rvm

查看Ruby可用版本

# rvm list known
# MRI Rubies
[ruby-]1.8.6[-p420]
[ruby-]1.8.7[-head] # security released on head
[ruby-]1.9.1[-p431]
[ruby-]1.9.2[-p330]
[ruby-]1.9.3[-p551]
[ruby-]2.0.0[-p648]
[ruby-]2.1[.10]
[ruby-]2.2[.7]
[ruby-]2.3[.4]
[ruby-]2.4[.1]

能夠看到最新的版本是2.4.1，那麼咱們裝最新的吧。

# rvm install 2.4.1
Searching for binary rubies, this might take some time.
No binary rubies available for: centos/6/x86_64/ruby-2.4.1.
Continuing with compilation. Please read 'rvm help mount' to get more information on binary rubies.
Checking requirements for centos.
Installing requirements for centos.
Installing required packages: libffi-devel, libyaml-devel......
Requirements installation successful.
Installing Ruby from source to: /usr/local/rvm/rubies/ruby-2.4.1, this may take a while depending on your cpu(s)...
ruby-2.4.1 - #downloading ruby-2.4.1, this may take a while depending on your connection...

curl: (35) SSL connect error
There was an error(35).
Checking fallback: https://ftp.ruby-lang.org/pub/ruby/2.4/ruby-2.4.1.tar.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11.9M  100 11.9M    0     0  1753k      0  0:00:07  0:00:07 --:--:-- 2919k
ruby-2.4.1 - #extracting ruby-2.4.1 to /usr/local/rvm/src/ruby-2.4.1....
ruby-2.4.1 - #applying patch /usr/local/rvm/patches/ruby/2.4.1/random_c_using_NR_prefix.patch.
ruby-2.4.1 - #configuring..................................................................
ruby-2.4.1 - #post-configuration..
ruby-2.4.1 - #compiling..............................................................................................
ruby-2.4.1 - #installing.........................
ruby-2.4.1 - #making binaries executable..
ruby-2.4.1 - #downloading rubygems-2.6.14
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  751k  100  751k    0     0   443k      0  0:00:01  0:00:01 --:--:--  628k
No checksum for downloaded archive, recording checksum in user configuration.
ruby-2.4.1 - #extracting rubygems-2.6.14....
ruby-2.4.1 - #removing old rubygems.........
ruby-2.4.1 - #installing rubygems-2.6.14...........................
ruby-2.4.1 - #gemset created /usr/local/rvm/gems/ruby-2.4.1@global
ruby-2.4.1 - #importing gemset /usr/local/rvm/gemsets/global.gems...............................................
ruby-2.4.1 - #generating global wrappers........
ruby-2.4.1 - #gemset created /usr/local/rvm/gems/ruby-2.4.1
ruby-2.4.1 - #importing gemsetfile /usr/local/rvm/gemsets/default.gems evaluated to empty gem list
ruby-2.4.1 - #generating default wrappers........
ruby-2.4.1 - #adjusting #shebangs for (gem irb erb ri rdoc testrb rake).
Install of ruby-2.4.1 - #complete 
Ruby was built without documentation, to build it run: rvm docs generate-ri

至此，咱們升級了Ruby的版本。

4）安裝gem redis接口，成功！

gem install redis
Fetching: redis-4.0.1.gem (100%)
Successfully installed redis-4.0.1
Parsing documentation for redis-4.0.1
Installing ri documentation for redis-4.0.1
Done installing documentation for redis after 0 seconds
1 gem installed

5）安裝rubygems，成功！

# yum install -y rubygems
Loaded plugins: fastestmirror, security
Setting up Install Process
Loading mirror speeds from cached hostfile
base                                                                                                                                                   | 3.7 kB     00:00     
didi_jenkins_enable                                                                                                                                    | 1.5 kB     00:00     
didi_op_toa_enable                                                                                                                                     | 1.5 kB     00:00     
didi_txjenkins_enable                                                                                                                                  | 1.5 kB     00:00     
didi_update                                                                                                                                            | 1.5 kB     00:00     
epel                                                                                                                                                   | 4.3 kB     00:00     
extras                                                                                                                                                 | 3.4 kB     00:00     
tmprepo                                                                                                                                                | 1.5 kB     00:00     
updates                                                                                                                                                | 3.4 kB     00:00     
Package rubygems-1.3.7-5.el6.noarch already installed and latest version
Nothing to do

至此，咱們的Ruby和運行redis-trib.rb須要的環境安裝完成了。

三、Redis集羣搭建

有了Ruby執行環境，能夠開始將以前的6個實例組建成集羣了。

命令方式：

ruby ./bin/redis-trib.rb create --replicas 1 10.93.84.53:7000 10.93.84.53:7001 10.93.84.53:7002 10.93.84.53:7003 10.93.84.53:7004 10.93.84.53:7005

--replicas 1表示爲集羣的master節點建立1個副本。那麼6個實例裏，有三個master，有三個是slave。

後面跟上6個實例就行了，形式就是ip:port

執行狀況：

# ruby ./bin/redis-trib.rb create --replicas 1 10.93.84.53:7000 10.93.84.53:7001 10.93.84.53:7002 10.93.84.53:7003 10.93.84.53:7004 10.93.84.53:7005
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
10.93.84.53:7000
10.93.84.53:7001
10.93.84.53:7002
Adding replica 10.93.84.53:7003 to 10.93.84.53:7000
Adding replica 10.93.84.53:7004 to 10.93.84.53:7001
Adding replica 10.93.84.53:7005 to 10.93.84.53:7002
M: 6346ae8c7af7949658619fcf4021cc7aca454819 10.93.84.53:7000
   slots:0-5460 (5461 slots) master
M: 5ac973bceab0d486c497345fe884ff54d1bb225a 10.93.84.53:7001
   slots:5461-10922 (5462 slots) master
M: cc46a4a1c0ec3f621b6b5405c6c10b7cffe73932 10.93.84.53:7002
   slots:10923-16383 (5461 slots) master
S: 92f62ec93a0550d962f81213ca7e9b3c9c996afd 10.93.84.53:7003
   replicates 6346ae8c7af7949658619fcf4021cc7aca454819
S: 942c9f97dc68198c39f425d13df0d8e3c40c5a58 10.93.84.53:7004
   replicates 5ac973bceab0d486c497345fe884ff54d1bb225a
S: a92a81532b63652bbd862be6f19a9bd8832e5e05 10.93.84.53:7005
   replicates cc46a4a1c0ec3f621b6b5405c6c10b7cffe73932
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 10.93.84.53:7000)
M: 6346ae8c7af7949658619fcf4021cc7aca454819 10.93.84.53:7000
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: a92a81532b63652bbd862be6f19a9bd8832e5e05 10.93.84.53:7005
   slots: (0 slots) slave
   replicates cc46a4a1c0ec3f621b6b5405c6c10b7cffe73932
M: 5ac973bceab0d486c497345fe884ff54d1bb225a 10.93.84.53:7001
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 942c9f97dc68198c39f425d13df0d8e3c40c5a58 10.93.84.53:7004
   slots: (0 slots) slave
   replicates 5ac973bceab0d486c497345fe884ff54d1bb225a
S: 92f62ec93a0550d962f81213ca7e9b3c9c996afd 10.93.84.53:7003
   slots: (0 slots) slave
   replicates 6346ae8c7af7949658619fcf4021cc7aca454819
M: cc46a4a1c0ec3f621b6b5405c6c10b7cffe73932 10.93.84.53:7002
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

能夠看到16384個slot都已經建立完成，而且創建了3個master和對應的replica:

Using 3 masters:
10.93.84.53:7000
10.93.84.53:7001
10.93.84.53:7002
Adding replica 10.93.84.53:7003 to 10.93.84.53:7000
Adding replica 10.93.84.53:7004 to 10.93.84.53:7001
Adding replica 10.93.84.53:7005 to 10.93.84.53:7002

。。。

[OK] All 16384 slots covered.

四、驗證集羣狀態

登陸集羣客戶端，-c標識以集羣方式登陸

./bin/redis-cli -h 10.93.84.53 -p 7000 -c

查看集羣狀態

10.93.84.53:7000> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:8
cluster_my_epoch:8
cluster_stats_messages_sent:215
cluster_stats_messages_received:215
10.93.84.53:7000> cluster nodes
942c9f97dc68198c39f425d13df0d8e3c40c5a58 10.93.84.53:7004 slave 5ac973bceab0d486c497345fe884ff54d1bb225a 0 1507806791940 5 connected
5ac973bceab0d486c497345fe884ff54d1bb225a 10.93.84.53:7001 master - 0 1507806788937 2 connected 5461-10922
a92a81532b63652bbd862be6f19a9bd8832e5e05 10.93.84.53:7005 slave cc46a4a1c0ec3f621b6b5405c6c10b7cffe73932 0 1507806790939 6 connected
cc46a4a1c0ec3f621b6b5405c6c10b7cffe73932 10.93.84.53:7002 master - 0 1507806789937 3 connected 10923-16383
6346ae8c7af7949658619fcf4021cc7aca454819 10.93.84.53:7000 myself,slave 92f62ec93a0550d962f81213ca7e9b3c9c996afd 0 0 1 connected
92f62ec93a0550d962f81213ca7e9b3c9c996afd 10.93.84.53:7003 master - 0 1507806792941 8 connected 0-5460

一些原理

redis cluster在設計的時候，就考慮到了去中心化，去中間件，也就是說，集羣中的每一個節點都是平等的關係，都是對等的，每一個節點都保存各自的數據和整個集羣的狀態。每一個節點都和其餘全部節點鏈接，並且這些鏈接保持活躍，這樣就保證了咱們只須要鏈接集羣中的任意一個節點，就能夠獲取到其餘節點的數據。

Redis集羣沒有並使用傳統的一致性哈希來分配數據，而是採用另一種叫作哈希槽（hash slot）的方式來分配的，一致性哈希對向集羣中新增和刪除實例的支持很好，可是哈希槽對向集羣新增實例或者刪除實例的話，須要額外的操做，須要手動的將slot從新平均的分配到新集羣的實例中。

redis cluster 默認分配了 16384 個slot，當咱們set一個key時，會用CRC16算法來取模獲得所屬的slot，而後將這個key分到哈希槽區間的節點上，具體算法就是：CRC16(key)%16384。

Redis 集羣會把數據存在一個master節點，而後在這個master和其對應的salve之間進行數據同步。當讀取數據時，也根據一致性哈希算法到對應的master節點獲取數據。只有當一個master 掛掉以後，纔會啓動一個對應的salve節點，充當master。

須要注意的是：必需要3個或以上的主節點，不然在建立集羣時會失敗，而且當存活的主節點數小於總節點數的一半時，整個集羣就沒法提供服務了。

五、python集羣客戶端

以Python Redis Cluster集羣的使用方式爲例，簡單說明一下如何使用，讓你們更直觀的瞭解一下Redis集羣。

Redis集羣方式與單機方式在python客戶端實現上是有很大不一樣的。

包不一樣，單機依賴包redis-py，集羣依賴包redis-py-cluster
對同時操做多個keys的命令(mset, mget, sinter, ...)，redis-py-cluster重寫StriceRedis(單機Redis操做類)了其方法，而這些方法喪失了命令的原子性。
Pipelines在集羣中的表現也不一樣了。在單機版中，pipeline是批量batch提交redis批量執行的，可是在集羣版中，是one by one提交redis執行完成直接返回，在客戶端從新組合成一個列表返回。因此集羣中只是看起來像是批量執行。
一些命令是會Fanout(扇形)發送到集羣中各個shard中執行並彙總的，如ping，keys等。一些命令是不提倡使用(blocked)，如watch，unwatch等。

我只是簡單翻譯了官網的一些內容，相信信息你們能夠參考：

http://redis-py-cluster.readthedocs.io/en/master/commands.html

http://redis-py-cluster.readthedocs.io/en/master/limitations-and-differences.html

1）安裝redis-py-cluster

簡單的經過pip安裝redis-py-cluster包。若是安裝失敗，能夠自助下載安裝。

# pip install redis-py-cluster     
Collecting redis-py-cluster
  Downloading redis_py_cluster-1.3.4-py2.py3-none-any.whl
Requirement already satisfied: redis>=2.10.2 in /home/idata/pythonEnv/idataPlatEnv/lib/python2.7/site-packages/redis-2.10.5-py2.7.egg (from redis-py-cluster)
Installing collected packages: redis-py-cluster
Successfully installed redis-py-cluster-1.3.4

2）一個簡單的demo

封裝了RedisCluster操做類，實現了一些方法，其實就是作了一層封裝。

封裝的意義是：我喜歡對這些封裝增長一些裝飾器，控制異常和重試等邏輯。

# -*- coding:utf-8 -*-

from rediscluster import StrictRedisCluster

redis_nodes = [
        {'host': '10.93.84.53', 'port': 7000},
        {'host': '10.93.84.53', 'port': 7001},
        {'host': '10.93.84.53', 'port': 7002},
        {'host': '10.93.84.53', 'port': 7003},
        {'host': '10.93.84.53', 'port': 7004},
        {'host': '10.93.84.53', 'port': 7005},
    ]


class RedisCluster(object):

    def __init__(self, redis_nodes):
        self.cluster = StrictRedisCluster(startup_nodes=redis_nodes)

    # 無差異的方法
    def set(self, name, value, ex=None, px=None, nx=False, xx=False):
        return self.cluster.set(name, value, ex, px, nx, xx)
    
    # 無差異的方法
    def get(self, name):
        return self.cluster.get(name)
    
    # 扇形發送的命令
    def cluster_info(self):
        return self.cluster.cluster_info()

    # 重寫StrictRedis的方法
    def mset(self, *args, **kwargs):
        return self.cluster.mset(args, kwargs)
    
    # 重寫StrictRedis的方法
    def mget(self, keys, *args):
        return self.cluster.mget(keys, args)
    
    
cluster = RedisCluster(redis_nodes)
cluster.cluster_info()