Redis服務之Redis Cluster

  在上一篇博客中咱們聊到了redis的高可用組件sentinel的相關配置,回顧請參考http://www.javashuo.com/article/p-kronkjpt-nq.html;sentinel在redis主從同步架構中主要起到了監控集羣master是否正常,若是master不正常,或者宕機,那麼sentinel會提高一個slave當選新的master,從而保證了redis服務的正常使用;但對於redis的單機寫入問題仍是一直存在;在sentinel+主從同步架構中,程序寫數據,始終是把讀寫請求發送給master(固然,若是有語句路由器讀寫請求是能夠分開的);這樣一來對於master來說它就承擔了全部的寫操做,很顯然這種在寫操做很是的頻繁的場景,單臺master確定沒法承受這麼大的壓力;爲了解決單機master寫入數據的瓶頸問題, redis 官方在 redis 3.0 版本以後推出了無中心架構的 redis cluster 機制;所謂無中心架構的cluster意思是集羣中的每個成員都是中心節點,每一個成員擁有整個集羣的狀態元數據,但各自又只存儲整個集羣的部分數據;對於redis cluster這種架構來說,全部redis節點使用(ping機制)互聯,若是集羣中某個節點的失敗,是須要整個集羣中超過半數的節點監測都失效纔算真正的失效;客戶端不須要 proxy 便可直接鏈接 redis,應用程序須要寫所有的 redis 服務器 IP。redis cluster 把全部的 redis node 映射到 0-16383 個槽位(slot)上,讀寫須要到指定的 redis node 上進行操做,所以有多少個 reids node 至關於 redis 併發擴展了多少倍。Redis cluster 預先分配 16384 個(slot)槽位,當須要在 redis 集羣中寫入一個 key -value 的時候,會使用 CRC16(key) mod 16384 以後的值,決定將 key 寫入值哪個槽位從而決定寫入哪個 Redis 節點上,從而有效解決redis單機讀寫瓶頸。以下所示html

  提示:客戶端每次讀寫都要通過CRC16對讀寫的KEY作hash計算,而後把hash後的結果再和16384作取模運算,最終結果會落到0-16383的某一個槽位上;咱們事先在定義集羣時,就把0-5460號槽位都分給了master01,意思就是隻要讀寫的數據的KEY經過CRC16hash計算取得的結果%16384,最後的結果在0-5460這個範圍內,就把本次的讀/寫請求調度到master01上;以此類推,範圍在5461-10922就把讀/寫請求調度到master02,範圍在10923-16383就把讀/寫請求調度到master03;經過上面的圖咱們也能看出各個master都只保存整個集羣的一部分數據,若是某一個master宕機,那麼就會致使對應master上分的槽位對應數據丟失;爲了解決各個master單點的問題,咱們還須要對每一個master作主從,以下圖node

  提示:爲了防止某個master宕機後致使數據丟失和寫操做不能執行,對應slave應該可以及時的提高爲master,相似sentinel的功能,可是咱們不須要在部署sentinel了,由於redis cluster就支持主從切換;這樣一來咱們的集羣就變成了3主3從的集羣;接下咱們就準備一個實驗環境來部署上面的架構;python

  部署redis集羣linux

  環境說明web

角色 ip地址 端口
Master01 192.168.0.41 6379
Slave01 192.168.0.42 6380
Master02 192.168.0.42 6379
Slave02 192.168.0.43 6380
Master03 192.168.0.43 6379
Slave03 192.168.0.41 6380

 

 

 

 

 

 

 

 

  

  準備啓動3臺服務器,每一個服務器上啓動兩個redis實例,端口分別是6379(master)和6380(slave),爲了防止master和slave落在同一個節點,有意將他們交叉組合,如上表格;redis

  建立redis cluster咱們須要注意每一個 redis node 節點儘可能採用相同的硬件配置,相同的密碼;全部 redis 服務器必須沒有任何數據;json

  在node01上編譯安裝redis,有關redis的編譯安裝和部署請參考http://www.javashuo.com/article/p-ckvacwaq-kp.htmlapi

  建立目錄結構ruby

[root@node01 ~]# tree /usr/local/redis/
/usr/local/redis/
├── 6379
│   ├── etc
│   │   ├── redis.conf
│   │   └── sentinel.conf
│   ├── logs
│   └── run
├── 6380
│   ├── etc
│   │   ├── redis.conf
│   │   └── sentinel.conf
│   ├── logs
│   └── run
└── bin
    ├── redis-benchmark
    ├── redis-check-aof
    ├── redis-check-rdb
    ├── redis-cli
    ├── redis-sentinel -> redis-server
    └── redis-server

9 directories, 10 files
[root@node01 ~]#

  提示:在/usr/local/redis目錄下分別建立6379和6380,而後在其下都建立etc,logs,run 等子目錄,如上所示bash

  修改配置文件

[root@node01 ~]# grep -E "^(bind|requirepass|masterauth|cluster)" /usr/local/redis/6379/etc/redis.conf 
bind 0.0.0.0
masterauth admin
requirepass admin
cluster-enabled yes
cluster-config-file redis-cluster_6379.conf
[root@node01 ~]# grep -E "^(bind|requirepass|masterauth|cluster)" /usr/local/redis/6380/etc/redis.conf   
bind 0.0.0.0
masterauth admin
requirepass admin
cluster-enabled yes
cluster-config-file redis-cluster_6380.conf
[root@node01 ~]# 

  提示:6379和6380的配置文件內容都是同樣的,惟一區別是端口不一樣,主要修改bind 監聽地址,設置密碼,開啓集羣以及指定對應集羣生成的配置文件名稱;

  把node01上的/usr/local/redis複製給node02和node03並保存到/usr/local/目錄下

  提示:我上面的三臺主機都是作了免密登陸,因此互相拷貝數據不須要輸入密碼;有關免密登陸的說明和配置能夠參考http://www.javashuo.com/article/p-rjibahxc-bk.html

  分別在node01 node02 node03上啓動redis

  到此咱們就把6個redis實例啓動了;接下來咱們來建立集羣;Redis 3 和 4 版本須要使用到集羣管理工具 redis-trib.rb,這個工具是 redis 官方推出的管理 redis 集羣的工具,集成在redis 的源碼 src 目錄下,是基於 redis 提供的集羣命令封裝成簡單、便捷、實用的操做工具,redis-trib.rb是 redis 做者用 ruby 開發完成的,因此咱們要使用redis-trib.rb首先要解決ruby環境的問題;

  安裝ruby環境

[root@node01 ~]# rz
rz waiting to receive.
 zmodem trl+C ȡ

  100%   15621 KB 15621 KB/s 00:00:01       0 Errors

[root@node01 ~]# ls
redis-4.0.9.tar.gz  ruby-2.5.5.tar.gz
[root@node01 ~]# tar xf ruby-2.5.5.tar.gz  -C /usr/local/src/
[root@node01 ~]# cd /usr/local/src/
[root@node01 src]# cd ruby-2.5.5/
[root@node01 ruby-2.5.5]# ./configure 
checking for ruby... false
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking target system type... x86_64-pc-linux-gnu
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
……省略部份內容……
checking for setjmp type... __builtin_setjmp
checking for prefix of external symbols... NONE
checking pthread.h usability... yes
checking pthread.h presence... yes
checking for pthread.h... yes
checking if make is GNU make... yes
.ext/include/x86_64-linux/ruby/config.h updated
configure: ruby library version = 2.5.0
configure: creating ./config.status
config.status: creating GNUmakefile
config.status: creating Makefile
config.status: creating ruby-2.5.pc
---
Configuration summary for ruby version 2.5.5

   * Installation prefix: /usr/local
   * exec prefix:         ${prefix}
   * arch:                x86_64-linux
   * site arch:           ${arch}
   * RUBY_BASE_NAME:      ruby
   * ruby lib prefix:     ${libdir}/${RUBY_BASE_NAME}
   * site libraries path: ${rubylibprefix}/${sitearch}
   * vendor path:         ${rubylibprefix}/vendor_ruby
   * target OS:           linux
   * compiler:            gcc
   * with pthread:        yes
   * enable shared libs:  no
   * dynamic library ext: so
   * CFLAGS:              ${optflags} ${debugflags} ${warnflags}
   * LDFLAGS:             -L. -fstack-protector -rdynamic \
                          -Wl,-export-dynamic
   * optflags:            -O3
   * debugflags:          -ggdb3
   * warnflags:           -Wall -Wextra -Wno-unused-parameter \
                          -Wno-parentheses -Wno-long-long \
                          -Wno-missing-field-initializers \
                          -Wno-tautological-compare \
                          -Wno-parentheses-equality \
                          -Wno-constant-logical-operand -Wno-self-assign \
                          -Wunused-variable -Wimplicit-int -Wpointer-arith \
                          -Wwrite-strings -Wdeclaration-after-statement \
                          -Wimplicit-function-declaration \
                          -Wdeprecated-declarations \
                          -Wno-packed-bitfield-compat \
                          -Wsuggest-attribute=noreturn \
                          -Wsuggest-attribute=format
   * strip command:       strip -S -x
   * install doc:         yes
   * man page type:       doc

---
[root@node01 ruby-2.5.5]# 

  提示:redis-trib.rb這個工具在集羣任意一臺安裝便可,不須要每臺都安裝;

  編譯

[root@node01 ruby-2.5.5]# make -j 2
        CC = gcc
        LD = ld
        LDSHARED = gcc -shared
        CFLAGS = -O3 -ggdb3 -Wall -Wextra -Wno-unused-parameter -Wno-parentheses -Wno-long-long -Wno-missing-field-initializers -Wno-tautological-compare -Wno-parentheses-equality -Wno-constant-logical-operand -Wno-self-assign -Wunused-variable -Wimplicit-int -Wpointer-arith -Wwrite-strings -Wdeclaration-after-statement -Wimplicit-function-declaration -Wdeprecated-declarations -Wno-packed-bitfield-compat -Wsuggest-attribute=noreturn -Wsuggest-attribute=format -std=gnu99 
        XCFLAGS = -D_FORTIFY_SOURCE=2 -fstack-protector -fno-strict-overflow -fvisibility=hidden -fexcess-precision=standard -DRUBY_EXPORT -fPIE
        CPPFLAGS =   -I. -I.ext/include/x86_64-linux -I./include -I. -I./enc/unicode/10.0.0
        DLDFLAGS = -fstack-protector -pie  
        SOLIBS = 
        LANG = en_US.UTF-8
        LC_ALL = 
        LC_CTYPE = 
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

compiling ./main.c
compiling dmydln.c
compiling miniinit.c
compiling dmyext.c
compiling miniprelude.c
making dummy probes.h
compiling bignum.c
compiling class.c
compiling compar.c
compiling compile.c
compiling complex.c
compiling cont.c
compiling debug.c
compiling debug_counter.c
compiling dir.c
compiling dln_find.c
compiling encoding.c
compiling enum.c
compiling enumerator.c
……省略部份內容……
*** Fix the problems, then remove these directories and try again if you want.
make[1]: Leaving directory `/usr/local/src/ruby-2.5.5'
Generating RDoc documentation
Parsing sources...
100% [871/871]  vsnprintf.c                                                                           

Generating RI format into /usr/local/src/ruby-2.5.5/.ext/rdoc...

  Files:        871

  Classes:     1324 ( 565 undocumented)
  Modules:      286 ( 121 undocumented)
  Constants:   2181 ( 555 undocumented)
  Attributes:  1066 ( 251 undocumented)
  Methods:    10080 (2161 undocumented)

  Total:      14937 (3653 undocumented)
   75.54% documented

  Elapsed: 25.0s

[root@node01 ruby-2.5.5]# 

  安裝ruby環境

[root@node01 ruby-2.5.5]# make install
        CC = gcc
        LD = ld
        LDSHARED = gcc -shared
        CFLAGS = -O3 -ggdb3 -Wall -Wextra -Wno-unused-parameter -Wno-parentheses -Wno-long-long -Wno-missing-field-initializers -Wno-tautological-compare -Wno-parentheses-equality -Wno-constant-logical-operand -Wno-self-assign -Wunused-variable -Wimplicit-int -Wpointer-arith -Wwrite-strings -Wdeclaration-after-statement -Wimplicit-function-declaration -Wdeprecated-declarations -Wno-packed-bitfield-compat -Wsuggest-attribute=noreturn -Wsuggest-attribute=format -std=gnu99 
        XCFLAGS = -D_FORTIFY_SOURCE=2 -fstack-protector -fno-strict-overflow -fvisibility=hidden -fexcess-precision=standard -DRUBY_EXPORT -fPIE
        CPPFLAGS =   -I. -I.ext/include/x86_64-linux -I./include -I. -I./enc/unicode/10.0.0
        DLDFLAGS = -fstack-protector -pie  
        SOLIBS = 
        LANG = en_US.UTF-8
        LC_ALL = 
        LC_CTYPE = 
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

generating enc.mk
making srcs under enc
make[1]: Entering directory `/usr/local/src/ruby-2.5.5'
make[1]: Nothing to be done for `srcs'.
make[1]: Leaving directory `/usr/local/src/ruby-2.5.5'
generating transdb.h
transdb.h unchanged
generating makefiles ext/configure-ext.mk
ext/configure-ext.mk updated
make[1]: Entering directory `/usr/local/src/ruby-2.5.5'
make[1]: Leaving directory `/usr/local/src/ruby-2.5.5'
generating makefile exts.mk
exts.mk updated
make[1]: Entering directory `/usr/local/src/ruby-2.5.5'
make[2]: Entering directory `/usr/local/src/ruby-2.5.5/ext/fcntl'
make[2]: Leaving directory `/usr/local/src/ruby-2.5.5/ext/fcntl'
make[2]: Entering directory `/usr/local/src/ruby-2.5.5/ext/rubyvm'
……省略部份內容……
installing binary commands:         /usr/local/bin
installing base libraries:          /usr/local/lib
installing arch files:              /usr/local/lib/ruby/2.5.0/x86_64-linux
installing pkgconfig data:          /usr/local/lib/pkgconfig
installing command scripts:         /usr/local/bin
installing library scripts:         /usr/local/lib/ruby/2.5.0
installing common headers:          /usr/local/include/ruby-2.5.0
installing manpages:                /usr/local/share/man/man1
installing extension objects:       /usr/local/lib/ruby/2.5.0/x86_64-linux
installing extension objects:       /usr/local/lib/ruby/site_ruby/2.5.0/x86_64-linux
installing extension objects:       /usr/local/lib/ruby/vendor_ruby/2.5.0/x86_64-linux
installing extension headers:       /usr/local/include/ruby-2.5.0/x86_64-linux
installing extension scripts:       /usr/local/lib/ruby/2.5.0
installing extension scripts:       /usr/local/lib/ruby/site_ruby/2.5.0
installing extension scripts:       /usr/local/lib/ruby/vendor_ruby/2.5.0
installing extension headers:       /usr/local/include/ruby-2.5.0/ruby
installing default gems from lib:   /usr/local/lib/ruby/gems/2.5.0 (build_info, cache, doc, extensions, gems, specifications)
                                    cmath 1.0.0
                                    csv 1.0.0
                                    fileutils 1.0.2
                                    ipaddr 1.2.0
                                    rdoc 6.0.1
                                    scanf 1.0.0
                                    webrick 1.4.2
installing default gems from ext:   /usr/local/lib/ruby/gems/2.5.0 (build_info, cache, doc, extensions, gems, specifications)
                                    bigdecimal 1.3.4
                                    date 1.0.0
                                    etc 1.0.0
                                    fcntl 1.0.0
                                    fiddle 1.0.0
                                    io-console 0.4.6
                                    json 2.1.0
                                    psych 3.0.2
                                    sdbm 1.0.0
                                    stringio 0.0.1
                                    strscan 1.0.0
installing bundled gems:            /usr/local/lib/ruby/gems/2.5.0 (build_info, cache, doc, extensions, gems, specifications)
                                    minitest 5.10.3
                                    rake 12.3.0
                                    xmlrpc 0.3.0
                                    did_you_mean 1.2.0
                                    net-telnet 0.1.1
                                    power_assert 1.1.1
                                    test-unit 3.2.7
installing rdoc:                    /usr/local/share/ri/2.5.0/system
installing capi-docs:               /usr/local/share/doc/ruby
[root@node01 ruby-2.5.5]# 

  提示:到此ruby環境就安裝好了,這裏注意一點,yum安裝的ruby版本有點低,因此咱們這裏選擇編譯安裝;

  gem安裝redis包

[root@node01 ruby-2.5.5]# gem install redis
ERROR:  Loading command: install (LoadError)
        cannot load such file -- zlib
ERROR:  While executing gem ... (NoMethodError)
    undefined method `invoke_with_build_args' for nil:NilClass
[root@node01 ruby-2.5.5]#

  提示:gem安裝提示咱們沒有zlib,遇到這個問題咱們用yum安裝zlib-devel包,而後在把zlib集成到ruby環境便可;

[root@node01 ruby-2.5.5]# yum install zlib-devel
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package zlib-devel.x86_64 0:1.2.7-18.el7 will be installed
--> Processing Dependency: zlib = 1.2.7-18.el7 for package: zlib-devel-1.2.7-18.el7.x86_64
--> Running transaction check
---> Package zlib.x86_64 0:1.2.7-17.el7 will be updated
---> Package zlib.x86_64 0:1.2.7-18.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

===========================================================================================================
 Package                    Arch                   Version                      Repository            Size
===========================================================================================================
Installing:
 zlib-devel                 x86_64                 1.2.7-18.el7                 base                  50 k
Updating for dependencies:
 zlib                       x86_64                 1.2.7-18.el7                 base                  90 k

Transaction Summary
===========================================================================================================
Install  1 Package
Upgrade             ( 1 Dependent package)

Total download size: 140 k
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(1/2): zlib-devel-1.2.7-18.el7.x86_64.rpm                                           |  50 kB  00:00:00     
  /2): zlib-1.2.7-18.el7.x86_64.rpm                                                 |  90 kB  00:00:00     
-----------------------------------------------------------------------------------------------------------
Total                                                                      509 kB/s | 140 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : zlib-1.2.7-18.el7.x86_64                                                                1/3 
  Installing : zlib-devel-1.2.7-18.el7.x86_64                                                          2/3 
  Cleanup    : zlib-1.2.7-17.el7.x86_64                                                                3/3 
  Verifying  : zlib-devel-1.2.7-18.el7.x86_64                                                          1/3 
  Verifying  : zlib-1.2.7-18.el7.x86_64                                                                2/3 
  Verifying  : zlib-1.2.7-17.el7.x86_64                                                                3/3 

Installed:
  zlib-devel.x86_64 0:1.2.7-18.el7                                                                         

Dependency Updated:
  zlib.x86_64 0:1.2.7-18.el7                                                                               

Complete!
[root@node01 ruby-2.5.5]#

  集成zlib庫到Ruby環境

[root@node01 ruby-2.5.5]# cd ext/zlib/
[root@node01 zlib]# ruby extconf.rb
checking for deflateReset() in -lz... yes
checking for zlib.h... yes
checking for crc32_combine() in zlib.h... yes
checking for adler32_combine() in zlib.h... yes
checking for z_crc_t in zlib.h... yes
creating Makefile
[root@node01 zlib]# make 
make: *** No rule to make target `/include/ruby.h', needed by `zlib.o'.  Stop.
[root@node01 zlib]#

  提示:遇到這種問題,咱們須要打開上面生成的Makefile 把zlib.o: $(top_srcdir)/include/ruby.h替換爲zlib.o: ../../include/ruby.h便可

  提示:修改好Makefile,再次make

"Makefile" 282L, 8468C written
[root@node01 zlib]# make
compiling zlib.c
linking shared-object zlib.so
[root@node01 zlib]# make install
/usr/bin/install -c -m 0755 zlib.so /usr/local/lib/ruby/site_ruby/2.5.0/x86_64-linux
[root@node01 zlib]#

  再次使用gem安裝redis包

[root@node01 zlib]# gem install redis
ERROR:  While executing gem ... (Gem::Exception)
    Unable to require openssl, install OpenSSL and rebuild Ruby (preferred) or use non-HTTPS sources
[root@node01 zlib]#

  提示:它提示咱們缺乏openssl庫,處理方法安裝openssl-devel包,而後把openssl庫集成到ruby環境中;

  安裝openssl-devel包

[root@node01 zlib]# yum install openssl-devel
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package openssl-devel.x86_64 1:1.0.2k-19.el7 will be installed
--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-19.el7 for package: 1:openssl-devel-1.0.2k-19.el7.x86_64
--> Processing Dependency: krb5-devel(x86-64) for package: 1:openssl-devel-1.0.2k-19.el7.x86_64
--> Running transaction check
---> Package krb5-devel.x86_64 0:1.15.1-46.el7 will be installed
……省略部份內容……
Installed:
  openssl-devel.x86_64 1:1.0.2k-19.el7                                                                     

Dependency Installed:
  keyutils-libs-devel.x86_64 0:1.5.8-3.el7                krb5-devel.x86_64 0:1.15.1-46.el7                
  libcom_err-devel.x86_64 0:1.42.9-17.el7                 libkadm5.x86_64 0:1.15.1-46.el7                  
  libselinux-devel.x86_64 0:2.5-15.el7                    libsepol-devel.x86_64 0:2.5-10.el7               
  libverto-devel.x86_64 0:0.2.5-4.el7                     pcre-devel.x86_64 0:8.32-17.el7                  

Dependency Updated:
  e2fsprogs.x86_64 0:1.42.9-17.el7                    e2fsprogs-libs.x86_64 0:1.42.9-17.el7               
  krb5-libs.x86_64 0:1.15.1-46.el7                    libcom_err.x86_64 0:1.42.9-17.el7                   
  libselinux.x86_64 0:2.5-15.el7                      libselinux-python.x86_64 0:2.5-15.el7               
  libselinux-utils.x86_64 0:2.5-15.el7                libsepol.x86_64 0:2.5-10.el7                        
  libss.x86_64 0:1.42.9-17.el7                        openssl.x86_64 1:1.0.2k-19.el7                      
  openssl-libs.x86_64 1:1.0.2k-19.el7                

Complete!
[root@node01 zlib]#

  集成openssl到ruby環境中

[root@node01 zlib]# cd ..
[root@node01 ext]# ls
bigdecimal        date      fcntl   json      psych     ripper        Setup.nt  -test-
cgi               dbm       fiber   nkf       pty       rubyvm        socket    win32
configure-ext.mk  digest    fiddle  objspace  racc      sdbm          stringio  win32ole
continuation      etc       gdbm    openssl   rbconfig  Setup         strscan   zlib
coverage          extmk.rb  io      pathname  readline  Setup.atheos  syslog
[root@node01 ext]# cd openssl/
[root@node01 openssl]# ls
depend             ossl_asn1.c    ossl_digest.c   ossl_ocsp.c      ossl_pkey_rsa.c     ossl_x509ext.c
deprecation.rb     ossl_asn1.h    ossl_digest.h   ossl_ocsp.h      ossl_rand.c         ossl_x509.h
extconf.rb         ossl_bio.c     ossl_engine.c   ossl_pkcs12.c    ossl_rand.h         ossl_x509name.c
exts.mk            ossl_bio.h     ossl_engine.h   ossl_pkcs12.h    ossl_ssl.c          ossl_x509req.c
History.md         ossl_bn.c      ossl.h          ossl_pkcs7.c     ossl_ssl.h          ossl_x509revoked.c
lib                ossl_bn.h      ossl_hmac.c     ossl_pkcs7.h     ossl_ssl_session.c  ossl_x509store.c
Makefile           ossl.c         ossl_hmac.h     ossl_pkey.c      ossl_version.h      ruby_missing.h
mkmf.log           ossl_cipher.c  ossl_kdf.c      ossl_pkey_dh.c   ossl_x509attr.c
openssl.gemspec    ossl_cipher.h  ossl_kdf.h      ossl_pkey_dsa.c  ossl_x509.c
openssl_missing.c  ossl_config.c  ossl_ns_spki.c  ossl_pkey_ec.c   ossl_x509cert.c
openssl_missing.h  ossl_config.h  ossl_ns_spki.h  ossl_pkey.h      ossl_x509crl.c
[root@node01 openssl]# ruby extconf.rb 
checking for t_open() in -lnsl... no
checking for socket() in -lsocket... no
checking for openssl/ssl.h... yes
checking for OpenSSL version is 1.0.1 or later... yes
checking for RAND_egd()... yes
……省略部份內容……
checking for X509_get0_notBefore()... no
checking for SSL_SESSION_get_protocol_version()... no
checking for EVP_PBE_scrypt()... no
creating extconf.h
creating Makefile
[root@node01 openssl]# make
compiling openssl_missing.c
make: *** No rule to make target `/include/ruby.h', needed by `ossl.o'.  Stop.
[root@node01 openssl]# 

  提示:這個錯誤和剛纔繼承zlib庫到ruby環境中相似,處理方式同上面的處理方式同樣修改Makefile文件,在makefile定義變量的位置增長top_srcdir = ../..

  再次make

Makefile" 1458L, 49182C written
[root@node01 openssl]# make
compiling ossl.c
compiling ossl_asn1.c
compiling ossl_bio.c
compiling ossl_bn.c
compiling ossl_cipher.c
compiling ossl_config.c
compiling ossl_digest.c
compiling ossl_engine.c
compiling ossl_hmac.c
compiling ossl_kdf.c
compiling ossl_ns_spki.c
compiling ossl_ocsp.c
compiling ossl_pkcs12.c
compiling ossl_pkcs7.c
compiling ossl_pkey.c
compiling ossl_pkey_dh.c
compiling ossl_pkey_dsa.c
compiling ossl_pkey_ec.c
compiling ossl_pkey_rsa.c
compiling ossl_rand.c
compiling ossl_ssl.c
compiling ossl_ssl_session.c
compiling ossl_x509.c
compiling ossl_x509attr.c
compiling ossl_x509cert.c
compiling ossl_x509crl.c
compiling ossl_x509ext.c
compiling ossl_x509name.c
compiling ossl_x509req.c
compiling ossl_x509revoked.c
compiling ossl_x509store.c
linking shared-object openssl.so
[root@node01 openssl]# make install
/usr/bin/install -c -m 0755 openssl.so /usr/local/lib/ruby/site_ruby/2.5.0/x86_64-linux
installing default openssl libraries
[root@node01 openssl]# 

  再次用gem安裝redis包

[root@node01 openssl]# gem install redis
Fetching: redis-4.2.1.gem (100%)
Successfully installed redis-4.2.1
Parsing documentation for redis-4.2.1
Installing ri documentation for redis-4.2.1
Done installing documentation for redis after 0 seconds
1 gem installed
[root@node01 openssl]# 

  提示:到此ruby環境就準備好了,redis包也安裝好了,接下來咱們把redis-trib.rb鏈接到/usr/bin/下,而後執行redis-trib.rb,若是可以正常執行說明ruby環境沒有問題,若是不能則還須要調整

  軟鏈接redis-trib.rb到/usr/bin/

[root@node01 openssl]# find /usr/local/src/redis-4.0.9/ -name "redis-trib.rb"
/usr/local/src/redis-4.0.9/src/redis-trib.rb
[root@node01 openssl]# ln -s /usr/local/src/redis-4.0.9/src/redis-trib.rb /usr/bin/
[root@node01 openssl]# redis-trib.rb 
Usage: redis-trib <command> <options> <arguments ...>

  create          host1:port1 ... hostN:portN
                  --replicas <arg>
  check           host:port
  info            host:port
  fix             host:port
                  --timeout <arg>
  reshard         host:port
                  --from <arg>
                  --to <arg>
                  --slots <arg>
                  --yes
                  --timeout <arg>
                  --pipeline <arg>
  rebalance       host:port
                  --weight <arg>
                  --auto-weights
                  --use-empty-masters
                  --timeout <arg>
                  --simulate
                  --pipeline <arg>
                  --threshold <arg>
  add-node        new_host:new_port existing_host:existing_port
                  --slave
                  --master-id <arg>
  del-node        host:port node_id
  set-timeout     host:port milliseconds
  call            host:port command arg arg .. arg
  import          host:port
                  --from <arg>
                  --copy
                  --replace
  help            (show this help)

For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.
[root@node01 openssl]#

  提示:能夠看到redis-trib.rb能夠正常打印出幫助信息,說明當前ruby環境可以支持redis-trib.rb運行;

  添加密碼到剛纔下載的redis包的庫文件中

  提示:這裏須要把對應密碼字符串用引號把它引發來;

  建立集羣

[root@node01 ~]# redis-trib.rb create --replicas 1 192.168.0.41:6379 192.168.0.42:6379 192.168.0.43:6379 192.168.0.41:6380 192.168.0.42:6380 192.168.0.43:6380
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.0.41:6379
192.168.0.42:6379
192.168.0.43:6379
Adding replica 192.168.0.42:6380 to 192.168.0.41:6379
Adding replica 192.168.0.43:6380 to 192.168.0.42:6379
Adding replica 192.168.0.41:6380 to 192.168.0.43:6379
M: 8c785e6ec3f8f7ff4fb7768765da8b8a93f26855 192.168.0.41:6379
   slots:0-5460 (5461 slots) master
M: e99b0b450e78719d63520cb6efc068d5e8d4d081 192.168.0.42:6379
   slots:5461-10922 (5462 slots) master
M: a7ace08c36f7d55c4f28463d72865aa1ff74829e 192.168.0.43:6379
   slots:10923-16383 (5461 slots) master
S: 62ece0b80b83c0f1f078b07fc1687bb8376f76b3 192.168.0.41:6380
   replicates a7ace08c36f7d55c4f28463d72865aa1ff74829e
S: dbfff4c49a94c0ee55d14401ccc9245af3655427 192.168.0.42:6380
   replicates 8c785e6ec3f8f7ff4fb7768765da8b8a93f26855
S: 91169e71359deed96f8778cf31c823dbd6ded350 192.168.0.43:6380
   replicates e99b0b450e78719d63520cb6efc068d5e8d4d081
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join......
>>> Performing Cluster Check (using node 192.168.0.41:6379)
M: 8c785e6ec3f8f7ff4fb7768765da8b8a93f26855 192.168.0.41:6379
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: 91169e71359deed96f8778cf31c823dbd6ded350 192.168.0.43:6380
   slots: (0 slots) slave
   replicates e99b0b450e78719d63520cb6efc068d5e8d4d081
M: e99b0b450e78719d63520cb6efc068d5e8d4d081 192.168.0.42:6379
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
M: a7ace08c36f7d55c4f28463d72865aa1ff74829e 192.168.0.43:6379
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
S: dbfff4c49a94c0ee55d14401ccc9245af3655427 192.168.0.42:6380
   slots: (0 slots) slave
   replicates 8c785e6ec3f8f7ff4fb7768765da8b8a93f26855
S: 62ece0b80b83c0f1f078b07fc1687bb8376f76b3 192.168.0.41:6380
   slots: (0 slots) slave
   replicates a7ace08c36f7d55c4f28463d72865aa1ff74829e
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@node01 ~]# 

  提示:從上面的信息能夠了解到,咱們給指定了6個redis實例的地址和端口,它給咱們建立了3個master3個slave,而且在3個master上平均分配了16384個槽位;若是能看到後面的ok all 16384 slots covered,說明集羣建立成功;--replicates用來指定集羣中每一個master的副本數量,1表示每一個master有一個副本;

  查看集羣狀態信息

[root@node01 ~]# redis-trib.rb info 192.168.0.41:6379                    
192.168.0.41:6379 (8c785e6e...) -> 0 keys | 5461 slots | 1 slaves.
192.168.0.42:6379 (e99b0b45...) -> 0 keys | 5462 slots | 1 slaves.
192.168.0.43:6379 (a7ace08c...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
[root@node01 ~]# redis-trib.rb check 192.168.0.41:6379     
>>> Performing Cluster Check (using node 192.168.0.41:6379)
M: 8c785e6ec3f8f7ff4fb7768765da8b8a93f26855 192.168.0.41:6379
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: 91169e71359deed96f8778cf31c823dbd6ded350 192.168.0.43:6380
   slots: (0 slots) slave
   replicates e99b0b450e78719d63520cb6efc068d5e8d4d081
M: e99b0b450e78719d63520cb6efc068d5e8d4d081 192.168.0.42:6379
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
M: a7ace08c36f7d55c4f28463d72865aa1ff74829e 192.168.0.43:6379
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
S: dbfff4c49a94c0ee55d14401ccc9245af3655427 192.168.0.42:6380
   slots: (0 slots) slave
   replicates 8c785e6ec3f8f7ff4fb7768765da8b8a93f26855
S: 62ece0b80b83c0f1f078b07fc1687bb8376f76b3 192.168.0.41:6380
   slots: (0 slots) slave
   replicates a7ace08c36f7d55c4f28463d72865aa1ff74829e
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@node01 ~]# 

  提示:從上面的信息能夠看到192.168.0.41/42/43的6379端口都是master,而且都有一個slave節點;

  確認master狀態

[root@node01 ~]# redis-cli -h 192.168.0.41
192.168.0.41:6379> AUTH admin
OK
192.168.0.41:6379> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:606
cluster_stats_messages_pong_sent:586
cluster_stats_messages_sent:1192
cluster_stats_messages_ping_received:581
cluster_stats_messages_pong_received:606
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:1192
192.168.0.41:6379> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.0.42,port=6380,state=online,offset=854,lag=0
master_replid:303943ae1e9f54e3525abda2fd1efff905bf08e3
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:854
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:854
192.168.0.41:6379> quit
[root@node01 ~]# redis-cli -h 192.168.0.42
192.168.0.42:6379> auth admin
OK
192.168.0.42:6379> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.0.43,port=6380,state=online,offset=896,lag=0
master_replid:f4e1b75b8f30be622868814fc4618a4584f3aa32
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:896
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:896
192.168.0.42:6379> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:2
cluster_stats_messages_ping_sent:662
cluster_stats_messages_pong_sent:645
cluster_stats_messages_meet_sent:5
cluster_stats_messages_sent:1312
cluster_stats_messages_ping_received:644
cluster_stats_messages_pong_received:667
cluster_stats_messages_meet_received:1
cluster_stats_messages_received:1312
192.168.0.42:6379> quit
[root@node01 ~]# redis-cli -h 192.168.0.43
192.168.0.43:6379> AUTH admin
OK
192.168.0.43:6379> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:3
cluster_stats_messages_ping_sent:683
cluster_stats_messages_pong_sent:787
cluster_stats_messages_meet_sent:4
cluster_stats_messages_sent:1474
cluster_stats_messages_ping_received:786
cluster_stats_messages_pong_received:687
cluster_stats_messages_meet_received:1
cluster_stats_messages_received:1474
192.168.0.43:6379> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.0.41,port=6380,state=online,offset=952,lag=1
master_replid:7cbc7d52452717cd49a0e543f6efdccafe5fe1bd
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:952
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:952
192.168.0.43:6379> quit
[root@node01 ~]# 

  提示:能夠看到集羣狀態都是正常ok的,而且41的6379對應42的6380,42的6379對應43的6380,43的6379對應41的6380,恰好master和slave都是錯開的;

  查看集羣各node對應關係

  提示:上面的node對應是依據id號來的。到此redis集羣就正常的跑起來了;

  驗證:在master上任意寫入一個數據看看會發生什麼?

  提示:在master01上寫入k1,它提示咱們到192.168.0.43:6379上去寫;這是由於咱們寫入k1計算的槽位是12706,這個槽位是在master03上,因此在master01上就不能正常的寫入;這同時也告訴咱們,要使用redis集羣,客戶端還必須可以智能去理解redis協議,經過集羣返回的信息,把對應請求調度到對應master上執行;

  驗證讀請求

  提示:從上面的信息能夠肯定一點,這三個主節點他們互相數據是徹底隔離的,也就說master01不知道master02上到底有沒有對應的數據,它只知道應該去master02上去查看才知道;這就是咱們以前說的,每一個節點擁有整個集羣的所有狀態信息(元數據),而每一個節點只擁有部分數據在本節點;

  驗證:把任意一個master節點宕機,看看對應slave是否可以被提高爲master,接管master呢?

  提示:能夠看到當master01宕機之後,對應slave就被提高爲master;

  查看master01的配置文件是否有改動?

[root@node01 ~]# cat /usr/local/redis/redis-cluster_6379.conf 
91169e71359deed96f8778cf31c823dbd6ded350 192.168.0.43:6380@16380 master - 0 1596648345000 8 connected 5461-10922
dbfff4c49a94c0ee55d14401ccc9245af3655427 192.168.0.42:6380@16380 slave 8c785e6ec3f8f7ff4fb7768765da8b8a93f26855 0 1596648345510 9 connected
62ece0b80b83c0f1f078b07fc1687bb8376f76b3 192.168.0.41:6380@16380 slave a7ace08c36f7d55c4f28463d72865aa1ff74829e 0 1596648343893 4 connected
e99b0b450e78719d63520cb6efc068d5e8d4d081 192.168.0.42:6379@16379 slave 91169e71359deed96f8778cf31c823dbd6ded350 0 1596648344903 8 connected
8c785e6ec3f8f7ff4fb7768765da8b8a93f26855 192.168.0.41:6379@16379 myself,master - 0 1596648345000 9 connected 0-5460
a7ace08c36f7d55c4f28463d72865aa1ff74829e 192.168.0.43:6379@16379 master - 0 1596648345000 3 connected 10923-16383
vars currentEpoch 9 lastVoteEpoch 0
[root@node01 ~]# ps -ef |grep redis
root      15980      1  0 Aug05 ?        00:00:17 redis-server 0.0.0.0:6380 [cluster]
root      30867      1  0 01:19 ?        00:00:02 redis-server 0.0.0.0:6379 [cluster]
root      30949  14690  0 01:30 pts/0    00:00:00 grep --color=auto redis
[root@node01 ~]# redis-cli -a admin
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.0.42,port=6380,state=online,offset=4476,lag=0
master_replid:ed690ba2766e6e334ffc298fc6aba8be80465aa8
master_replid2:776b5f996d78171e7c5ab5419d65c6c14a253bda
master_repl_offset:4476
second_repl_offset:4071
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:3595
repl_backlog_histlen:882
127.0.0.1:6379> quit
[root@node01 ~]# kill -9 30867
[root@node01 ~]# redis-cli -h 192.168.0.42 -p 6380
192.168.0.42:6380> AUTH admin
OK
192.168.0.42:6380> info replication
# Replication
role:master
connected_slaves:0
master_replid:d46c0a3761b61352b1fc75986ab3343e04a89b83
master_replid2:ed690ba2766e6e334ffc298fc6aba8be80465aa8
master_repl_offset:4518
second_repl_offset:4519
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:4071
repl_backlog_histlen:448
192.168.0.42:6380> quit
[root@node01 ~]# cat /usr/local/redis/redis-cluster_6379.conf
91169e71359deed96f8778cf31c823dbd6ded350 192.168.0.43:6380@16380 master - 0 1596648345000 8 connected 5461-10922
dbfff4c49a94c0ee55d14401ccc9245af3655427 192.168.0.42:6380@16380 slave 8c785e6ec3f8f7ff4fb7768765da8b8a93f26855 0 1596648345510 9 connected
62ece0b80b83c0f1f078b07fc1687bb8376f76b3 192.168.0.41:6380@16380 slave a7ace08c36f7d55c4f28463d72865aa1ff74829e 0 1596648343893 4 connected
e99b0b450e78719d63520cb6efc068d5e8d4d081 192.168.0.42:6379@16379 slave 91169e71359deed96f8778cf31c823dbd6ded350 0 1596648344903 8 connected
8c785e6ec3f8f7ff4fb7768765da8b8a93f26855 192.168.0.41:6379@16379 myself,master - 0 1596648345000 9 connected 0-5460
a7ace08c36f7d55c4f28463d72865aa1ff74829e 192.168.0.43:6379@16379 master - 0 1596648345000 3 connected 10923-16383
vars currentEpoch 9 lastVoteEpoch 0
[root@node01 ~]# grep slaveof /usr/local/redis/6379/etc/redis.conf 
# Master-Slave replication. Use slaveof to make a Redis instance a copy of
# slaveof <masterip> <masterport>
[root@node01 ~]# 

  提示:從上面的信息能夠看到,在master01沒有宕機前和宕機之後slave被提高爲master後,對應的配置信息都沒有發生變化;我上面爲了驗證配置文件是否發生變化,先是把master01上線,而後把對應slave下線在上線,還原master01爲master角色;

  從新把master上線,看看它的配置文件有什麼變化?角色還會是master嗎?

[root@node01 ~]# redis-server /usr/local/redis/6379/etc/redis.conf     
[root@node01 ~]# ss -tnl
State       Recv-Q Send-Q                                        Local Address:Port                                                       Peer Address:Port              
LISTEN      0      511                                                       *:6379                                                                  *:*                  
LISTEN      0      511                                                       *:6380                                                                  *:*                  
LISTEN      0      128                                                       *:22                                                                    *:*                  
LISTEN      0      100                                               127.0.0.1:25                                                                    *:*                  
LISTEN      0      511                                                       *:16379                                                                 *:*                  
LISTEN      0      511                                                       *:16380                                                                 *:*                  
LISTEN      0      128                                                      :::22                                                                   :::*                  
LISTEN      0      100                                                     ::1:25                                                                   :::*                  
[root@node01 ~]# cat /usr/local/redis/redis-cluster_6379.conf 
91169e71359deed96f8778cf31c823dbd6ded350 192.168.0.43:6380@16380 master - 1596649311329 1596649311326 8 connected 5461-10922
8c785e6ec3f8f7ff4fb7768765da8b8a93f26855 192.168.0.41:6379@16379 myself,slave dbfff4c49a94c0ee55d14401ccc9245af3655427 0 1596649311326 9 connected
e99b0b450e78719d63520cb6efc068d5e8d4d081 192.168.0.42:6379@16379 slave 91169e71359deed96f8778cf31c823dbd6ded350 0 1596649311331 8 connected
a7ace08c36f7d55c4f28463d72865aa1ff74829e 192.168.0.43:6379@16379 master - 0 1596649311332 3 connected 10923-16383
62ece0b80b83c0f1f078b07fc1687bb8376f76b3 192.168.0.41:6380@16380 slave a7ace08c36f7d55c4f28463d72865aa1ff74829e 0 1596649311331 4 connected
dbfff4c49a94c0ee55d14401ccc9245af3655427 192.168.0.42:6380@16380 master - 0 1596649311332 10 connected 0-5460
vars currentEpoch 10 lastVoteEpoch 0
[root@node01 ~]# redis-cli 
127.0.0.1:6379> AUTH admin
OK
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:192.168.0.42
master_port:6380
master_link_status:up
master_last_io_seconds_ago:5
master_sync_in_progress:0
slave_repl_offset:4686
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:d46c0a3761b61352b1fc75986ab3343e04a89b83
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:4686
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:4519
repl_backlog_histlen:168
127.0.0.1:6379> 

  提示:能夠看到,從新把master01上線之後,它自動降級爲slave了,而且配置文件也從master需改爲slave了;這應該是redis cluster在檢測到192.168.0.41:6379上線後,觸發把內存中的配置同步到對應文件,使得對應配置文件也被修改了;到此一個redis cluster 就搭建測試完成了;

相關文章
相關標籤/搜索