codis集羣搭建

github文檔https://github.com/CodisLabs/codis/blob/master/doc/tutorial_zh.md

####環境

codis-proxy+codis-config：

192.168.10.93 proxy_1

192.168.10.94 proxy_2

192.168.10.95 proxy_3

zookeeper cluster： 192.168.10.91-93:2181

codis-server cluster 1： 192.168.10.91-92:6379

codis-server cluster 2： 192.168.10.91-92:6380

codis-server cluster 3: 192.168.10.93:6379 192.168.10.93:6380

####zookeeper集羣搭建 請見http://my.oschina.net/u/1791060/blog/654107

####安裝go # yum install go

# vi /etc/profile

export GOPATH=/usr

# source /etc/profile

####下載編譯codis

# yum install git -y

# go get -u -d github.com/CodisLabs/codis

# cd /usr/src/github.com/CodisLabs/codis/

# make

# make gotest

####配置codis # vi /usr/src/github.com/CodisLabs/codis/config.ini

coordinator=zookeeper
zk=192.168.10.91:2181,192.168.10.92:2181,192.168.10.93:2181
product=test
dashboard_addr=192.168.10.95:18087
password=
backend_ping_period=5
session_max_timeout=1800
session_max_bufsize=131072
session_max_pipeline=1024
zk_session_timeout=30000
proxy_id=proxy_3       #每臺proxy和config都不同

####啓動codis

1. 在192.168.10.95上啓動dashboard

# nohup /usr/src/github.com/CodisLabs/codis/bin/codis-config -c /usr/src/github.com/CodisLabs/codis/config.ini dashboard & #nohup啓動dashboard

2016/04/07 21:58:03 dashboard.go:160: [INFO] dashboard listening on addr: :18087
2016/04/07 21:58:03 dashboard.go:143: [INFO] dashboard node created: /zk/codis/db_test/dashboard, {"addr": "192.168.10.95:18087", "pid": 13708}
2016/04/07 21:58:03 dashboard.go:144: [WARN] ********** Attention **********
2016/04/07 21:58:03 dashboard.go:145: [WARN] You should use `kill {pid}` rather than `kill -9 {pid}` to stop me,
2016/04/07 21:58:03 dashboard.go:146: [WARN] or the node resisted on zk will not be cleaned when I'm quiting and you must remove it manually
......

查看http://192.168.10.95:18087/admin/是否可正常打開

2.初始化slot信息 # /usr/src/github.com/CodisLabs/codis/bin/codis-config -c /usr/src/github.com/CodisLabs/codis/config.ini slot init

{
  "msg": "OK",
  "ret": 0
}

3.啓動3臺codis-server #和redis-server啓動同樣 # vi /etc/codis.conf #配置文件以下，本身改

daemonize yes
pidfile "/tmp/codis.pid"
port 6379
timeout 300
loglevel debug
logfile "/tmp/codis.log"
databases 16
slave-read-only no
save 900 5
save 300 50
save 60 50000
rdbcompression yes
dbfilename "dump.rdb"
dir "/tmp/"
appendonly no
appendfsync always
maxmemory 1gb

# vi /etc/codis2.conf #配置文件以下，本身改

daemonize yes
pidfile "/tmp/codis2.pid"
port 6380
timeout 300
loglevel debug
logfile "/tmp/codis2.log"
databases 16
slave-read-only no
save 900 5
save 300 50
save 60 50000
rdbcompression yes
dbfilename "dump2.rdb"
dir "/tmp/"
appendonly no
appendfsync always
maxmemory 1gb

# /usr/src/github.com/CodisLabs/codis/bin/codis-server /etc/codis.conf #啓動codis-server

# /usr/src/github.com/CodisLabs/codis/bin/codis-server /etc/codis2.conf

####添加server group,id爲最小1開始的證書，或者在dashboard上添加 # /usr/src/github.com/CodisLabs/codis/bin/codis-config -c /usr/src/github.com/CodisLabs/codis/config.ini server -h

usage:
        codis-config server list
        codis-config server add <group_id> <redis_addr> <role>
        codis-config server remove <group_id> <redis_addr>
        codis-config server promote <group_id> <redis_addr>
        codis-config server add-group <group_id>
        codis-config server remove-group <group_id>

• 添加group1，192.168.10.91-92:6379 原來想把三臺 7379的codis都加進group 1，可是發現始終只有1臺是offline的就算了 # /usr/src/github.com/CodisLabs/codis/bin/codis-config -c

/usr/src/github.com/CodisLabs/codis/config.ini server add 1 192.168.10.91:6379 master { "msg": "OK", "ret": 0 }

# /usr/src/github.com/CodisLabs/codis/bin/codis-config -c

/usr/src/github.com/CodisLabs/codis/config.ini server add 1 192.168.10.92:6379 slave { "msg": "OK", "ret": 0 }

-添加group2，192.168.10.91-92:6380 略，用命令行添加吧，第一次用dashboard添加的時候dashboard直接崩潰了 結果以下：

-設置slot id 默認slot爲0-1023，共1024個slot,設置group1爲0-511,group2 512-1023

# /usr/src/github.com/CodisLabs/codis/bin/codis-config -c /usr/src/github.com/CodisLabs/codis/config.ini slot -h

usage:
        codis-config slot init [-f]
        codis-config slot info <slot_id>
        codis-config slot set <slot_id> <group_id> <status>
        codis-config slot range-set <slot_from> <slot_to> <group_id> <status>
        codis-config slot migrate <slot_from> <slot_to> <group_id> [--delay=<delay_time_in_ms>]
        codis-config slot rebalance [--delay=<delay_time_in_ms>]

# /usr/src/github.com/CodisLabs/codis/bin/codis-config -c /usr/src/github.com/CodisLabs/codis/config.ini slot range-set 0 511 1 online

{
  "msg": "OK",
  "ret": 0
}

# /usr/src/github.com/CodisLabs/codis/bin/codis-config -c /usr/src/github.com/CodisLabs/codis/config.ini slot range-set 512 1023 2 online

{
  "msg": "OK",
  "ret": 0
}

####啓動codis-proxy # /usr/src/github.com/CodisLabs/codis/bin/codis-proxy -c /usr/src/github.com/CodisLabs/codis/config.ini -L /tmp/codis-proxy.log --cpu=2 --addr=0.0.0.0:19000 --http-addr=0.0.0.0:11000

_____  ____    ____/ /  (_)  _____
 / ___/ / __ \  / __  /  / /  / ___/
/ /__  / /_/ / / /_/ /  / /  (__  )
\___/  \____/  \__,_/  /_/  /____/

####使用codis-proxy # /data/app/redis/bin/redis-cli -h 192.168.10.93 -p 19000

192.168.10.93:19000> 
192.168.10.93:19000> set m 1
OK
192.168.10.93:19000> get m
"1"

####遇到的坑

• dashboard死掉用起不來 問了度娘，要在zookeeper裏執行rmr /zk/codis 實際上會把全部信息幹掉，只須要rmr /zk/codis/db_test/dashboard

• 加錯codis group id 不當心把group2的機器加到12裏去了，刪除居然報錯，我去 還得在zookeeper裏刪除rmr /zk/codis/db_test/servers/group_3

• 不當心點錯，把slot0的數據遷移到group0上，實際上不可能有group0，理所固然報錯了，後面的任務就一直pending 解決方法就是去zookeeper裏找到對應的task刪除,後面的任務就能運行了

[zk: localhost:2181(CONNECTED) 2] get /zk/codis/db_test/migrate_tasks/0000000000
{"slot_id":0,"new_group":0,"delay":0,"create_at":"1460043974","percent":0,"status":"error"}
[zk: localhost:2181(CONNECTED) 3] rm /zk/codis/db_test/migrate_tasks/0000000000