kubernetes上部署Rabbitmq高可用記錄

部署RabbitMq

RabbitMQ集羣的兩種模式
1）普通模式：默認的集羣模式,隊列消息只存在單個節點上
2）鏡像模式：隊列爲鏡像隊列，隊列消息存在每一個節點上

鏡像模式結構

官方解釋：鏡像模式是經過policy策略方式實現的

配置同步：
    1.Ha mode 同步模式,如下可選：
        1)all 全部的節點都將被同步
        2)exactly 指定個數的節點被同步
        3)nodes 指定的名稱的節點被同步
    2.Ha params 上一項中的參數
    3.Ha async mode 同步方式，自動或手動

主要部署步驟

使用statefulset啓動N個rabbitmq節點

mq-5c0f5af17b04e-0                  1/1       Running                 0          6m
mq-5c0f5af17b04e-1                  1/1       Running                 0          5m
mq-5c0f5af17b04e-2                  1/1       Running                 0          3m

每一個節點執行如下初始化腳本

if [ -z "$(grep rabbitmq /etc/resolv.conf)" ]; then
    sed "s/^search \([^ ]\+\)/search ${MY_POD_NAME%-*}.\1 \1/" /etc/resolv.conf > /etc/resolv.conf.new;
    cat /etc/resolv.conf.new > /etc/resolv.conf;
    rm /etc/resolv.conf.new;
fi;

#檢查集羣狀態
until rabbitmqctl node_health_check; do sleep 1; done;

# 加入集羣，設置同步策略（鏡像模式）
if [ -z "$(rabbitmqctl cluster_status | grep ${MY_POD_NAME%-*}-0)" ]; then
    rabbitmqctl stop_app;
    rabbitmqctl reset;
    rabbitmqctl join_cluster rabbit@${MY_POD_NAME%-*}-0;
    rabbitmqctl start_app;
    rabbitmqctl set_policy mirror-queues "^" '{"ha-mode":"all","ha-sync-mode":"automatic"}'
fi;

進入容器，查看集羣狀態

root@mq-5c0f5af17b04e-0:/# rabbitmqctl cluster_status
Cluster status of node rabbit@mq-5c0f5af17b04e-0 ...
[{nodes,[{disc,['rabbit@mq-5c0f5af17b04e-0','rabbit@mq-5c0f5af17b04e-1',
                'rabbit@mq-5c0f5af17b04e-2']}]},
 {running_nodes,['rabbit@mq-5c0f5af17b04e-2','rabbit@mq-5c0f5af17b04e-1',
                 'rabbit@mq-5c0f5af17b04e-0']},
 {cluster_name,<<"rabbit@mq-5c0f5af17b04e-0.mq-5c0f5af17b04e.itfarm3.svc.cluster.local">>},
 {partitions,[]},
 {alarms,[{'rabbit@mq-5c0f5af17b04e-2',[]},
          {'rabbit@mq-5c0f5af17b04e-1',[]},
          {'rabbit@mq-5c0f5af17b04e-0',[]}]}]

在web頁面中查看集羣狀態

查看隊列

測試高可用

鏡像同步

生產一條消息，查看狀態

//全部節點都存在消息
root@mq-5c0f5af17b04e-0:/#  rabbitmqctl list_queues
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
mirror-admin    1

root@mq-5c0f5af17b04e-1:/#  rabbitmqctl list_queues
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
mirror-admin    1

root@mq-5c0f5af17b04e-2:/# rabbitmqctl list_queues
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
mirror-admin    1

故障模擬

//刪除多個節點，查看狀態
$ kubectl.exe delete po mq-5c0f5af17b04e-0 mq-5c0f5af17b04e-1
pod "mq-5c0f5af17b04e-0" deleted
pod "mq-5c0f5af17b04e-1" deleted

//重啓後，故障節點恢復隊列消息
root@mq-5c0f5af17b04e-0:/#  rabbitmqctl list_queues
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
mirror-admin    1

root@mq-5c0f5af17b04e-1:/#  rabbitmqctl list_queues
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
mirror-admin    1