Keepalived實現高可用

原文發表於cu2017-03-27 php

參考文檔:html

  1. keepalived user guide:http://www.keepalived.org/pdf/UserGuide.pdf
  2. 安裝文檔:源碼解壓包中的INSTALL文檔

 本文涉及keepalived的安裝,簡單配置,爲haproxy作高可用。前端

一.環境準備

1. 操做系統

CentOS-7-x86_64-Everything-1511node

2. Keepalived版本

截至2017-03-22,keepalived版本是1.3.5: linux

http://www.keepalived.org/software/keepalived-1.3.5.tar.gznginx

3. 拓撲圖

  1. 採用VMware ESXi虛擬出的2臺服務器node1/2,前端訪問地址10.11.4.151/152,後端地址192.168.4.151/2;
  2. Web1服務器爲採用docker技術生成的1臺服務器,已安裝並啓動nginx與php服務,ip地址192.168.4.171;
  3. Web2/3同Web1服務器,ip地址192.168.4.172/173;
  4. 計劃在node1/2兩臺服務器上部署keepalive&haproxy,利用keepalived虛擬出vip:10.11.4.150作高可用;
  5. Haproxy相關配置請參考:http://www.cnblogs.com/netonline/p/7593762.html,調整後將靜態網頁指向web1/2服務器的index.html,將動態網頁指向web1/2服務器的index.php,其餘指向web3服務器;
  6. 以web1爲例,設置測試頁面,以方便後續查看驗證結果。

二.Keepalived安裝配置

如下流程均在node1節點完成,node2節點請參考node1作適當修改。 web

1. 依賴軟件

#升級或者安裝相關軟件,不是必需都安裝一次;
#通常libnl3-devel ipset-devel iptables-devel libnfnetlink-devel popt popt-static popt-devel等並無預安裝到系統中;
#net-snmp-devel是須要開啓相關功能才須要
[root@elk-node1 ~]# yum install openssl-devel libnl3-devel ipset-devel iptables-devel libnfnetlink-devel popt popt-static popt-devel gcc kernel-headers kernel-devel net-snmp-devel -y 

2. 下載

[root@elk-node1 ~]# cd /usr/local/src/
[root@elk-node1 src]#wget http://www.keepalived.org/software/keepalived-1.3.5.tar.gz 

3. 編譯安裝

#編譯前可經過」./configure --help」查看相關編譯參數;
#此編譯未帶「--with-kernel-dir」參數,通常認爲採用」--with-kernel-dir=/usr/src/kernels/(version)」指定到內核效果更好,這裏環境比較簡單,實際使用後並無明顯的問題;
#這裏未指定是由於centos7在編譯使用參數以後找不到」linux/netlink.h」頭文件,即便在相應目錄下能找到相應頭文件,搜了一下也沒有找到對應的解決方案
[root@elk-node1 src]# tar -zxvf keepalived-1.3.5.tar.gz
[root@elk-node1 src]# cd keepalived-1.3.5
[root@elk-node1 keepalived-1.3.5]# ./configure --prefix=/usr/local/keepalived 
[root@elk-node1 keepalived-1.3.5]# make
[root@elk-node1 keepalived-1.3.5]# make install 

4. 配置開機啓動

1)啓動相關命令

#軟連接
[root@elk-node1 ~]# cd /usr/local/keepalived/
[root@elk-node1 keepalived]# ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
[root@elk-node1 keepalived]# ln -s /usr/local/keepalived/sbin/keepalived /usr/sbin/

2)配置文件

#軟連接
[root@elk-node1 keepalived]# mkdir -p /etc/keepalived
[root@elk-node1 keepalived]# ln -s /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/ 

3)開機啓動

#centos7編譯安裝目錄下,默認沒有」/etc/rc.d/init.d/keepalived」文件,即自啓腳本,須要手工配置,前提是將啓動相關命令,配置文件等按腳本定義的目錄放置;
#啓動時,可能須要運行:systemctl daemon-reload再重啓keepalived
[root@elk-node1 keepalived]# touch /etc/rc.d/init.d/keepalived
[root@elk-node1 keepalived]# chmod +x /etc/rc.d/init.d/keepalived 
[root@elk-node1 keepalived]# vim /etc/rc.d/init.d/keepalived
#!/bin/sh
#
# keepalived   High Availability monitor built upon LVS and VRRP
#
# chkconfig:   - 86 14
# description: Robust keepalive facility to the Linux Virtual Server project \
#              with multilayer TCP/IP stack checks.

### BEGIN INIT INFO
# Provides: keepalived
# Required-Start: $local_fs $network $named $syslog
# Required-Stop: $local_fs $network $named $syslog
# Should-Start: smtpdaemon httpd
# Should-Stop: smtpdaemon httpd
# Default-Start: 
# Default-Stop: 0 1 2 3 4 5 6
# Short-Description: High Availability monitor built upon LVS and VRRP
# Description:       Robust keepalive facility to the Linux Virtual Server
#                    project with multilayer TCP/IP stack checks.
### END INIT INFO

# Source function library.
. /etc/rc.d/init.d/functions

exec="/usr/sbin/keepalived"
prog="keepalived"
config="/etc/keepalived/keepalived.conf"

[ -e /etc/sysconfig/$prog ] && . /etc/sysconfig/$prog

lockfile=/var/lock/subsys/keepalived

start() {
    [ -x $exec ] || exit 5
    [ -e $config ] || exit 6
    echo -n $"Starting $prog: "
    daemon $exec $KEEPALIVED_OPTIONS
    retval=$?
    echo
    [ $retval -eq 0 ] && touch $lockfile
    return $retval
}

stop() {
    echo -n $"Stopping $prog: "
    killproc $prog
    retval=$?
    echo
    [ $retval -eq 0 ] && rm -f $lockfile
    return $retval
}

restart() {
    stop
    start
}

reload() {
    echo -n $"Reloading $prog: "
    killproc $prog -1
    retval=$?
    echo
    return $retval
}

force_reload() {
    restart
}

rh_status() {
    status $prog
}

rh_status_q() {
    rh_status &>/dev/null
}


case "$1" in
    start)
        rh_status_q && exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart)
        $1
        ;;
    reload)
        rh_status_q || exit 7
        $1
        ;;
    force-reload)
        force_reload
        ;;
    status)
        rh_status
        ;;
    condrestart|try-restart)
        rh_status_q || exit 0
        restart
        ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}"
        exit 2
esac
exit $?

#設置開機啓動
[root@elk-node1 keepalived]# chkconfig --add keepalived
[root@elk-node1 keepalived]# chkconfig --level 35 keepalived on
[root@elk-node1 keepalived]# vim /usr/lib/systemd/system/keepalived.service
#修改PIDFile,以下:
PIDFile=/var/run/keepalived.pid 

5. Keepalived配置文件

[root@elk-node1 ~]# vim /usr/local/keepalived/etc/keepalived/keepalived.conf
#=====================================================
# keepalived.conf 配置
#------------------------------------------------------------
# 一、Keepalived 配置文件以block形式組織,每一個塊內容都包含在{}
# 二、「#」,「!」開頭行爲註釋
# 三、keepalived 配置爲三類:
#    (1)全局配置:對整個keepalived都生效的配置
#    (2)VRRPD 配置:核心配置,主要實現keepalived高可用功能
#    (3)LVS配置
#=====================================================

! Configuration File for keepalived

########################
#  全局配置
########################
# global_defs 全局配置標識;
global_defs {

# notification_email用於設置報警郵件地址; 能夠設置多個,每行一個; 設置郵件報警需開啓本機Sendmail服務
   notification_email {
     root@localhost.local
   }

# 設置郵件發送地址, smtp server地址, 鏈接smtp sever超時時間
   notification_email_from root@localhost.local
   smtp_server 10.11.4.151
   smtp_connect_timeout 30

# 表示運行keepalived服務器標識,郵件發送時在主題中顯示的信息
   router_id Haproxy_DEVEL
}

######################
#  服務檢測配置
######################
# 服務探測,chk_haproxy爲服務名返回0說明服務是正常的
    vrrp_script chk_haproxy {
        script "/usr/local/keepalived/etc/chk_haproxy.sh"

#每隔1秒探測一次
        interval 1

#haproxy在線,權重加2
#        weight 2
}

######################
#  VRRPD配置
######################
# VRRPD配置標識,VI_1是實例名稱
vrrp_instance VI_1 {

# 指定Keepalvied角色,MASTER(必須大寫)表示此主機爲主服務器,BACKUP則是表示爲備用服務器;
# 這裏由於配置非搶佔模式,nopreempt只做用於BACKUP,將2臺主機均配置爲BACKUP
    state BACKUP

# 指定HA監測網絡的接口
    interface eth0

# 虛擬路由標識,標識爲數字,1-255可選;
# 同1個VRRP實例使用惟一的標識,MASTER_ID = BACKUP_ID
    virtual_router_id 51

# 定義節點優先級,數字越大表示節點的優先級越高;
# 同1個VRRP_instance下,MASTE_PRIORITY > BACKUP_PRIORITY
    priority 100

# MASTER與BACKUP主機之間同步檢查的時間間隔,單位爲秒
    advert_int 1

# 從實際應用角度,建議配置非搶佔模式,防止網絡頻繁切換震盪
    nopreempt


# 設定節點間通訊驗證類型與密碼,驗證類型主要有PASS和AH兩種;
# 同1個vrrp_instance,MASTER驗證密碼和BACKUP保持一致
    authentication {
        auth_type PASS
        auth_pass 987654
    }

# 設置虛擬IP地址(VIP),又叫作漂移IP地址;
# 可設置多個,1行1個;
# keepalived經過「ip address add」命令的形式將VIP添加到系統
    virtual_ipaddress {
        10.11.4.150
    }

# 腳本追蹤,對應服務檢測
    track_script {
        chk_haproxy
    }
}

##############################################
# LVS配置,這裏keepalived只作高可用,並不作lvs
##############################################
# virtual_server LVS配置標識
# 格式: virtual_server VIP port [IP 和 port 之間空格隔開] 
# virtual_server 10.11.4.150 443 {
# 設置健康檢查時間間隔,單位爲秒
#    delay_loop 6

# 設置負載調度算法,經常使用調度算法是: rr、wlc,另有:lc、lblc、sh、dh等
#    lb_algo rr

# 設置LVS實現負載均衡的機制,有NAT、TUN和DR三種模式可選
#    lb_kind NAT

# 會話保持時間,其對動態網頁很是有用,爲集羣系統中的seesion共享提供了一個很好的解決方案;
# 用戶的請求會一直分發到某個服務節點,直至超過這個會話的保持時間(指最大無響應超時時間),
# 即用戶操做動態頁面若是在50s沒有執行任何操做則被分發到另外的節點
#    persistence_timeout 50

# 轉發協議類型
#    protocol TCP

# 設置real server段開始的標識 [ IP爲真實IP地址]
# 格式:real_server realIP port [IP 和 port 之間空格隔開]
#    real_server 192.168.201.100 443 {
# real server節點的權值,權值大小用數字表示,數字越大,權值越高
#        weight 1
# 健康檢查 SSL_GET
#        SSL_GET {
# 指定SSL檢查的URL信息,能夠指定多個
#            url { 
# 詳細的URL路徑
#              path /index.html
# SSL檢查後的摘要信息,能夠經過genhash命令工具獲取,命令以下:
# [root@elk-node1 bin]# /usr/local/keepalived/bin/genhash -s 192.168.4.171 -p 80 -u /index.html
#              digest ff20ad2481f97b1754ef3e12ecd3a9cc
#            }
#            url {
#              path /mrtg/
#              digest 9b3a0c85a887a256d6939da88aabd8cd
#            }
# 無響應超時時間,單位爲秒
#            connect_timeout 3
# 重試次數
#            nb_get_retry 3
# 重試間隔
#            delay_before_retry 3
#        }
#    }
#} 

6. Keepalived檢測腳本

#檢測haproxy服務是否正常運行,若是沒有則嘗試拉起來,若是嘗試失敗則重啓keepalived服務,切換keepalived的vip
[root@elk-node1 ~]# touch /usr/local/keepalived/etc/chk_haproxy.sh
[root@elk-node1 ~]# chmod 755 /usr/local/keepalived/etc/chk_haproxy.sh
[root@elk-node1 ~]# vim /usr/local/keepalived/etc/chk_haproxy.sh
#!/bin/bash
# check haproxy process, if there isn't any process, try to start the process once,
# check it again after 3s, if there isn't any process still, restart keepalived process, change state.
# 2017-03-22 v0.1 
if [ $(ps -C haproxy --no-header | wc -l) -eq 0 ]; then
     /etc/rc.d/init.d/haproxy start
     sleep 3

     if [ $(ps -C haproxy --no-header | wc -l) -eq 0 ]; then
          /etc/rc.d/init.d/keepalived restart
     fi
fi

# another method to check haproxy process
#killall -0 haproxy 
#if [[ $? -ne 0 ]];then 
#  /etc/rc.d/init.d/keepalived restart 
#fi

三.驗證 

1. 啓動

[root@elk-node1 ~]# service keepalived start
[root@elk-node2 ~]# service keepalived start 

2. 查看日誌

1)Node1

[root@elk-node1 ~]# tailf /var/log/messages

  1. 以BACKUP模式啓動;
  2. 切換到MASTER模式;
  3. 得到vip 10.11.4.150,開始對外發送免費arp通告。 

2)Node2 

[root@elk-node2 ~]# tailf /var/log/messages

  1. 兩個相關子進程啓動;
  2. 啓動後進入BACKUP模式。 

3. VIP

#使用的是"ip address add"添加的vip到系統中,因"ifconfig"命令看不到效果
[root@elk-node1 ~]# ip address show eth0

Node1的網卡eth0已經得到vip 10.11.4.150。 算法

4. 故障切換

1)Haproxy故障拉起

[root@elk-node1 ~]# date ; service haproxy stop
[root@elk-node1 ~]# date ; service haproxy status

  1. 手工中止haproxy服務;
  2. 由於keepalived配置文件中定義了拉起haproxy服務的腳本,能夠看到1s的時間內,haproxy服務又開始運行了。

2)Node1日誌 

 

  1. 日誌顯示haproxy服務中止後再被拉起;
  2. Keepalived進入FAULT STATE,進而轉到BACKUP STATE;
  3. Node1的eth0網卡的vip被刪除。

3)Node2日誌 

  1. Node2轉到MASTER STATE;
  2. Node2得到vip 10.11.4.150,並開始對外發免費arp通告。

4)Node2 VIP

[root@elk-node2 ~]# ip address show eth0

Node2的網卡eth0已經得到vip 10.11.4.150。docker

相關文章
相關標籤/搜索