Keepalived+Nginx高可用集羣搭建筆記

前言

Keepalived是以VRRP（Virtual Router Redundancy Protocol，虛擬路由冗餘協議）協議爲實現基礎的，這個協議能夠認爲是實現了路由器高可用的協議，將多臺提供相同功能的路由器組成一個路由器組。

原理：在整個Keepalived集羣中會有一個MASTER和多個BACKUP，master節點上有一個對外提供服務的Virtual IP(VIP)，而且MASTER會發組播的心跳信息，當BACKUP收不到VRRP包時就認爲MASTER宕掉了，這時就須要根據VRRP優先級來選舉一個BACKUP做爲MASTER，當MASTER恢復時，BACKUP又會釋放在MASTER故障時自身接管的IP資源和服務，恢復到原來的備用角色，這樣就能夠保證路由器的高可用。

環境說明

• 操做系統：CentOS 7 (Minimal Install)

# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
複製代碼

• 演示環境

VIP	IP	主機名
10.10.0.10	10.10.0.11	master
10.10.0.10	10.10.0.12	backup

部署

更換服務器源倉庫

# mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
# curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
# yum makecache
# yum -y update
複製代碼

安裝Keepalived

yum安裝

Keepalived可使用yum直接安裝，在master服務器和backup服務器執行：

# yum -y install keepalived
複製代碼

源碼編譯安裝

參考文檔：Keepalived官方文檔

安裝依賴庫

# yum -y install openssl-devel libnl3-devel ipset-devel iptables-devel file-devel net-snmp-devel glib2-devel json-c-devel pcre2-devel libnftnl-devel libmnl-devel
複製代碼

下載Keepalived

# wget https://github.com/acassen/keepalived/archive/v2.0.18.tar.gz
複製代碼

解壓Keepalived

# tar -zxvf v2.0.18.tar.gz
# cd keepalived-2.0.18
複製代碼

開始安裝

# ./build_setup
./build_setup:行3: aclocal: 未找到命令
./build_setup:行4: autoheader: 未找到命令
./build_setup:行5: automake: 未找到命令
./build_setup:行6: autoreconf: 未找到命令
複製代碼

若是出現如上報錯，安裝autotools系列工具

# yum -y install aclocal autoheader automake autoreconf
複製代碼

繼續

# ./configure
# make && make install
複製代碼

最後複製相關配置文件到系統默認路徑

# mkdir /etc/keepalived
# cp ./keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
# cp ./keepalived/etc/init.d/keepalived /etc/init.d/
# cp ./keepalived/etc/sysconfig/keepalived /etc/sysconfig/
複製代碼

修改/usr/lib/systemd/system/keepalived.service中PIDFile的值爲/var/run/keepalived.pid。

配置Keepalived

Keepalived提供了兩種模式

• 搶佔式：MASTER與BACKUP節點上state配置不一樣，當MASTER節點宕掉後由BACKUP節點接手MASTER節點的VIP與服務，在MASTER節點恢復後從新由MASTER節點來接手VIP與服務，BACKUP節點繼續回到備用狀態。
• 非搶佔式：MASTER與BACKUP節點上state配置都爲BACKUP，且在vrrp_instance塊下兩個節點都增長nopreempt，表示不爭搶VIP。兩個節點啓動後默認都爲BACKUP狀態，雙方在發送組播信息後，會根據優先級來選舉一個MASTER出來，因爲二者都配置了nopreempt，因此MASTER從故障中恢復後不會搶佔VIP，這樣會避免VIP切換可能形成的服務延遲。

MASTER節點

首先，咱們先確認下網卡及IP

# ip addr show | grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
    inet 10.10.0.11/8 brd 10.255.255.255 scope global noprefixroute ens192
    inet6 fd08:815:48b2::e91/128 scope global noprefixroute
    inet6 fd08:815:48b2:0:d419:f3f5:85de:b72/64 scope global noprefixroute
    inet6 fe80::49a2:321d:8cf6:651a/64 scope link noprefixroute
複製代碼

能夠看到本次使用的是ens192這塊網卡，IP爲：10.10.0.11，而後咱們編輯keepalived配置文件

# vim /etc/keepalived/keepalived.conf
複製代碼

配置以下：

! Configuration File for keepalived

global_defs {
   # email 收件人
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   # email 發件人
   notification_email_from Alexandre.Cassen@firewall.loc
   # email SMTP服務器地址
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   # 標識本節點的ID,一般爲hostname
   router_id akiya01
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}
# vrrp實例，相同實例的備節點名字要相同
vrrp_instance VI_1 {
    # 指定keepalived的角色，「MASTER」表示此主機是主服務器，「BACKUP」表示此主機是備用服務器
    state MASTER
    # 指定網卡接口，這裏改成咱們當前使用的網卡「ens192」
    interface ens192
    # 虛擬路由標識，這個標識是一個數字，同一個vrrp實例使用惟一的標識
    # 即同一vrrp_instance下，MASTER和BACKUP必須是一致的
    virtual_router_id 51
     # 定義優先級；數字越大，優先級越高（0-255）
    # 在同一個vrrp_instance下，「MASTER」的優先級必須大於「BACKUP」的優先級
    priority 100
    # 設定MASTER與BACKUP負載均衡器之間同步檢查的時間間隔，單位是秒
    advert_int 1
    # 設置驗證類型和密碼
    authentication {
        # 設置驗證類型，主要有PASS和AH兩種
        auth_type PASS
        # 設置驗證密碼，在同一個vrrp_instance下，MASTER與BACKUP必須使用相同的密碼才能正常通訊
        auth_pass akiya
    }
    # 有故障時是否激活郵件通知
    #smtp_alert
    # 禁止搶佔服務
    # 默認狀況，當MASTER服務掛掉以後，BACKUP自動升級爲MASTER並接替它的任務
    # 當MASTER服務恢復後，升級爲MASTER的BACKUP服務又自動降爲BACKUP，把工做權交給原MASTER
    # 當配置了nopreempt，MASTER從掛掉到恢復，再也不將服務搶佔過來。
    #nopreempt
    # 虛擬IP，兩個節點設置必須同樣。能夠設置多個，一行寫一個
    virtual_ipaddress {
        # 虛擬IP爲10.10.0.10/8;綁定接口爲ens192;別名ha:net，主備相同
        10.10.0.10/8 dev ens192 label ha:net
    }
}
複製代碼

BACKUP節點

BACKUP配置基本與Master一致，僅有部分地方變更

• state角色爲BACKUP
• interface爲網卡的ID，須要根據機器實際狀況確認填寫
• virtual_route_id要和MASTER一致，默認爲51
• priority要比MASTER小

修改BACKUP節點Keepalived配置，部署配置以下：

! Configuration File for keepalived
...
rrp_instance VI_1 {
    # 指定Keepalived的角色，BACKUP表示此主機是備用節點
    state BACKUP
    # 確認網卡的ID
    interface ens192
    # 即同一vrrp_instance下，「MASTER」和「BACKUP」必須是一致的
    virtual_router_id 51
    # 優先級，比MASTER小
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    # 虛擬IP，兩個節點設置必須同樣。能夠設置多個，一行寫一個
    virtual_ipaddress {
        # 虛擬IP爲10.10.0.10/8;綁定接口爲ens192;別名ha:net，主備相同
        10.10.0.10/8 dev ens192 label ha:net
    }
}
複製代碼

啓動服務

配置完MASTER與BACKUP節點後，咱們就能夠啓動並測試服務了

添加防火牆規則

由於vrrp使用224.0.0.18這個組播地址

# firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --in-interface ens192 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
# firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 --out-interface ens192 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
# firewall-cmd --reload
複製代碼

查看規則

# firewall-cmd --direct --get-rules ipv4 filter INPUT
0 --in-interface ens192 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
# firewall-cmd --direct --get-rules ipv4 filter OUTPUT
0 --out-interface ens192 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
複製代碼

啓動Keepalived

啓動Keepalived並添加到開機自啓

# systemctl start keepalived
# systemctl enable keepalived
複製代碼

而後咱們再次查看MASTER節點IP能夠發現新增了一個

# ip addr show | grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
    inet 10.10.0.11/8 brd 10.255.255.255 scope global noprefixroute ens192
    inet 10.10.0.10/32 scope global ha:net
    inet6 fd08:815:48b2::e91/128 scope global noprefixroute
    inet6 fd08:815:48b2:0:d419:f3f5:85de:b72/64 scope global noprefixroute
    inet6 fe80::49a2:321d:8cf6:651a/64 scope link noprefixroute
複製代碼

一樣在BACKUP節點上查看IP結果爲

# ip addr show | grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
    inet 10.10.0.12/8 brd 10.255.255.255 scope global noprefixroute ens192
    inet6 fd08:815:48b2::1ca/128 scope global noprefixroute
    inet6 fd08:815:48b2:0:b840:33aa:f6de:253b/64 scope global noprefixroute
    inet6 fe80::a96d:fe89:d95:3dfd/64 scope link noprefixroute
複製代碼

測試Keepalived

安裝tcpdump工具

# yum -y install tcpdump
複製代碼

在MASTER節點上執行以下命令

# tcpdump -i ens192 vrrp -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
10:10:24.193943 IP 10.10.0.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
10:10:25.194972 IP 10.10.0.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
10:10:26.196009 IP 10.10.0.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
10:10:27.197038 IP 10.10.0.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
...
複製代碼

若是關閉MASTER上的Keepalived則無包可抓，而且VIP會對應的漂移到BACKUP上去。

配置日誌

Keepalived默認日誌輸出到系統日誌/var/log/messages中，由於系統日誌不少，在查詢時相對麻煩。

咱們能夠將Keepalived日誌單獨拿出來，這裏須要修改日誌輸出路徑。

1. 修改Keepalived配置

# vim /etc/sysconfig/keepalived
複製代碼

更改以下：

# Options for keepalived. See `keepalived --help' output and keepalived(8) and
# keepalived.conf(5) man pages for a list of all options. Here are the most
# common ones :
#
# --vrrp               -P    Only run with VRRP subsystem.
# --check              -C    Only run with Health-checker subsystem.
# --dont-release-vrrp  -V    Dont remove VRRP VIPs & VROUTEs on daemon stop.
# --dont-release-ipvs  -I    Dont remove IPVS topology on daemon stop.
# --dump-conf          -d    Dump the configuration data.
# --log-detail         -D    Detailed log messages.
# --log-facility       -S    0-7 Set local syslog facility (default=LOG_DAEMON)
#

KEEPALIVED_OPTIONS="-D"
複製代碼

修改KEEPALIVED_OPTIONS="-D"爲KEEPALIVED_OPTIONS="-D -d -S 0"，-S指定syslog的facility

2. 修改/etc/rsyslog.conf，在末尾添加

...
local0.*            /var/log/keepalived.log
複製代碼

3. 重啓日誌記錄服務

# systemctl restart rsyslog
複製代碼

4. 重啓Keepalived

# systemctl restart keepalived
複製代碼

5. 查看日誌

# ls -lh /var/log/keepalived.log
-rw-------. 1 root root 14K 9月  30 13:22 /var/log/keepalived.log
# head -n 10 /var/log/keepalived.log
Sep 30 13:22:52 master Keepalived[30707]: Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Sep 30 13:22:52 master Keepalived[30707]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 30 13:22:52 master Keepalived[30708]: Starting Healthcheck child process, pid=30709
Sep 30 13:22:52 master Keepalived[30708]: Starting VRRP child process, pid=30710
Sep 30 13:22:52 master Keepalived_healthcheckers[30709]: Initializing ipvs
Sep 30 13:22:52 master Keepalived_healthcheckers[30709]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 30 13:22:52 master Keepalived_healthcheckers[30709]: ------< Global definitions >------
Sep 30 13:22:52 master Keepalived_healthcheckers[30709]: Router ID = ha01
Sep 30 13:22:52 master Keepalived_healthcheckers[30709]: Smtp server = 192.168.200.1
Sep 30 13:22:52 master Keepalived_healthcheckers[30709]: Smtp server port = 25
複製代碼

Keepalived+Nginx

在實際狀況中，業務中止而Keepalived服務還在工做的狀況會致使VIP沒法找到對應的服務，這時就須要寫守護進程腳本，下面以Nginx爲例。

Nginx安裝

• 增長對應的Nginx源

# rpm -ivh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
複製代碼

• yum安裝Nginx

# yum -y install nginx
複製代碼

• NGINX啓動、中止、重啓、開機自啓

# systemctl start nginx # 啓動Nginx服務
# systemctl stop nginx # 中止Nginx服務
# systemctl restart nginx # 重啓Nginx服務
# systemctl enable nginx # 開機自啓Nginx服務
# nginx -t # 檢查配置文件正確性
# nginx -s reload # 平滑重載配置
複製代碼

• 檢查啓動是否成功啓動

# curl -i localhost
複製代碼

建立Nginx服務檢測腳本

分別在主備服務器的/etc/keepalived目錄下建立nginx_check.sh腳本，腳本以下：

#!/bin/bash
# author:akiya
A=$(ps -C nginx --no-header | wc -l)
if [ $A -eq 0 ]; then
    systemctl start nginx
    sleep 2
    if [ $(ps -C nginx --no-header | wc -l) -eq 0 ]; then
        systemctl stop keepalived
    fi
fi
複製代碼

爲腳本添加可執行權限

# chmod +x /etc/keepalived/nginx_check.sh
複製代碼

此腳本用於Keepalived定時檢測Nginx的服務狀態，若是Nginx中止，會嘗試從新啓動Nginx，若是啓動失敗，會將Keepalived服務中止，使IP漂移到備用節點上。

修改Keepalived配置

在/etc/keepalived/keepalived.conf中增長檢測腳本配置

global_defs {
   ...
   }
...
# keepalived會定時執行腳本並對腳本執行的結果進行分析,動態調整vrrp_instance的優先級
# 若是腳本執行結果爲0,而且weight配置的值大於0,則優先級相應的增長。
# 若是腳本執行結果非0,而且weight配置的值小於 0,則優先級相應的減小。
# 其餘狀況,維持本來配置的優先級,即配置文件中priority對應的值。
vrrp_script chk_nginx {
       script "/etc/keepalived/nginx_check.sh"
       interval 2  #每2秒檢測一次nginx的運行狀態
       weight -20  #失敗一次，將本身的優先級-20
}
vrrp_instance VI_1 {
    ...
    virtual_ipaddress {
        10.10.0.10/8 dev ens192 label ha:net
    }
    track_script {
        # Nginx存活狀態監測腳本
        chk_nginx
    }
}
複製代碼

問題處理

Unable to access script

我在使用yum安裝的版本爲1.3.5，在配置文件中編寫vrrp_script塊後，啓動服務遇到一個問題Unable to access script，經查資料發現Git Issues中有提到這個問題，新版本目前已解決。

部分報錯相關日誌以下：

Sep 30 14:25:42 master Keepalived_vrrp[30930]:     chk_nginx no match, ignoring...
Sep 30 14:26:04 master Keepalived_vrrp[30944]:     nginx_check no match, ignoring...
Sep 30 14:44:18 master Keepalived_vrrp[30980]: Unable to access script `/etc/keepalived/nginx_check.sh`
Sep 30 14:44:18 master Keepalived_vrrp[30980]: Disabling track script chk_nginx since not found
複製代碼

若是使用yum安裝能夠在安裝前查看下對應的包信息

# yum info keepalived
複製代碼

default user...

使用編譯安裝後（安裝版本2.0.18），添加Nginx檢測腳本並啓動Keepalived服務後，日誌顯示 default user 'keepalived_script' for script execution does not exist - please create.

解決方法：在配置文件中添加運行檢測腳本的用戶或組便可

! Configuration File for keepalived

global_defs {
...
   script_user root
   enable_script_security
}
...
複製代碼