nginx和keeplive實現負載均衡高可用

時間 2019-11-18

原文原文鏈接

1、 Keeplive服務介紹

Keeplive期初是專門爲LVS設計的，專門用來監控LVS集羣系統中各個服務節點的狀態，後來又加入VRRP的功能，所以除了配合LVS服務之外，也能夠做爲其餘服務（nginx，haroxy）的高可用軟件，VRRP是Virtual Router Redundancy Protocol（虛擬路由冗餘協議）的縮寫，VRRP出現的目的就是爲了解決靜態路由出現的單點故障問題，它可以保證網絡的不間斷，穩定的運行。因此keepalive一方面具備LVS cluster nodes healthchecks功能，另外一方面也具備LVS directors failoverhtml

1.1 Keepalived的用途

Keepalive服務的兩大用途：healthcheck和failovernode

ha failover功能：實現LB Master主機和Backup主機之間故障轉移和自動切換nginx

這是針對有兩個負載均衡器Direator同時工做而採起的故障轉移措施，當主負載均衡器失效或者出現故障時，備份的負載均衡器（BACKUP）將自動接管主負載均衡器的全部工做（vip資源以及相關服務）：一旦主負載均衡器故障恢復，MASTER又會接管回它原來的工做，二備份複雜均衡器（BACKUP）會釋放master是小事它接管的工做，此時二者將恢復到最初各自的角色web

1.2 LVS cluster nodes healthchecks功能

在keeplive.conf配置記憶能夠實現LVS的功能vim

keeplive能夠對LVS下面的集羣節點進行健康檢查bash

rs healthcheck功能：負載均衡按期檢查RS的可用性決定是否給其分發請求服務器

當虛擬的服務器中的某一個甚至是幾個真實的服務器同時出現故障沒法提供服務時，負載均衡器會自動將失效的RS服務器從轉發隊列中清除出去，從而保證用戶的訪問不收影響；當故障的RS服務器被修復後，系統又自動的將他們加入轉發隊列，分發請求提供正常服務。網絡

工做原理app

1.3 keepalive故障切換轉換原理

Keepalived高可用對之間是經過 VRRP進行通訊的， VRRP是遑過競選機制來肯定主備的，主的優先級高於備，所以，工做時主會優先得到全部的資源，備節點處於等待狀態，當主掛了的時候，備節點就會接管主節點的資源，而後頂替主節點對外提供服務。負載均衡

在 Keepalived服務對之間，只有做爲主的服務器會一直髮送 VRRP廣播包,告訴備它還活着，此時備不會槍佔主，當主不可用時，即備監聽不到主發送的廣播包時，就會啓動相關服務接管資源，保證業務的連續性.接管速度最快能夠小於1秒。

1.4 VRRP協議的簡單介紹

1) VRRP,全稱 Virtual Router Redundancy Protocol,中文名爲虛擬路由冗餘協議，VRRP的出現是爲了解決靜態路由的單點故障。

2) VRRP是經過一種竟選協議機制來將路由任務交給某臺 VRRP路由器的。

3) VRRP用 IP多播的方式（默認多播地址（224.0_0.18))實現高可用對之間通訊。

4) 工做時主節點發包，備節點接包，當備節點接收不到主節點發的數據包的時候，就啓動接管程序接管主節點的開源。備節點能夠有多個，經過優先級競選，但通常 Keepalived系統運維工做中都是一對。

5) VRRP使用了加密協議加密數據，但Keepalived官方目前仍是推薦用明文的方式配置認證類型和密碼

2、配置Keepalived實現高可用

2.1 安裝keepalive

[root@lb01 ~]# cd /usr/local/src/

[root@lb01 src]# wget https://www.keepalived.org/software/keepalived-2.0.15.tar.gz

[root@lb01 src]# tar -xf keepalived-2.0.15.tar.gz

[root@lb01 src]# cd keepalived-2.0.15

[root@lb01 keepalived-2.0.15]# ./configure

Linker flags             :  -pie
Extra Lib                :  -lm -lcrypto  -lssl 
Use IPVS Framework       : Yes
IPVS use libnl           : No
IPVS syncd attributes    : No
IPVS 64 bit stats        : No
HTTP_GET regex support   : No
fwmark socket support    : Yes
Use VRRP Framework       : Yes
Use VRRP VMAC            : Yes
Use VRRP authentication  : Yes
With ip rules/routes     : Yes
With track_process       : Yes
With linkbeat            : Yes
Use BFD Framework        : No
SNMP vrrp support        : No
SNMP checker support     : No
SNMP RFCv2 support       : No
SNMP RFCv3 support       : No
DBUS support             : No
SHA1 support             : No
Use JSON output          : No
libnl version            : None
Use IPv4 devconf         : No
Use iptables             : Yes
Use libiptc              : No
Use libipset             : No
Use nftables             : No
init type                : systemd
Strict config checks     : No
Build genhash            : Yes
Build documentation      : No

[root@lb01 keepalived-2.0.15]# make

[root@lb01 keepalived-2.0.15]# make install

lb02相同操做

[root@lb02 ~]# cd /usr/local/src/
[root@lb02 src]# wget https://www.keepalived.org/software/keepalived-2.0.15.tar.gz
[root@lb02 src]# tar -xf keepalived-2.0.15.tar.gz
[root@lb02 src]# cd keepalived-2.0.15
[root@lb02 keepalived-2.0.15]# ./configure 
[root@lb02 keepalived-2.0.15]# make
[root@lb02 keepalived-2.0.15]# make install

[root@lb01 keepalived-2.0.15]# vim /usr/lib/systemd/system/keepalived.service

[Unit]
Description=LVS and VRRP High Availability Monitor
After=syslog.target network-online.target

[Service]
Type=forking
PIDFile=/var/run/keepalived.pid
KillMode=process
EnvironmentFile=-/etc/sysconfig/keepalived
ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

2.2 配置文件

[root@lb01 keepalived-2.0.15]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived      #！註釋

global_defs {                            #全局變量
   notification_email {
     283365585@qq.com                    #收件人
   }                                     #郵件發件人
   notification_email_from Alexandre.Cassen@firewall.loc   
   smtp_server 192.168.200.1             #郵件服務器地址
   smtp_connect_timeout 30               #超時時間
   router_id LVS_01
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}
    
vrrp_instance VI_1 {               #keepalive或者vrrp的一個實例
    state MASTER                   #狀態
    interface ens33                #通訊端口
    virtual_router_id 51           #實例ID
    priority 150                   #優先級
    advert_int 1                   #心跳的間隔
    authentication {               #服務器之間經過密碼驗證
        auth_type PASS
        auth_pass 1111
    }   
    virtual_ipaddress {
        10.0.0.131                 #VIP
    }   
}

2.3 啓動看效果

[root@lb01 keepalived]# systemctl start keepalived

[root@lb01 keepalived]# ps -ef|grep keep

[root@lb01 keepalived]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:12:2e:59 brd ff:ff:ff:ff:ff:ff
    inet 172.25.254.131/24 brd 172.25.254.255 scope global dynamic ens33
       valid_lft 1085sec preferred_lft 1085sec
   inet 10.0.0.131/24 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::8068:96e2:b57b:be1d/64 scope link 
       valid_lft forever preferred_lft forever
3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:12:2e:63 brd ff:ff:ff:ff:ff:ff

2.4 keepalive的VIP實現形式

[root@lb01 keepalived]# ifconfig ens33:0 10.0.0.18 up

[root@lb01 keepalived]# ip addr add 10.0.0.19 dev ens33

[root@lb01 keepalived]# ip addr

inet 172.25.254.131/24 brd 172.25.254.255 scope global dynamic ens33
       valid_lft 1583sec preferred_lft 1583sec
    inet 10.0.0.131/24 scope global ens33
       valid_lft forever preferred_lft forever
    inet 10.0.0.18/8 brd 10.255.255.255 scope global ens33:0
       valid_lft forever preferred_lft forever
    inet 10.0.0.19/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::8068:96e2:b57b:be1d/64 scope link 
       valid_lft forever preferred_lft forever

[root@lb01 keepalived]# ip addr del 10.0.0.19 dev ens33

[root@lb01 keepalived]# ifconfig ens33:0 10.0.0.18 down

[root@lb01 keepalived]# scp /etc/keepalived/keepalived.conf 172.25.254.133:/etc/keepalived/

[root@lb02 ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   notification_email {
        283365585@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_02
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
    10.0.0.131/24
    }
}

[root@lb02 ~]# systemctl start keepalived

[root@lb02 ~]# ps -ef |grep keep

配置成功

2.5 檢測keepalibve效果

關閉MASTER的keepalive服務

[root@lb01 keepalived]# systemctl stop keepalived

[root@lb01 keepalived]# ip addr|grep 10.0.0.131

查看BACKUP端，是否有10.0.0.131

[root@lb02 ~]# ip addr|grep 10.0.0.131

[root@lb01 keepalived]# systemctl start keepalived

成功

3、結合nginx實現高可用

3.1 配置

[root@lb01 keepalived]# cd /usr/local/nginx/conf/

[root@lb01 conf]# vim nginx.conf

worker_processes  1;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
upstream  web_pools {
        server 172.25.254.134:80 weight=5;
        server 172.25.254.135:80 weight=5;
#       server 172.25.254.158:80 weight=5  backup;

}
    server {
        listen       80;
        server_name  www.lbtest.com;
        location / {
           # root   html;
           # index  index.html index.htm;
            proxy_set_header Host $host;
            proxy_pass http://web_pools;
        }
    }
}

[root@lb01 conf]# scp nginx.conf 172.25.254.133:/usr/local/nginx/conf/

[root@lb01 conf]# nginx  -s reload
[root@lb01 conf]# curl 172.25.254.134
172.25.254.134
[root@lb01 conf]# curl 172.25.254.135
172.25.254.135
[root@lb01 conf]# nginx  -s reload
[root@lb02 ~]# curl 172.25.254.134
172.25.254.134
[root@lb02 ~]# curl 172.25.254.135
172.25.254.135

在獲取到VIP後，不在同一網段，爲了方便測試，把VIP設置爲172.25.254.254

[root@lb01 ~]# ip addr|grep 172.25.254.254

3.2 發現不能訪問的問題

訪問測試，發現不能訪問，也不能ping通

[root@lb01 conf]# curl 172.25.254.254

curl: (7) Failed connect to 172.25.254.254:80; Connection timed out

[root@lb01 conf]# ping 172.25.254.254

2 packets transmitted, 0 received, 100% packet loss, time 999ms

解決

[root@lb01 conf]# vim /etc/keepalived/keepalived.conf

# vrrp_strict   # 註釋掉vrrp_strict

[root@lb01 conf]# systemctl restart keepalived

3.3 測試

[root@lb01 conf]# curl 172.25.254.254
172.25.254.135
[root@lb01 conf]# curl 172.25.254.254
172.25.254.134
[root@lb01 conf]# curl 172.25.254.254
172.25.254.135
[root@lb01 conf]# curl 172.25.254.254
172.25.254.134

關閉MASTER測試

[root@lb01 conf]# systemctl stop keepalived   #這時VIP已經在BACKUP上，可是能夠正常訪問
[root@lb01 conf]# curl 172.25.254.254
172.25.254.135
[root@lb01 conf]# curl 172.25.254.254
172.25.254.134
[root@lb01 conf]# curl 172.25.254.254
172.25.254.135
[root@lb01 conf]# curl 172.25.254.254
172.25.254.134
[root@lb01 conf]# curl 172.25.254.254
172.25.254.135

使用主機，域名訪問

172.25.254.254  www.lbtest.com  #寫hosts

正常訪問

4、keepalived的其餘特性

4.1 監控自動遷移腳本

keepalived解決的是主機級別的冗餘，當nginx宕掉的時候，keepalive並不會遷移，這時VIP依然在該主機上，客戶就不能訪問到網站

使用腳本監控，當nginx掛掉，自動停掉keepalive，是VIP漂移，是業務不受影響

[root@lb01 conf]# mkdir /script

[root@lb01 conf]# vim /script/monitor.sh

#!/bin/bash
while true
do
if [ `ps -ef |grep nginx|grep -v grep |wc -l` -lt 2 ]
  then
      systemctl stop keepalived
fi
sleep 5
done

[root@lb01 conf]# cd /script/

[root@lb01 script]# chmod +x monitor.sh

[root@lb01 script]# /script/monitor.sh &

關掉nginx

[root@lb01 script]# nginx -s stop

VIP漂移到BACKUP上

[root@lb02 ~]# ip addr|grep 254.254

inet 172.25.254.254/24 scope global secondary ens33

訪問：

[root@lb01 script]# curl 172.25.254.254
172.25.254.134
[root@lb01 script]# curl 172.25.254.254
172.25.254.135
[root@lb01 script]# curl 172.25.254.254
172.25.254.134
[root@lb01 script]# curl 172.25.254.254
172.25.254.135

4.2 keepalive高可用腦裂腳本

[root@lb01 script]# ps -ef |grep monitor
root 80993 68563 0 07:04 pts/0 00:00:00 /bin/bash /script/monitor.sh
[root@lb01 script]# kill -9 80993
[1]+  Killed                  /script/monitor.sh
[root@lb01 script]# ps -ef |grep monitor
root      82773  68563  0 07:13 pts/0    00:00:00 grep --color=auto monitor
[root@lb01 script]# systemctl restart keepalived

[root@lb02 ~]# mkdir /script

檢測腦裂腳本，實現形式，當BACKUP能夠ping通主，可是VIP依然在BACKUP，則認定爲腦裂

[root@lb02 script]# vim check_split_brain.sh

#!/bin/bash
while true
do
ping -c 2 -W 3 172.25.254.131 &>/dev/null
  if [ $? -eq 0 -a `ip addr|grep 172.25.254.254|wc -l` -eq 1 ]
    then
      echo "ha is split brain warning"
else
      echo "ha is OK"
fi
sleep 3
done

[root@lb02 ~]# sh /script/check_split_brain.sh

ha is OK
ha is OK
[root@lb02 ~]# systemctl start firewalld
ha is split brain warning     
ha is split brain warning
[root@lb02 ~]# ip addr |grep 172.25.254.254
inet 172.25.254.254/24 scope global secondary ens33
[root@lb01 ~]# ip addr |grep 172.25.254.254
inet 172.25.254.254/24 scope global secondary ens33
[root@node4 ~]# systemctl stop firewalld
ha is OK
ha is OK

4.3 修改日誌文件路徑

配置文件默認在/var/log/messages

[root@lb01 ~]# tail -f /var/log/messages
Apr 13 07:41:26 node2 Keepalived_vrrp[82796]: Sending gratuitous ARP on ens33 for 172.25.254.254
Apr 13 07:41:26 node2 Keepalived_vrrp[82796]: Sending gratuitous ARP on ens33 for 172.25.254.254
Apr 13 07:41:26 node2 Keepalived_vrrp[82796]: Sending gratuitous ARP on ens33 for 172.25.254.254
Apr 13 07:41:26 node2 Keepalived_vrrp[82796]: VRRP_Instance(VI_1) Received advert with lower priority 100, ours 150, forcing new election
Apr 13 07:41:26 node2 Keepalived_vrrp[82796]: Sending gratuitous ARP on ens33 for 172.25.254.254
Apr 13 07:41:26 node2 Keepalived_vrrp[82796]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 172.25.254.254

修改日誌文件位置

[root@lb01 ~]# vi /etc/sysconfig/keepalived

# Options for keepalived. See `keepalived --help' output and keepalived(8) and
# keepalived.conf(5) man pages for a list of all options. Here are the most
# common ones :
#
# --vrrp               -P    Only run with VRRP subsystem.
# --check              -C    Only run with Health-checker subsystem.
# --dont-release-vrrp  -V    Dont remove VRRP VIPs & VROUTEs on daemon stop.
# --dont-release-ipvs  -I    Dont remove IPVS topology on daemon stop.
# --dump-conf          -d    Dump the configuration data.
# --log-detail         -D    Detailed log messages.
# --log-facility       -S    0-7 Set local syslog facility (default=LOG_DAEMON)
#

KEEPALIVED_OPTIONS="-D -S 0 -d"

[root@lb01 ~]# vim /etc/rsyslog.conf

local0.*                                                /var/log/keepalived.log

[root@lb01 ~]# systemctl restart rsyslog

[root@lb01 ~]# systemctl restart keepalived

[root@lb01 ~]# tail -f /var/log/keepalived.log

Apr 13 07:46:23 node2 Keepalived_vrrp[84692]: Sending gratuitous ARP on ens33 for 172.25.254.254
Apr 13 07:46:23 node2 Keepalived_vrrp[84692]: Sending gratuitous ARP on ens33 for 172.25.254.254
Apr 13 07:46:23 node2 Keepalived_vrrp[84692]: Sending gratuitous ARP on ens33 for 172.25.254.254
Apr 13 07:46:23 node2 Keepalived_vrrp[84692]: VRRP_Instance(VI_1) Received advert with lower priority 100, ours 150, forcing new election
Apr 13 07:46:23 node2 Keepalived_vrrp[84692]: Sending gratuitous ARP on ens33 for 172.25.254.254
Apr 13 07:46:23 node2 Keepalived_vrrp[84692]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 172.25.254.254

4.4 keepalived多實例

[root@lb01 ~]# vim /etc/keepalived/keepalived.conf

  1 ! Configuration File for keepalived
  2 
  3 global_defs {
  4    notification_email {
  5         283365585@qq.com
  6    }
  7    notification_email_from Alexandre.Cassen@firewall.loc
  8    smtp_server 192.168.200.1
  9    smtp_connect_timeout 30
 10    router_id LVS_01
 11    vrrp_skip_check_adv_addr
 12   # vrrp_strict
 13    vrrp_garp_interval 0
 14    vrrp_gna_interval 0
 15 }
 16 
 17 vrrp_instance VI_1 {
 18     state MASTER
 19     interface ens33
 20     virtual_router_id 51
 21     priority 150
 22     advert_int 1
 23     authentication {
 24         auth_type PASS
 25         auth_pass 1111
 26     }
 27     virtual_ipaddress {
 28     172.25.254.254/24
 29     }
 30 }
 31 vrrp_instance VI_2 {
 32     state BACKUP
 33     interface ens33
 34     virtual_router_id 52
 35     priority 100
 36     advert_int 1
 37     authentication {
 38         auth_type PASS
 39         auth_pass 1111
 40     }
 41     virtual_ipaddress {
 42     172.25.254.253/24
 43     }
 44 }

[root@lb02 ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   notification_email {
        283365585@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_02
   vrrp_skip_check_adv_addr
  # vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
    172.25.254.254/24
    }
}
vrrp_instance VI_2 {
    state MASTER
    interface ens33
    virtual_router_id 52
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
    172.25.254.253/24
    }
}

檢測

[root@lb01 ~]# systemctl restart keepalived
[root@lb02 ~]# systemctl restart keepalived
[root@lb01 ~]# ip addr
 inet 172.25.254.131/24 brd 172.25.254.255 scope global dynamic ens33
       valid_lft 499sec preferred_lft 499sec
    inet 172.25.254.254/24 scope global secondary ens33
       valid_lft forever preferred_lft forever
[root@lb02 ~]# ip addr
inet 172.25.254.133/24 brd 172.25.254.255 scope global dynamic ens33
       valid_lft 422sec preferred_lft 422sec
    inet 172.25.254.253/24 scope global secondary ens33
       valid_lft forever preferred_lft forever
[root@lb02 ~]# systemctl stop keepalived
[root@lb02 ~]# ip addr|grep 172.25.254
    inet 172.25.254.133/24 brd 172.25.254.255 scope global dynamic ens33
[root@lb01 ~]# ip addr |grep 172.25.254
    inet 172.25.254.131/24 brd 172.25.254.255 scope global dynamic ens33
    inet 172.25.254.254/24 scope global secondary ens33
    inet 172.25.254.253/24 scope global secondary ens33
[root@lb02 ~]# systemctl start keepalived
[root@lb02 ~]# ip addr|grep 172.25.254
    inet 172.25.254.133/24 brd 172.25.254.255 scope global dynamic ens33
    inet 172.25.254.253/24 scope global secondary ens33
[root@lb01 ~]# ip addr |grep 172.25.254
    inet 172.25.254.131/24 brd 172.25.254.255 scope global dynamic ens33
    inet 172.25.254.254/24 scope global secondary ens33