搭建LVS+Keepalived高可用負載均衡集羣

LVS+Keepalived 介紹

LVS
LVS是Linux Virtual Server的簡寫，意即Linux虛擬服務器，是一個虛擬的服務器集羣系統。本項目在1998年5月由章文嵩博士成立，是中國國內最先出現的自由軟件項目之一。目前有三種IP負載均衡技術（VS/NAT、VS/TUN和VS/DR）；
十種調度算法（rrr|wrr|lc|wlc|lblc|lblcr|dh|sh|sed|nq）。
Keepalvied
Keepalived在這裏主要用做RealServer的健康狀態檢查以及LoadBalance主機和BackUP主機之間failover的實現

集羣IP地址規劃

節點類型	ip	主機名
Master DR	192.168.2.128	centos.peng.com
Backup DR	192.168.2.129	centos1.peng.com
Real Server (nginx)	192.168.2.130	centos2.peng.com
Real Server (nginx)	192.168.2.131	centos3.peng.com
VIP	192.168.2.200	www.peng.com

安裝LVS

檢查系統是否支持lvs

[root@centos ~]# modprobe -l |grep ip_vs

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_dh.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_ftp.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_lblc.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_lblcr.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_lc.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_nq.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_rr.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_sed.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_sh.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_wlc.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_wrr.ko

若有相似以上輸出，說明系統支持IPVS模塊

安裝ipvsadm

[root@centos ~]# tar zxvf ipvsadm-1.24.tar.gz

[root@centos ~]# cd ipvsadm-1.24

[root@centos ipvsadm-1.24]# ln -s /usr/src/kernels/2.6.18-348.el5-x86_64/ /usr/src/linux/

[root@centos ipvsadm-1.24]# make

[root@centos ipvsadm-1.24]# make install

安裝keepalived

[root@centos1 ~]# tar zxvf keepalived-1.1.19.tar.gz

[root@centos1 ~]# cd keepalived-1.1.19

[root@centos1 keepalived-1.1.19]#./configure  --with-kernel-dir=/usr/src/kernels/2.6.18-348.el5-x86_64

[root@centos1 keepalived-1.1.19]# make

[root@centos1 keepalived-1.1.19]# make install

[root@centos1 keepalived-1.1.19]# ln -s /usr/local/sbin/keepalived /sbin/

Real Server 配置

LVS的DR和TUN模式下，用戶的請求不通過DR直接返回給用戶，所以須要在Real Server上添加VIP。用腳本realserver.sh在Real Server上執行，實現自動添加VIP的功能。

#!/bin/bash

VIP=192.168.2.200      

/etc/rc.d/init.d/functions  

case "$1" in

start)

      /sbin/ifconfig lo:0 $VIP netmask 255.255.255.255 broadcast $VIP

      /sbin/route add -host $VIP dev lo:0

      echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

      echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

      echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore

      echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

      sysctl -p >/dev/null 2>&1

      echo "RealServer Start OK"

      ;;

stop)

      /sbin/ifconfig lo:0 down

      /sbin/route del $VIP >/dev/null 2>&1

      echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore

      echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce

      echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore

      echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce

      echo "RealServer Stoped"

      ;;

*)

      echo "Usage: $0 {start|stop}"

      exit 1

esac

exit 0

配置keepalived

Keepalived的配置文件爲/etc/Keepalived/Keepalived.conf

配置以下所示：

! Configuration File for keepalived

global_defs {

  notification_email {

         79907277@qq.com

  }

  notification_email_from 79907277@qq.com

  smtp_server smtp.qq.com

  smtp_connect_timeout 30

  router_id LVS_DEVEL

}

vrrp_instance VI_1 {

state MASTER

   interface eth0

   virtual_router_id 51

priority 100

   advert_int 1

   authentication {

       auth_type PASS

       auth_pass 1111

   }

   virtual_ipaddress {

       192.168.2.200

   }

}

virtual_server 192.168.2.200 80 {

   delay_loop 6

   lb_algo rr

   lb_kind DR

   nat_mask 255.255.255.0

   persistence_timeout 50

protocol TCP

real_server 192.168.2.130 80 {

       weight 1

       TCP_CHECK {

           connect_timeout 3

           nb_get_retry 3

           delay_before_retry 3

                 }

       }

   real_server 192.168.2.131 80 {

       weight 1

       TCP_CHECK {

           connect_timeout 3

           nb_get_retry 3

           delay_before_retry 3

                 }

       }

}

Backup DR的配置文件基本相同，state MASTER 換成 state backup ，priority 100 換成prority 80 便可。

啓動keepalived 和realserver.sh

[root@centos keepalived]# /etc/init.d/keepalived start

[root@centos3 ~]# ./realserver.sh

   啓動realserver.sh腳本後，Realserver ip地址以下所示：

[root@centos2 ~]# ifconfig

eth0      Link encap:Ethernet  HWaddr 00:0C:29:76:A3:21  

         inet addr:192.168.2.130  Bcast:192.168.2.255  Mask:255.255.255.0

         inet6 addr: fe80::20c:29ff:fe76:a321/64 Scope:Link

         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

         RX packets:8902 errors:0 dropped:0 overruns:0 frame:0

         TX packets:6316 errors:0 dropped:0 overruns:0 carrier:0

         collisions:0 txqueuelen:1000

         RX bytes:2713466 (2.5 MiB)  TX bytes:1065526 (1.0 MiB)

lo        Link encap:Local Loopback  

         inet addr:127.0.0.1  Mask:255.0.0.0

         inet6 addr: ::1/128 Scope:Host

         UP LOOPBACK RUNNING  MTU:16436  Metric:1

         RX packets:4559 errors:0 dropped:0 overruns:0 frame:0

         TX packets:4559 errors:0 dropped:0 overruns:0 carrier:0

         collisions:0 txqueuelen:0

         RX bytes:10562584 (10.0 MiB)  TX bytes:10562584 (10.0 MiB)

lo:0      Link encap:Local Loopback  

         inet addr:192.168.2.200  Mask:255.255.255.255

         UP LOOPBACK RUNNING  MTU:16436  Metric:1

測試LVS+keepalived

在兩臺Real Server上利用nginx新建兩臺web服務器，都將監聽的虛擬VIP指向www.peng.com。經過兩臺客戶端（一臺win 7，一臺opensuse）訪問www.peng.com。訪問結果以下所示：

Win7 下的IE訪問：

Opensuse下的firefox訪問：

能夠看到，經過不一樣客戶端訪問到的是不一樣的realserver，一臺是centos2，一臺是centos3。

利用ipvsadm查看以下：

Lvs監聽到了192.168.2.130:80和192.168.2.131:80兩臺RealServer，131產生了4個連接，130產生了1個連接。

由此能夠肯定經過LVS+keepalived搭建的高可用負載均衡web服務順利完成。

模擬DR故障

當Master DR當機的時候，咱們經過關閉keepalived服務來模擬這一場景。看一下，Backup DR是怎樣繼續提供服務的。觀察此時Backup DR日誌，以下所示：

[root@centos1 ~]# tail -f /var/log/messages

Aug  6 05:45:23 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE

Aug  6 05:45:24 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE

Aug  6 05:45:24 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.

Aug  6 05:45:24 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.200

Aug  6 05:45:24 centos1 Keepalived_vrrp: Netlink reflector reports IP 192.168.2.200 added

Aug  6 05:45:24 centos1 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.2.200 added

Aug  6 05:45:29 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.200

能夠看到Backup DR 立刻將本身調整爲MASTER STATE ，並接管了虛擬ip192.168.2.200，讓web服務仍然可以正常訪問。

當從新啓動Master DR上的keepalived服務，咱們的Master DR從當機中恢復以後。觀察Backup DR日誌以下：

Aug  6 05:45:24 centos1 Keepalived_vrrp: Netlink reflector reports IP 192.168.2.200 added

Aug  6 05:45:24 centos1 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.2.200 added

Aug  6 05:45:29 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.200

Aug  6 05:52:34 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert

Aug  6 05:52:34 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE

Aug  6 05:52:34 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.

Aug  6 05:52:34 centos1 Keepalived_vrrp: Netlink reflector reports IP 192.168.2.200 removed

Aug  6 05:52:34 centos1 Keepalived_healthcheckers: Netlink reflector reports IP192.168.2.200 removed

能夠看到Backup DR 立刻又乖乖的將本身變成了BACKUP STATE 並移除了虛擬ip192.168.2.200，繼續作着萬年的備胎。

模擬RealServer故障

當RealServer當機的時候，咱們經過關閉RealServer上的 nginx服務來模擬這一場景。看一下，DR是怎樣運行的。觀察此時Master DR日誌，以下所示：

Aug  6 05:52:34 centos Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE

Aug  6 05:52:35 centos Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE

Aug  6 05:52:35 centos Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.

Aug  6 05:52:35 centos Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.200

Aug  6 05:52:35 centos Keepalived_vrrp: Netlink reflector reports IP 192.168.2.200 added

Aug  6 05:52:35 centos Keepalived_healthcheckers: Netlink reflector reports IP 192.168.2.200 added

Aug  6 05:52:40 centos Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.200

Aug  6 05:58:34 centos Keepalived_healthcheckers: TCP connection to [192.168.2.130:80] failed !!!

Aug  6 05:58:34 centos Keepalived_healthcheckers: Removing service [192.168.2.130:80] from VS [192.168.2.200:80]

Aug  6 05:58:34 centos Keepalived_healthcheckers: SMTP connection ERROR to [127.0.0.1:25].

能夠看到keepalived檢測到192.168.2.130:80服務當機，立刻將其從VS中移除。此時VS中應該只有192.168.2.131:80提供服務，咱們經過瀏覽器看一下，以下所示：

果然如此，經過兩個客戶端訪問到的都是centos3提供的服務。