LVS/DR + keepalived負載均衡高可用實現

時間  2019-12-24
標籤 lvs keepalived 負載 均衡 可用 實現 欄目 負載均衡 简体版
原文   原文鏈接

1、keepalived簡介前端

keepalived是分佈式部署解決系統高可用的軟件，結合lvs(LinuxVirtual Server)使用，解決單機宕機的問題。
keepalived是一個基於VRRP協議來實現IPVS的高可用的解決方案。對於LVS負載均衡來講，若是前端的調度器direct發生故障，則後端的realserver是沒法接受請求並響應的。所以，保證前端direct的高可用性是很是關鍵的，不然後端的服務器是沒法進行服務的。而咱們的keepalived就能夠用來解決單點故障（如LVS的前端direct故障）問題。keepalived的主要工做原理是：運行keepalived的兩臺服務器，其中一臺爲MASTER，另外一臺爲BACKUP，正常狀況下，全部的數據轉換功能和ARP請求響應都是由MASTER完成的，一旦MASTER發生故障，則BACKUP會立刻接管MASTER的工做，這種切換時很是迅速的。linux

2、測試環境web

下面拿4臺虛擬機進行環境測試,實驗環境爲centos6.6 x86_64,具體用途和ip以下算法

服務器類型後端

IP地址centos

Lvs VIPbash

192.168.214.89服務器

Keepalived Master網絡

192.168.214.85負載均衡

Keepalived Backup

192.168.214.86

            Realserver A

192.168.214.87

            Realserver B

192.168.214.88

 

3、軟件安裝

1、安裝lvs所需包ipvsadm

yum install -y ipvsadm

ln -s /usr/src/kernels/`uname -r` /usr/src/linux

lsmod |grep ip_vs

 

#注意Centos 6.X安裝lvs，使用1.26版本。而且須要先安裝yuminstall libnl* popt* -y

 

執行ipvsadm(modprobe ip_vs)把ip_vs模塊加載到內核

[root@test85 ~]# ipvsadm -L -n

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

 -> RemoteAddress:Port          Forward Weight ActiveConn InActConn

 

#IP Virtual Server version 1.2.1 ---- ip_vs內核模塊版本


2、安裝keepalived

yum install -y keepalived

chkconfig keepalived on

注：在centos7系列系統中開機自動啓動使用systemctl enable keepalived


4、keepalived配置

先看下214.85keepalived主機上的配置

[root@test85 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
#配置報警郵箱
   notification_email {
     charles@test.com
   }
   notification_email_from reportlog@test.com
   smtp_server mail.test.com
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}
vrrp_sync_group VG1 {
   group {
      VI_1
   }
}

vrrp_instance VI_1 {
    state MASTER  #指定keepalived的角色，MASTER表示此主機是主服務器，BACKUP表示此主機是備用服務器
    interface eth0  #指定HA監測網絡的接口
    lvs_sync_daemon_inteface eth0
virtual_router_id 55
#虛擬路由標識，這個標識是一個數字，同一個vrrp實例使用惟一的標識。即同一vrrp_instance下，MASTER和BACKUP必須是一致的
    priority 100 #定義優先級，數字越大，優先級越高，在同一個vrrp_instance下，MASTER的優先級必須大於BACKUP的優先級
    advert_int 1 #設定MASTER與BACKUP負載均衡器之間同步檢查的時間間隔，單位是秒
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {    #設置虛擬IP地址
        192.168.214.89  
    }
}

virtual_server 192.168.214.89 80 {
    delay_loop 6          #(每隔6秒查詢realserver狀態)
    lb_algo rr             #(lvs 算法)
    lb_kind DR            #(使用lvs的DR模式)
    #nat_mask 255.255.255.0
    persistence_timeout 10  #(同一IP的鏈接10秒內被分配到同一臺realserver)
    protocol TCP  #(用TCP協議檢查realserver狀態)

    real_server 192.168.214.87 80 {
        weight 100          #(權重)
       TCP_CHECK {
            connect_timeout 3   #(3秒無響應超時)
            connect_port 80
            nb_get_retry 3
            delay_before_retry 3
        }
    }
    real_server 192.168.214.88 80 {
        weight 100
       TCP_CHECK {
            connect_timeout 3
            connect_port 80
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}

再看下214.86keepalived備機上的配置

! Configuration File for keepalived

global_defs {
   notification_email {
     charles@test.com
   }
   notification_email_from reportlog@test.com
   smtp_server mail.test.com
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}
vrrp_sync_group VG1 {
   group {
      VI_1
   }
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    lvs_sync_daemon_inteface eth0
    virtual_router_id 55
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.214.89
    }
}

virtual_server 192.168.214.89 80 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
    #nat_mask 255.255.255.0
    persistence_timeout 10
    protocol TCP

    real_server 192.168.214.87 80 {
        weight 100
       TCP_CHECK {
            connect_timeout 3
            connect_port 80
            nb_get_retry 3
            delay_before_retry 3
        }
    }
    real_server 192.168.214.88 80 {
        weight 100
       TCP_CHECK {
            connect_timeout 3
            connect_port 80
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}


5、後端realserver操做

DR模式須要在後端真實機上運行如下腳本

#!/bin/bash
# description: Config realserver lo
#Written by :Charles

VIP1=192.168.214.89
. /etc/rc.d/init.d/functions

case 「$1」 in
start)
       ifconfig lo:0 $VIP1 netmask 255.255.255.255 broadcast $VIP1
       /sbin/route add –host $VIP1 dev lo:0
       echo 「1」 >/proc/sys/net/ipv4/conf/lo/arp_ignore
       echo 「2」 >/proc/sys/net/ipv4/conf/lo/arp_announce
       echo 「1」 >/proc/sys/net/ipv4/conf/all/arp_ignore
       echo 「2」 >/proc/sys/net/ipv4/conf/all/arp_announce
       sysctl –p >/dev/null 2>&1
       echo 「RealServer Start OK」
       ;;
stop)
       ifconfig lo:0 down
       route del $VIP1 >/dev/null 2>&1
       echo 「0」 >/proc/sys/net/ipv4/conf/lo/arp_ignore
       echo 「0」 >/proc/sys/net/ipv4/conf/lo/arp_announce
       echo 「0」 >/proc/sys/net/ipv4/conf/all/arp_ignore
       echo 「0」 >/proc/sys/net/ipv4/conf/all/arp_announce
       echo 「RealServer Stoped」
       ;;
*)
       echo 「Usage: $0 {start|stop}」
       exit 1
esac

exit 0


#執行realserver.sh start開啓，stop關閉

#腳本設置成755權限，並放入rc.local下讓其開機啓動運行


6、啓動keepalived服務及查看相關信息

在214.85和214.86上分別啓動keepalived服務

 

在214.85keepalived主機上查看信息

經過ip addr能夠看到vip 地址已經綁定在eth0網口上

[root@test85 ~]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu65536 qdisc noqueue state UNKNOWN

   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

   inet 127.0.0.1/8 scope host lo

   inet6 ::1/128 scope host

      valid_lft forever preferred_lft forever

2: eth0:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen1000

   link/ether 00:0c:29:85:7a:67 brd ff:ff:ff:ff:ff:ff

   inet 192.168.214.85/24 brd 192.168.214.255 scope global eth0

    inet 192.168.214.89/32 scope global eth0

   inet6 fe80::20c:29ff:fe85:7a67/64 scope link

      valid_lft forever preferred_lft forever


在214.85上查看日誌信息，看到已成功進入keepalived主機模式

[root@test85 ~]# tail -f /var/log/messages

May 4 14:12:34 test85 Keepalived_vrrp[7977]: VRRP_Instance(VI_1) Entering MASTER STATE

May 4 14:12:34 test85 Keepalived_vrrp[7977]: VRRP_Instance(VI_1) settingprotocol VIPs.

May 4 14:12:34 test85 Keepalived_healthcheckers[7975]: Netlink reflector reports IP 192.168.214.89 added

May 4 14:12:34 test85 Keepalived_vrrp[7977]: VRRP_Instance(VI_1) Sendinggratuitous ARPs on eth0 for 192.168.214.89

May 4 14:12:34 test85 Keepalived_vrrp[7977]: VRRP_Group(VG1) Syncinginstances to MASTER state

May 4 14:12:36 test85 ntpd[1148]: Listen normally on 7 eth0 192.168.214.89UDP 123

May 4 14:12:36 test85 ntpd[1148]: peers refreshed

May 4 14:12:39 test85 Keepalived_vrrp[7977]: VRRP_Instance(VI_1) Sendinggratuitous ARPs on eth0 for 192.168.214.89

May 4 14:12:40 test85 root[7924] 192.168.5.80 53823 192.168.214.85 22:#1525414360

May 4 14:12:40 test85 root[7924] 192.168.5.80 53823 192.168.214.85 22: ipaddr


在214.86上查看日誌信息，看到已成功進入keepalived備機模式

May 4 14:12:37 web86 Keepalived_vrrp[31009]: Using LinkWatch kernel netlinkreflector...

May 4 14:12:37 web86 Keepalived_vrrp[31009]: VRRP_Instance(VI_1) Entering BACKUP STATE

May 4 14:12:37 web86 Keepalived_vrrp[31009]: VRRP sockpool: [ifindex(2),proto(112), unicast(0), fd(10,11)]

May 4 14:12:37 web86 Keepalived_healthcheckers[31007]: Opening file'/etc/keepalived/keepalived.conf'.

May 4 14:12:37 web86 Keepalived_healthcheckers[31007]: Configuration isusing : 14713 Bytes

May 4 14:12:37 web86 Keepalived_healthcheckers[31007]: Using LinkWatchkernel netlink reflector...

May 4 14:12:37 web86 Keepalived_healthcheckers[31007]: Activatinghealthchecker for service [192.168.214.87]:80

May 4 14:12:37 web86 Keepalived_healthcheckers[31007]: Activatinghealthchecker for service [192.168.214.88]:80


後端真實機啓動腳本後，查看網卡信息，看到vip已成功綁定在迴環口上。

[root@web87 ~]# ipaddr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu65536 qdisc noqueue state UNKNOWN

   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

   inet 127.0.0.1/8 scope host lo

    inet 192.168.214.89/32 brd 192.168.214.89 scope global lo:0

   inet6 ::1/128 scope host

      valid_lft forever preferred_lft forever

2: eth0:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen1000

   link/ether 00:0c:29:38:31:ad brd ff:ff:ff:ff:ff:ff

   inet 192.168.214.87/24 brd 192.168.214.255 scope global eth0

   inet6 fe80::20c:29ff:fe38:31ad/64 scope link

      valid_lft forever preferred_lft forever


經過ipvsadm –L –n查看相應lvs鏈接信息
[root@test85 ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.214.89:80 rr persistent 10
  -> 192.168.214.87:80            Route   100    2          2   
  -> 192.168.214.88:80            Route   100    0          0


7、keepalived測試

使用vip地址192.168.214.89訪問後端192.168.214.87和192.168.214.88的頁面

01.jpg

02.jpg


正常訪問沒問題後，咱們來模擬lvs集羣故障

首先把keepalived master主機214.85宕機，看備機可否接管過來，vip地址是否會漂移過來

在214.86上查看日誌，發現備機成功切換成了主機狀態，vip地址成功漂移了過來

May 4 14:35:34 web86 Keepalived_vrrp[31009]: VRRP_Instance(VI_1) Transition to MASTER STATE

May 4 14:35:34 web86 Keepalived_vrrp[31009]: VRRP_Group(VG1) Syncinginstances to MASTER state

May 4 14:35:35 web86 Keepalived_vrrp[31009]: VRRP_Instance(VI_1) EnteringMASTER STATE

May 4 14:35:35 web86 Keepalived_vrrp[31009]: VRRP_Instance(VI_1) settingprotocol VIPs.

May 4 14:35:35 web86 Keepalived_vrrp[31009]: VRRP_Instance(VI_1) Sendinggratuitous ARPs on eth0 for 192.168.214.89

May 4 14:35:35 web86 Keepalived_healthcheckers[31007]: Netlink reflectorreports IP 192.168.214.89 added

May 4 14:35:36 web86 ntpd[1230]: Listen normally on 7 eth0 192.168.214.89UDP 123

May 4 14:35:36 web86 ntpd[1230]: peers refreshed

May 4 14:35:40 web86 Keepalived_vrrp[31009]: VRRP_Instance(VI_1) Sendinggratuitous ARPs on eth0 for 192.168.214.89


而後，再把214.85主機恢復，因爲214.85擁有較高的優先級，會從214.86搶回MASTER狀態，相應的214.86會迴歸到原來的Backup狀態


214.85日誌記錄，從新回到了MASTER狀態

May 4 14:41:55 test85 Keepalived_vrrp[8066]: VRRP_Instance(VI_1) Transitionto MASTER STATE

May 4 14:41:55 test85 Keepalived_vrrp[8066]: VRRP_Instance(VI_1) Receivedlower prio advert, forcing new election

May 4 14:41:55 test85 Keepalived_vrrp[8066]: VRRP_Group(VG1) Syncinginstances to MASTER state

May 4 14:41:56 test85 Keepalived_vrrp[8066]: VRRP_Instance(VI_1) EnteringMASTER STATE

May 4 14:41:56 test85 Keepalived_vrrp[8066]: VRRP_Instance(VI_1) settingprotocol VIPs.

May 4 14:41:56 test85 Keepalived_vrrp[8066]: VRRP_Instance(VI_1) Sendinggratuitous ARPs on eth0 for 192.168.214.89

May 4 14:41:56 test85 Keepalived_healthcheckers[8064]: Netlink reflectorreports IP 192.168.214.89 added

May 4 14:41:58 test85 ntpd[1148]: Listen normally on 8 eth0 192.168.214.89UDP 123

May 4 14:41:58 test85 ntpd[1148]: peers refreshed

May 4 14:42:01 test85 Keepalived_vrrp[8066]: VRRP_Instance(VI_1) Sendinggratuitous ARPs on eth0 for 192.168.214.89


218.86日誌記錄,接收到了高優先級請求，從以前的MASTER狀態變回了BACKUP狀態

May 4 14:35:34 web86 Keepalived_vrrp[31009]: VRRP_Group(VG1) Syncinginstances to MASTER state

May 4 14:35:35 web86 Keepalived_vrrp[31009]: VRRP_Instance(VI_1) Entering MASTER STATE

May 4 14:35:35 web86 Keepalived_vrrp[31009]: VRRP_Instance(VI_1) settingprotocol VIPs.

May 4 14:35:35 web86 Keepalived_vrrp[31009]: VRRP_Instance(VI_1) Sendinggratuitous ARPs on eth0 for 192.168.214.89

May 4 14:35:35 web86 Keepalived_healthcheckers[31007]: Netlink reflectorreports IP 192.168.214.89 added

May 4 14:35:36 web86 ntpd[1230]: Listen normally on 7 eth0 192.168.214.89UDP 123

May 4 14:35:36 web86 ntpd[1230]: peers refreshed

May 4 14:35:40 web86 Keepalived_vrrp[31009]: VRRP_Instance(VI_1) Sendinggratuitous ARPs on eth0 for 192.168.214.89

May 4 14:36:41 web86 root[30963] 192.168.5.80 53824 192.168.214.86 22:#1525415801

May 4 14:36:41 web86 root[30963] 192.168.5.80 53824 192.168.214.86 22: ipaddr

May 4 14:41:55 web86 Keepalived_vrrp[31009]: VRRP_Instance(VI_1) Received higher prio advert

May 4 14:41:55 web86 Keepalived_vrrp[31009]: VRRP_Instance(VI_1) Entering BACKUP STATE

May 4 14:41:55 web86 Keepalived_vrrp[31009]: VRRP_Instance(VI_1) removingprotocol VIPs.

May 4 14:41:55 web86 Keepalived_vrrp[31009]: VRRP_Group(VG1) Syncinginstances to BACKUP state

May 4 14:41:55 web86 Keepalived_healthcheckers[31007]: Netlink reflectorreports IP 192.168.214.89 removed

May 4 14:41:56 web86 ntpd[1230]: Deleting interface #7 eth0,192.168.214.89#123, interface stats: received=0, sent=0, dropped=0,active_time=380 secs


最後，再模擬後端真實機214.87服務宕掉，看是否vip只請求214.88

經過日誌查看得知，keepalived集羣探測到後端真實機214.87的80端口不通，把它從vip請求列表中移除了

 

May  414:48:00 test85 Keepalived_healthcheckers[8064]: TCP connection to[192.168.214.87]:80 failed !!!

May 4 14:48:00 test85Keepalived_healthcheckers[8064]: Removing service [192.168.214.87]:80 from VS[192.168.214.89]:80

 

 

當從新探測到後端真實機214.87服務恢復後，又把它加入了請求列表中

May 4 14:52:55 test85 Keepalived_healthcheckers[8064]: TCP connection to[192.168.214.87]:80 success.

May 4 14:52:55 test85 Keepalived_healthcheckers[8064]: Adding service[192.168.214.87]:80 to VS [192.168.214.89]:80


若是想了解更多，請關注咱們的公衆號
公衆號ID:opdevos

掃碼關注

gongzhouhao.jpg

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程