Keepalived+LVS(dr)高可用負載均衡集羣的實現

時間  2019-11-09
標籤 keepalived+lvs keepalived lvs 可用 負載 均衡 集羣 實現 欄目 負載均衡 简体版
原文   原文鏈接

一 環境介紹

1.操做系統
CentOS Linux release 7.2.1511 (Core)php

2.服務
keepalived+lvs雙主高可用負載均衡集羣及LAMP應用
keepalived-1.2.13-7.el7.x86_64
ipvsadm-1.27-7.el7.x86_64
httpd-2.4.6-45.el7.centos.x86_64
mariadb-5.5.52-1.el7.x86_64
php-5.4.16-42.el7.x86_64html

二 原理及拓撲圖

1.vrrp協議
vrrp(Virtual Redundant Routing Protocol)協議：
在現實的網絡環境中，兩臺須要通訊的主機大多數狀況下並無直接的物理鏈接。對於這樣的狀況，它們之間路由怎樣選擇？主機如何選定到達目的主機的下一跳路由，這個問題一般的解決方法有兩種：
 在主機上使用動態路由協議(RIP、OSPF等)
 在主機上配置靜態路由
很明顯，在主機上配置動態路由是很是不切實際的，由於管理、維護成本以及是否支持等諸多問題。配置靜態路由就變得十分流行，但路由器(或者說默認網關default gateway)卻常常成爲單點故障。VRRP的目的就是爲了解決靜態路由單點故障問題，VRRP經過一競選(election)協議來動態的將路由任務交給LAN中虛擬路由器中的某臺VRRP路由器。前端

2.keepalived簡介
Keepalived 是一個基於VRRP協議來實現的LVS服務高可用方案，能夠利用其來避免單點故障。一個LVS服務會有2臺服務器運行Keepalived，一臺爲主服務器（MASTER），一臺爲備份服務器（BACKUP），可是對外表現爲一個虛擬IP，主服務器會發送特定的消息給備份服務器，當備份服務器收不到這個消息的時候，即主服務器宕機的時候，備份服務器就會接管虛擬IP，繼續提供服務，從而保證了高可用性。Keepalived是VRRP的完美實現。node

3.lvs-dr
Direct Routing，直接路由,經過爲請求報文從新封裝一個MAC首部進行轉發，源MAC是DIP所在的接口的MAC，目標MAC是某挑選出的RS的RIP所在接口的MAC地址；源IP/PORT，以及目標IP/PORT均保持不變。  
Director和各RS都得配置使用VIP； 
(1) 確保前端路由器將目標IP爲VIP的請求報文發往Director；
 (a) 在前端網關作靜態綁定；
 (b) 在RS上使用arptables；
 (c) 在RS上修改內核參數以限制arp通告及應答級別；
  arp_announce
  arp_ignore
(2) RS的RIP可使用私網地址，也能夠是公網地址；RIP與DIP在同一IP網絡；RIP的網關不能指向DIP，以確保響應報文不會經由Director； 
(3) RS跟Director要在同一個物理網絡；
(4) 請求報文要經由Director，但響應不能經由Director，而是由RS直接發往Client；
(5) 不支持端口映射。web

4.IP分配
VIP1:172.18.67.66
VIP2:172.18.67.88
DIP1:172.18.67.13
DIP2:172.18.67.14
RIP1:172.18.67.11
RIP2:172.18.67.12
CIP:172.18.67.3算法

5.拓撲圖vim

 

三 keepalived配置

1.安裝keepalived後端

[root@inode2 ~]# yum install -y keepalived
[root@inode3 ~]# yum install -y keepalived

 

2.高可用的ipvs雙主集羣配置  
第一個節點：centos

[root@inode2 ~]# cd /etc/keepalived/
[root@inode2 keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost                  　　　　　　　　　　　　　　#接受通知的郵件地址
}
notification_email_from kaadmin@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node2                 　　　　　　　　　　　　　　 #路由節點
vrrp_mcast_group4 224.0.67.67              　　　　　　　　#多播地址，範圍224.0.0.0~239.255.255.255
}
vrrp_instance myr1 {
state MASTER
interface eno16777736                　　　　　　　　　　　　#網卡接口
virtual_router_id 167                　　　　　　　　　　　　#虛擬路由ID號，0~255
priority 100                  　　　　　　　　　　　　　　　　#優先級，MASTER比BACKUP優先級高
advert_int 1
authentication {
 auth_type PASS
 auth_pass f1bf7fda
}
virtual_ipaddress {
 172.18.67.66/16 dev eno16777736 label eno16777736:0
}
track_interface {
 eno16777736
}
notify_master "/etc/keepalived/notify.sh master"         #調用通知腳本
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
vrrp_instance myr2 {
state BACKUP
interface eno16777736
virtual_router_id 168
priority 98
advert_int 1
authentication {
 auth_type PASS
 auth_pass f2bf7ade
}
virtual_ipaddress {
 172.18.67.88/16 dev eno16777736 label eno16777736:1
}
track_interface {
 eno16777736
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
virtual_server 172.18.67.66 80 {              #VIP
delay_loop 2
lb_algo wrr                  　　　　　　　　　　#lvs負載均衡調度算法
lb_kind DR                   　　　　　　　　　　#負載均衡類型
protocol TCP                  　　　　　　　　　 #傳輸協議
sorry_server 127.0.0.1 80               　　　 #localhost
real_server 172.18.67.11 80 {              　  #後端RIP
 weight 1                  　　　　　　　　　　　 #調度權重 
 HTTP_GET {                  　　　　　　　　　　#http請求方式
 url {
  path /
  status_code 200               　　　　　　　　#狀態碼
 }
 connect_timeout 2                　　　　　　  #鏈接超時
 nb_get_retry 3
 delay_before_retry 3
 }
}
real_server 172.18.67.12 80 {
 weight 1
 HTTP_GET {
  url {
   path /
   status_code 200
  }
  connect_timeout 2
  nb_get_retry 3
  delay_before_retry 3
  }
 }
}

 

第二個節點：bash

[root@inode3 ~]# cd /etc/keepalived/
[root@inode3 keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from kaadmin@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node3
vrrp_mcast_group4 224.0.67.67
}
vrrp_instance myr1 {
state BACKUP
interface eno16777736
virtual_router_id 167
priority 98
advert_int 1
authentication {
 auth_type PASS
 auth_pass f1bf7fda
}
virtual_ipaddress {
 172.18.67.66/16 dev eno16777736 label eno16777736:0
}
track_interface {
 eno16777736
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
vrrp_instance myr2 {
state MASTER
interface eno16777736
virtual_router_id 168
priority 100
advert_int 1
authentication {
 auth_type PASS
 auth_pass f2bf7ade
}
virtual_ipaddress {
 172.18.67.88/16 dev eno16777736 label eno16777736:1
}
track_interface {
 eno16777736
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
virtual_server 172.18.67.88 80 {
delay_loop 2
lb_algo wrr
lb_kind DR
protocol TCP
sorry_server 127.0.0.1 80
real_server 172.18.67.11 80 {
 weight 1
 HTTP_GET {
 url {
  path /
  status_code 200
 }
 connect_timeout 2
 nb_get_retry 3
 delay_before_retry 3
 }
}
real_server 172.18.67.12 80 {
 weight 1
 HTTP_GET {
  url {
   path /
   status_code 200
  }
  connect_timeout 2
  nb_get_retry 3
  delay_before_retry 3
  }
 }
}


3.郵件通知腳本
當雙主高可用集羣主備切換時可經過郵件通知管理員，此時在配置文件中可自動調用實現編輯好的腳本

[root@inode2 ~]# vim notify.sh
#!/bin/bash
#
contact='root@localhost'
notify() {
 mailsubject="$(hostname) to be $1, vip floating"
 mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
 echo "$mailbody" | mail -s "$mailsubject" $contact
}
case $1 in
master)
 notify master
 ;;
backup)
 notify backup
 ;;
fault)
 notify fault
 ;;
*)
 echo "Usage: $(basename $0) {master|backup|fault}"
 exit 1
 ;;
esac


節點二一樣配置。

四 後端RS服務器的配置

1.配置LAMP環境

[root@inode4 ~]# yum install httpd mariadb-server php -y
[root@inode5 ~]# yum install httpd mariadb-server php -y

 

2.簡單編輯測試網頁

[root@inode4 ~]# echo "RS1:172.18.67.11" > /var/www/html/index.html
[root@inode5 ~]# echo "RS2:172.18.67.12" > /var/www/html/index.html

 

3.修改RS內核參數
dr模型中，各主機上均須要配置VIP，解決地址衝突的方式有三種：
(1) 在前端網關作靜態綁定；
(2) 在各RS使用arptables；
(3) 在各RS修改內核參數，來限制arp響應和通告的級別；
限制響應級別：arp_ignore
 0：默認值，表示可以使用本地任意接口上配置的任意地址進行響應；
 1: 僅在請求的目標IP配置在本地主機的接收到請求報文接口上時，纔給予響應；
限制通告級別：arp_announce
 0：默認值，把本機上的全部接口的全部信息向每一個接口上的網絡進行通告；
 1：儘可能避免向非直接鏈接網絡進行通告；
 2：必須避免向非本網絡通告。
可經過編輯腳本實現：

[root@inode4 ~]# vim dr.sh
#!/bin/bash
#
case $1 in
start)
 echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
 echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_ignore
 echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
 echo 2 > /proc/sys/net/ipv4/conf/eth0/arp_announce
 ;;
stop)
 echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
 echo 0 > /proc/sys/net/ipv4/conf/eth0/arp_ignore
 echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
 echo 0 > /proc/sys/net/ipv4/conf/eth0/arp_announce
 ;;
*) 
 echo "Usage $(basename $0) start|stop"
 exit 1
 ;;
esac


同理第二個RS需一樣配置

4.添加路由
節點一：

[root@inode4 ~]# ifconfig lo:0 172.18.67.66 netmask 255.255.255.255 broadcast 172.18.67.66 up
[root@inode4 ~]# ifconfig lo:1 172.18.67.88 netmask 255.255.255.255 broadcast 172.18.67.88 up
[root@inode4 ~]# route add -host 172.18.67.66 dev lo:0
[root@inode4 ~]# route add -host 172.18.67.88 dev lo:1


節點二：

[root@inode5 ~]# ifconfig lo:0 172.18.67.88 netmask 255.255.255.255 broadcast 172.18.67.88 up
[root@inode5 ~]# ifconfig lo:1 172.18.67.66 netmask 255.255.255.255 broadcast 172.18.67.66 up
[root@inode5 ~]# route add -host 172.18.67.88 dev lo:0
[root@inode5 ~]# route add -host 172.18.67.66 dev lo:1

 

五 測試

1.啓動服務

RS:

[root@inode4 ~]# systemctl start httpd
[root@inode5 ~]# systemctl start httpd

 

節點一：

[root@inode2 ~]# systemctl start keepalived.service
[root@inode2 ~]# systemctl status -l  keepalived.service
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2017-05-14 01:19:27 CST; 17s ago
  Process: 2120 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 2121 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─2121 /usr/sbin/keepalived -D
           ├─2122 /usr/sbin/keepalived -D
           └─2123 /usr/sbin/keepalived -D
May 14 01:19:29 inode2 Keepalived_vrrp[2123]: Opening script file /etc/keepalived/notify.sh
May 14 01:19:29 inode2 Keepalived_healthcheckers[2122]: Netlink reflector reports IP 172.18.67.66 added
May 14 01:19:31 inode2 Keepalived_vrrp[2123]: VRRP_Instance(myr2) Transition to MASTER STATE
May 14 01:19:32 inode2 Keepalived_vrrp[2123]: VRRP_Instance(myr2) Entering MASTER STATE
May 14 01:19:32 inode2 Keepalived_vrrp[2123]: VRRP_Instance(myr2) setting protocol VIPs.
May 14 01:19:32 inode2 Keepalived_vrrp[2123]: VRRP_Instance(myr2) Sending gratuitous ARPs on eno16777736 for 172.18.67.88
May 14 01:19:32 inode2 Keepalived_vrrp[2123]: Opening script file /etc/keepalived/notify.sh
May 14 01:19:32 inode2 Keepalived_healthcheckers[2122]: Netlink reflector reports IP 172.18.67.88 added
May 14 01:19:34 inode2 Keepalived_vrrp[2123]: VRRP_Instance(myr1) Sending gratuitous ARPs on eno16777736 for 172.18.67.66
May 14 01:19:37 inode2 Keepalived_vrrp[2123]: VRRP_Instance(myr2) Sending gratuitous ARPs on eno16777736 for 172.18.67.88
[root@inode2 ~]# ip a l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:8b:08:6f brd ff:ff:ff:ff:ff:ff
    inet 172.18.67.13/16 brd 172.18.255.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet 172.18.67.66/16 scope global secondary eno16777736:0
       valid_lft forever preferred_lft forever
    inet 172.18.67.88/16 scope global secondary eno16777736:1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe8b:86f/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever

 

節點二：

[root@inode3 ~]# systemctl start keepalived.service
[root@inode3 ~]# systemctl status -l  keepalived.service
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2017-05-14 01:20:25 CST; 6s ago
  Process: 2110 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 2111 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─2111 /usr/sbin/keepalived -D
           ├─2112 /usr/sbin/keepalived -D
           └─2113 /usr/sbin/keepalived -D
May 14 01:20:25 inode3 Keepalived_vrrp[2113]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
May 14 01:20:26 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) Transition to MASTER STATE
May 14 01:20:26 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) Received lower prio advert, forcing new election
May 14 01:20:26 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) Received lower prio advert, forcing new election
May 14 01:20:26 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) Received lower prio advert, forcing new election
May 14 01:20:27 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) Entering MASTER STATE
May 14 01:20:27 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) setting protocol VIPs.
May 14 01:20:27 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) Sending gratuitous ARPs on eno16777736 for 172.18.67.88
May 14 01:20:27 inode3 Keepalived_vrrp[2113]: Opening script file /etc/keepalived/notify.sh
May 14 01:20:27 inode3 Keepalived_healthcheckers[2112]: Netlink reflector reports IP 172.18.67.88 added
[root@inode3 ~]# ip a l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:78:24:c3 brd ff:ff:ff:ff:ff:ff
    inet 172.18.67.14/16 brd 172.18.255.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet 172.18.67.88/16 scope global secondary eno16777736:1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe78:24c3/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever

 

2.訪問測試

[root@inode1 ~]# curl http://172.18.67.66
RS2:172.18.67.12
[root@inode1 ~]# curl http://172.18.67.66
RS1:172.18.67.11
[root@inode1 ~]# curl http://172.18.67.66
RS2:172.18.67.12
[root@inode1 ~]# curl http://172.18.67.66
RS1:172.18.67.11
[root@inode1 ~]# curl http://172.18.67.88
RS2:172.18.67.12
[root@inode1 ~]# curl http://172.18.67.88
RS1:172.18.67.11
[root@inode1 ~]# curl http://172.18.67.88
RS2:172.18.67.12
[root@inode1 ~]# curl http://172.18.67.88
RS1:172.18.67.11


高可用負載集羣成功搭建完成。

3.模擬一臺web服務器宕機
將RS1的httpd服務停掉，並再此訪問： 

[root@inode4:~]# systemctl stop httpd
[root@inode1 ~]# curl http://172.18.67.66
RS2:172.18.67.12
[root@inode1 ~]# curl http://172.18.67.66
RS2:172.18.67.12
[root@inode1 ~]# curl http://172.18.67.88
RS2:172.18.67.12
[root@inode1 ~]# curl http://172.18.67.88
RS2:172.18.67.12


發現照樣能夠訪問服務器，實際環境中兩臺RS的內容應該是如出一轍的，在這裏我爲了以示區別將兩臺內容編輯成不一樣。

4.模擬一臺高可用負載集羣宕機
將VS1的keepalived服務關閉並測試：

[root@inode2 ~]# systemctl stop keepalived.service
[root@inode1 ~]# curl http://172.18.67.88
RS1:172.18.67.11
[root@inode1 ~]# curl http://172.18.67.88
RS2:172.18.67.12
[root@inode1 ~]# curl http://172.18.67.88
RS1:172.18.67.11
[root@inode1 ~]# curl http://172.18.67.88
RS2:172.18.67.12

 

咱們發現即便高可用負載均衡集羣中的某一個主機宕機了，咱們任然能夠經過其中的一個IP訪問web服務器，體現出了高可用的實用性，而且在訪問中lvs調度器將客戶端請求按設置的權重分別向後端的服務器實現調度。

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程