keepalived+lvs

今天這裏來實現keepalived加lvs的dr模式，實現高可用。

keepalived用來管理lvs。

環境：

主機名	IP	系統	角色
tiandong63	192.168.199.3	rhel6.五、ipvsadm、keepalived	lvs server1（MASTER）
tiandong64	192.168.199.4	rhel6.五、ipvsadm、keepalived	lvs server2（BACKUP）
tiandong65	rip：192.168.199.5 DG：192.168.199.1 vip：192.168.199.111	rhel7.4	realserver1
tiandong66	rip：192.168.199.6 DG：192.168.199.1 vip：192.168.199.111	rhel7.3	realserver2

實戰：

lvs server配置

[root@tiandong63 ~]# yum install ipvsadm keepalived -y
[root@tiandong64 ~]# yum install ipvsadm keepalived -y

[root@tiandong63 ~]# /etc/init.d/keepalived start
[root@tiandong64 ~]# /etc/init.d/keepalived start
[root@tiandong63 ~]# rpm -ql keepalived    #查看keepalived的安裝路徑
[root@tiandong63 ~]# vim /etc/keepalived/keepalived.conf
1 ! Configuration File for keepalived
  2
  3 global_defs {     #全局定義
  4    notification_email {    #定義郵件
  5    root@localhost        #定義郵件地址
  6    }
  7    notification_email_from root@localhost     #定義郵件地址
  8    smtp_server localhost                   #郵件服務器
  9    smtp_connect_timeout 30            #郵件超時時間
 10    router_id tiandong63   #router_id能夠本身定義，可是必須惟一
 11 }
 12
 13 vrrp_instance apache {      #定義vrr組
 14     state MASTER      #vrrp實例的角色，MASTER必須大寫
 15     interface eth0         #對外訪問的網絡接口，和本身的一致
 16     virtual_router_id 51    #虛擬路由器id必須和從的一致
 17     priority 100              #主從優先級，主的要高於從
 18     advert_int 1    #廣播週期秒數
 19     authentication {
 20         auth_type PASS
 21         auth_pass 1111
 22     }
 23     virtual_ipaddress {
 24         192.168.199.111    #vip地址，真實環境這裏應該是公網ip
 25     }
 26 }
 27
 28 virtual_server 192.168.199.111 80 {      #:虛擬VIP地址 與 端口，DR架構WEB端口要和虛擬端口監聽一致。不然將沒法訪問
 29     delay_loop 6                    #健康檢查時間間隔，單位是秒
 30     lb_algo rr        #lvs算法
 31     lb_kind DR     #lvs的模式
 32     nat_mask 255.255.255.0
 33     protocol TCP       #使用TCP協議
 34     real_server 192.168.199.5 80 {       #真實的ip

35         weight 1
 36         TCP_CHECK {
 37                 connect_timeout 10     #鏈接超時時間
 38                 nb_get_retry 3
 39                 delay_before_retry 3
 40                 connect_port 80   #鏈接端口爲80，要和上面的保持一致
 41         }
 42     }
 43     real_server 192.168.199.6 80 {    #真實的ip
 44         weight 1
 45         TCP_CHECK {
 46                 connect_timeout 10
 47                 nb_get_retry 3
 48                 delay_before_retry 3
 49                 connect_port 80
 50         }
 51     }
 52 }
標紅的就是配置文件須要修改的地方，在從上必須修改如下幾個位置，其餘配置同樣，把配置文件拷貝到從上，而後修改：

 10    router_id tiandong64
 14     state BACKUP
 17     priority 90

realserver配置（1和2上面都得配置，直接執行腳本就能夠了。）

 

[root@tiandong65 ~]# more lvsdr.sh
#!/bin/bash
VIP=192.168.199.111
source /etc/init.d/functions
case $1 in
start)
    echo 'start LVS of RealServer DR'
    /sbin/ifconfig lo:1 $VIP broadcast $VIP netmask 255.255.255.255 up
    /sbin/route add -host $VIP dev lo:1
    echo '1' > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo '2' > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo '1' > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo '2' > /proc/sys/net/ipv4/conf/all/arp_announce
    ;;
stop)
    /sbin/ifconfig lo:1 down
    echo 'Close LVS of RealServer DR'
    echo '0' > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo '0' > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo '0' > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo '0' > /proc/sys/net/ipv4/conf/all/arp_announce
    ;;
*)
    echo "Usage:$0 (start|stop)"
exit 1
esac

[root@tiandong65 ~]# ./lvsdr.sh start

測試：

此時lvs server的主爲192.168.199.3，192.168.199.4是閒置的

能夠在從上面測試：

[root@tiandong64 ~]# curl 192.168.199.111
this is 192.168.199.6
[root@tiandong64 ~]# curl 192.168.199.111
this is 192.168.199.5
[root@tiandong64 ~]# curl 192.168.199.111
this is 192.168.199.6
[root@tiandong64 ~]# curl 192.168.199.111
this is 192.168.199.5

在主上查看鏈接狀態：
[root@tiandong63 ~]# ipvsadm -ln --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
TCP  192.168.199.111:80                  4       23        0     1908        0
  -> 192.168.199.5:80                    2       11        0      928        0
  -> 192.168.199.6:80                    2       12        0      980        0
測試lvs server是否會負載：

停了主（192.168.199.3）上面的keepalived，看一下從（192.168.199.4）上面的是否會開啓，是否會正常轉發：

[root@tiandong63 ~]# /etc/init.d/keepalived stop   中止主上面的keepalived
Stopping keepalived:                                       [  OK  ]
[root@tiandong64 ~]# ip a    在192.168.199.4上面查看
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:c7:20:71 brd ff:ff:ff:ff:ff:ff
    inet 192.168.199.4/24 brd 192.168.199.255 scope global eth0
    inet 192.168.199.111/32 scope global eth0      #vip已經飄過來了
    inet6 fe80::20c:29ff:fec7:2071/64 scope link
       valid_lft forever preferred_lft forever

在192.168.199.3上面測試：（此時該主機爲從）

[root@tiandong63 ~]# curl 192.168.199.111
this is 192.168.199.5
[root@tiandong63 ~]# curl 192.168.199.111
this is 192.168.199.6
[root@tiandong63 ~]# curl 192.168.199.111
this is 192.168.199.5
[root@tiandong63 ~]# curl 192.168.199.111
this is 192.168.199.6

在192.168.199.4上面查看鏈接狀態：
[root@tiandong64 ~]# ipvsadm -ln --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
TCP  192.168.199.111:80                  4       22        0     1848        0
  -> 192.168.199.5:80                    2       10        0      872        0
  -> 192.168.199.6:80                    2       12        0      976        0

當主上的keepalived恢復的話看一下狀態：

當主恢復了以後，vip有飄到了192.168.199.3上面，由於優先級比較高。

[root@tiandong63 ~]# /etc/init.d/keepalived start
[root@tiandong63 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:38:0b:14 brd ff:ff:ff:ff:ff:ff
    inet 192.168.199.3/24 brd 192.168.199.255 scope global eth0
    inet 192.168.199.111/32 scope global eth0
    inet6 fe80::20c:29ff:fe38:b14/64 scope link
       valid_lft forever preferred_lft forever

測試realserver出現故障的現象：

當一臺realserver的Apache服務出現故障時，是否會充lvs中剔除，

[root@tiandong63 ~]# ipvsadm -ln     正常狀況下的狀態。
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.199.111:80 rr
  -> 192.168.199.5:80             Route   1      0          0         
  -> 192.168.199.6:80             Route   1      0          0

當有一臺realserver出現故障：

[root@tiandong66 ~]# systemctl stop httpd    一臺服務器的Apache服務故障了

[root@tiandong63 ~]# ipvsadm -ln      此時查看只有一臺realsever了。
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.199.111:80 rr
  -> 192.168.199.5:80             Route   1      0          0

能夠查看lvs server上的日誌：

[root@tiandong63 ~]# tail -f /var/log/messages     把故障的主機移除了。
Dec 16 12:52:50 tiandong63 Keepalived_vrrp[3571]: VRRP_Instance(apache) Received lower prio advert, forcing new election
Dec 16 12:52:51 tiandong63 Keepalived_vrrp[3571]: VRRP_Instance(apache) Entering MASTER STATE
Dec 16 12:52:51 tiandong63 Keepalived_vrrp[3571]: VRRP_Instance(apache) setting protocol VIPs.
Dec 16 12:52:51 tiandong63 Keepalived_vrrp[3571]: VRRP_Instance(apache) Sending gratuitous ARPs on eth0 for 192.168.199.111
Dec 16 12:52:51 tiandong63 Keepalived_healthcheckers[3570]: Netlink reflector reports IP 192.168.199.111 added
Dec 16 12:52:56 tiandong63 Keepalived_vrrp[3571]: VRRP_Instance(apache) Sending gratuitous ARPs on eth0 for 192.168.199.111
Dec 16 12:58:35 tiandong63 Keepalived_healthcheckers[3570]: TCP connection to [192.168.199.6]:80 failed !!!
Dec 16 12:58:35 tiandong63 Keepalived_healthcheckers[3570]: Removing service [192.168.199.6]:80 from VS [192.168.199.111]:80
Dec 16 12:58:35 tiandong63 Keepalived_healthcheckers[3570]: Remote SMTP server [0.0.0.0]:25 connected.
Dec 16 12:58:35 tiandong63 Keepalived_healthcheckers[3570]: SMTP alert successfully sent.

當real server恢復以後：

[root@tiandong66 ~]# systemctl start httpd   恢復Apache服務
[root@tiandong63 ~]# tail -f /var/log/messages   查看日誌，把機器加入到lvs中了
Dec 16 12:58:35 tiandong63 Keepalived_healthcheckers[3570]: SMTP alert successfully sent.
Dec 16 13:01:35 tiandong63 Keepalived_healthcheckers[3570]: TCP connection to [192.168.199.6]:80 success.
Dec 16 13:01:35 tiandong63 Keepalived_healthcheckers[3570]: Adding service [192.168.199.6]:80 to VS [192.168.199.111]:80
Dec 16 13:01:35 tiandong63 Keepalived_healthcheckers[3570]: Remote SMTP server [0.0.0.0]:25 connected.
Dec 16 13:01:35 tiandong63 Keepalived_healthcheckers[3570]: SMTP alert successfully sent.
^C
[root@tiandong63 ~]# ipvsadm -ln     #查看
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.199.111:80 rr
  -> 192.168.199.5:80             Route   1      0          0         
  -> 192.168.199.6:80             Route   1      0          0 

OK了，有什麼問題隨時歡迎討論指教！！！！！

QQ：1127000383

192.168.199.7