雙主Keepalive與lvs實現自動監控主機狀態並實時切換

時間 2019-11-06

標籤 keepalive lvs 實現自動監控主機狀態實時切換欄目負載均衡简体版

原文原文鏈接

環境：

202.106.0.17: as Client
202.106.0.27: as router
192.168.205.37: as lvs1
192.168.205.47: as lvs2
192.168.205.57: as websrv1
192.168.205.67: as websrv2
192.168.205.77: as websrv3
192.168.205.87: as websrv4
注：全部操做系統默認停用firewalld,iptable爲空,關閉selinux

版本：

OS: centos 7 1810 with mini install
keepalived
httpd

目地：

使用keepalived 監控四臺web服務器，兩臺爲一組，各使用一個虛擬IP地址，lvs1爲vip1的主vip2的從，lvs2爲vip2的主，vip1的從，互相主備, 並使用lvs作爲負載均衡，並能自動的發現服務down機並自動移除故障主機，從而實現無端障調度。 html

配置router

開啓路由轉發功能

[root@router data]#ech 1 > /proc/sys/net/ipv4/ip_forward

增長一個10.1.1.1在router的eth1上, 因爲我要到lvs1和lvs2服務器時必須有路由，因爲是兩個主機的虛擬Ip是頗有可能飄移，因此當其中一臺down時沒法肯定在哪臺主機上，爲了安全不加路由，直接用IP直連。
```
[root@router data]#ip a a 10.1.1.1/24 dev eth0:1
```
先配置四臺real server

在app1的兩臺服務器192.168.205.57/67上運行以下腳本linux

[root@websrv1 data]#cat lvs_dr_rs.sh 
#!/bin/bash
vip=10.1.1.100
gateway=192.168.205.27
mask='24'
dev=lo:1
rpm -q httpd &> /dev/null || yum -y install httpd &>/dev/null
service httpd start &> /dev/null && echo "The httpd Server is Ready!"
echo "this is `hostname`" > /var/www/html/index.html

case $1 in
start)
    echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
    ip a a $vip/$mask dev $dev #broadcast $vip up
    ip route add default via $gateway dev eth0
    #route add -host $vip dev $dev
    echo "The RS Server is Ready!"
    ;;
stop)
    systemctl restart network
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo "The RS Server is Canceled!"
    ;;
*) 
    echo "Usage: $(basename $0) start|stop"
    exit 1
    ;;
esac

在app2的兩臺服務器192.168.205.77/87上運行以下腳本web

[root@websrv3 data]#cat lvs_dr_rs.sh 
#!/bin/bash
vip=10.1.1.200
gateway=192.168.205.27
mask='24'
dev=lo:1
rpm -q httpd &> /dev/null || yum -y install httpd &>/dev/null
service httpd start &> /dev/null && echo "The httpd Server is Ready!"
echo "this is `hostname`" > /var/www/html/index.html

case $1 in
start)
    echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
    ip address add $vip/$mask dev $dev #broadcast $vip up
    ip route add default via $gateway dev eth0
    #route add -host $vip dev $dev
    echo "The RS Server is Ready!"
    ;;
stop)
    systemctl restart network
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo "The RS Server is Canceled!"
    ;;
*) 
    echo "Usage: $(basename $0) start|stop"
    exit 1
    ;;
esac

在兩臺lvs上作一樣的動做（除特別說明）

安裝httpd服務，將sorry server放在本地的兩臺主機中

yum install httpd
echo server under maintenance > /var/www/html/index.html

必定要加默認路由到router,若是加正確能夠工做，但sorry server不會正常
```
ip route add default via 192.168.205.27 dev eth0
```
爲了訪問方便最好是將兩個lvs服務器之間作ssh key驗證，用下面的方法就不用在47上再作一次了，真接實現互認證
```
ssh-keygen
ssh-copy-id 127.0.0.1
scp -r /root/.ssh 192.168.205.47:/root
```

最好將host文件加入兩臺主機名稱解析

vi /etc/hosts
192.168.205.37 websrv1
192.168.205.47 websrv2
scp /etc/hosts 192.168.205.47:/etc

安裝keepalive, 爲了能看清如何加的lvs策略，咱們把iplvadm也裝上
```
yum install keepalived ipvsadm
```

在lvs1上修改配置文件

[root@lsv1 ~]#vi /etc/keepalived/keepalived.conf 
! Configuration File for keepalived
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalive@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS1
   vrrp_mcast_group4 224.0.0.100                                                                          
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 37
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass centos
    }
    virtual_ipaddress {
        10.1.1.100/24 dev eth0 label eth0:0
    }
}
vrrp_instance VI_2 {
    state BACKUP
    interface eth0
    virtual_router_id 47
    priority 80
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass centos
    }
    virtual_ipaddress {
        10.1.1.200/24 dev eth0 label eth0:1
    }
}
virtual_server 10.1.1.100 80 {
    delay_loop 6
    lb_algo rr 
    lb_kind DR
    protocol TCP
    sorry_server 127.0.0.1 80
    real_server 192.168.205.57 80 {
        weight 1
        HTTP_GET {
            url { 
              path /
              status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
    real_server 192.168.205.67 80 {
        weight 1
        HTTP_GET {
            url { 
              path /
              status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }                            
}
virtual_server 10.1.1.200 80 {
    delay_loop 6
    lb_algo rr 
    lb_kind DR
    protocol TCP
    sorry_server 127.0.0.1 80
    real_server 192.168.205.77 80 {
        weight 1
        HTTP_GET {
            url { 
              path /
              status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
    real_server 192.168.205.87 80 {
        weight 1
        HTTP_GET {
            url { 
              path /
              status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
}            
11. 爲了方便將lsv1的keepalive.conf複製到lvs2上，並進行修改
[root@lvs2 ~]#vi /etc/keepalived/keepalived.conf 
! Configuration File for keepalived
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalive@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS2
         vrrp_mcast_group4 224.0.0.100
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 37
    priority 80
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass centos
    }
    virtual_ipaddress {
        10.1.1.100/24 dev eth0 label eth0:0
    }
}
vrrp_instance VI_2 {
    state MASTER
    interface eth0
    virtual_router_id 47
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass centos
    }
    virtual_ipaddress {
        10.1.1.200/24 dev eth0 label eth0:1
    }
}
virtual_server 10.1.1.100 80 {
    delay_loop 6
    lb_algo rr 
    lb_kind DR
    protocol TCP
    sorry_server 127.0.0.1 80
    real_server 192.168.205.57 80 {
        weight 1
        HTTP_GET {
            url { 
              path /
              status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
    real_server 192.168.205.67 80 {
        weight 1
        HTTP_GET {
            url { 
              path /
              status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
}
virtual_server 10.1.1.200 80 {
    delay_loop 6
    lb_algo rr 
    lb_kind DR
    protocol TCP
    sorry_server 127.0.0.1 80
    real_server 192.168.205.77 80 {
        weight 1
        HTTP_GET {
            url { 
              path /
              status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
    real_server 192.168.205.87 80 {
        weight 1
        HTTP_GET {
            url { 
              path /
              status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
}

起動keepalived服務
```
systemctl start keepalived
```

在lvs1看到狀態爲RR調度

[root@lsv1 ~]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.1.1.100:80 rr
  -> 192.168.205.57:80            Route   1      0          0         
  -> 192.168.205.67:80            Route   1      0          0         
TCP  10.1.1.200:80 rr
  -> 192.168.205.77:80            Route   1      0          0         
  -> 192.168.205.87:80            Route   1      0          0

在lvs1中看到只一10.1.1.100IP，在Lvs2中會看到只有10.1.1.200IP

[root@lsv1 ~]#ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:56:e1:ea brd ff:ff:ff:ff:ff:ff
    inet 192.168.205.37/24 brd 192.168.205.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 10.1.1.100/24 scope global eth0:0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe56:e1ea/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@lvs2 ~]#ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:37:f9:93 brd ff:ff:ff:ff:ff:ff
    inet 192.168.205.47/24 brd 192.168.205.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 10.1.1.200/24 scope global eth0:1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe37:f993/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

測試

在client上運行一個循環進行測試

[root@client ~]#while : ;do curl 10.1.1.100; sleep 0.5; done
this is websrv1
this is websrv2
this is websrv1
this is websrv2
this is websrv1
this is websrv2
[root@client ~]#while : ;do curl 10.1.1.200; sleep 0.5; done 
this is websrv4
this is websrv3
this is websrv4
this is websrv3
this is websrv4

停掉web1，再測試，發現只會調度到web2

[root@websrv1 data]#systemctl stop httpd
[root@client ~]#while : ;do curl 10.1.1.100; sleep 0.5; done
this is websrv2
this is websrv2
this is websrv2
this is websrv2
this is websrv2

停掉web2，再測試，發現sorry server帶替工做，並在lvs1上能夠看到127.0.0.1加載

[root@websrv2 ~]#systemctl stop httpd
[root@client ~]#while : ;do curl 10.1.1.100; sleep 0.5; done
server under maintenance
server under maintenance
server under maintenance
server under maitnenance
[root@lsv1 ~]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.1.1.100:80 rr
  -> 127.0.0.1:80                 Route   1      0          4         
TCP  10.1.1.200:80 rr
  -> 192.168.205.77:80            Route   1      0          0         
  -> 192.168.205.87:80            Route   1      0          0

恢復兩個websrv1 和websrv2,並停掉lvs2, 發現沒有影響，但能夠看到兩個vip所有回到lvs1上

[root@websrv1 ~]#systemctl start httpd
[root@websrv2 ~]#systemctl start httpd
[root@lvs2 ~]#systemctl stop keepalived
[root@client ~]#while : ;do curl 10.1.1.100; sleep 0.5; done
this is websrv2
this is websrv1
this is websrv2
this is websrv1
[root@client ~]#while : ;do curl 10.1.1.200; sleep 0.5; done
this is websrv3
this is websrv4
this is websrv3
this is websrv4
[root@lsv1 ~]#ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:56:e1:ea brd ff:ff:ff:ff:ff:ff
    inet 192.168.205.37/24 brd 192.168.205.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 10.1.1.100/24 scope global eth0:0
       valid_lft forever preferred_lft forever
    inet 10.1.1.200/24 scope global secondary eth0:1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe56:e1ea/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

將lvs2恢復狀態，由於有搶佔功能又回到原來的主和備份上

[root@client ~]#while : ;do curl 10.1.1.100; sleep 0.5; done
this is websrv2
this is websrv1
this is websrv2
[root@client ~]#while : ;do curl 10.1.1.200; sleep 0.5; done
this is websrv4
this is websrv3
this is websrv4
[root@lsv1 ~]#ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:56:e1:ea brd ff:ff:ff:ff:ff:ff
    inet 192.168.205.37/24 brd 192.168.205.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 10.1.1.100/24 scope global eth0:0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe56:e1ea/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@lvs2 ~]#ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:37:f9:93 brd ff:ff:ff:ff:ff:ff
    inet 192.168.205.47/24 brd 192.168.205.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 10.1.1.200/24 scope global eth0:1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe37:f993/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever