keepalived+nginx實現HA高可用的web負載均衡

Keepalived 是一種高性能的服務器高可用或熱備解決方案, Keepalived 能夠用來防止服務器單點故障的發生,經過配合 Nginx 能夠實現 web 前端服務的高可用。
Keepalived 以 VRRP 協議爲實現基礎,用 VRRP 協議來實現高可用性(HA)。 VRRP(Virtual RouterRedundancy Protocol)協議是用於實現路由器冗餘的協議, VRRP 協議將兩臺或多臺路由器設備虛擬成一個設備,對外提供虛擬路由器 IP(一個或多個),而在路由器組內部,若是實際擁有這個對外 IP 的路由器若是工做正常的話就是 MASTER,或者是經過算法選舉產生, MASTER 實現針對虛擬路由器 IP 的各類網絡功能,如 ARP 請求, ICMP,以及數據的轉發等;其餘設備不擁有該虛擬 IP,狀態是 BACKUP,除了接收 MASTER 的VRRP 狀態通告信息外,不執行對外的網絡功能。當主機失效時, BACKUP 將接管原先 MASTER 的網絡功能。VRRP 協議使用多播數據來傳輸 VRRP 數據, VRRP 數據使用特殊的虛擬源 MAC 地址發送數據而不是自身網卡的 MAC 地址, VRRP 運行時只有 MASTER 路由器定時發送 VRRP 通告信息,表示 MASTER 工做正常以及虛擬路由器 IP(組), BACKUP 只接收 VRRP 數據,不發送數據,若是必定時間內沒有接收到 MASTER 的通告信息,各 BACKUP 將宣告本身成爲 MASTER,發送通告信息,從新進行 MASTER 選舉狀態。

html

ip規劃以下:定義VIP爲:172.16.23.132前端

nginx1:172.16.23.129       keepalived:172.16.23.129node

nginx2:172.16.23.130       keepalived:172.16.23.130nginx

httpd1:172.16.23.128git

httpd2:172.16.23.131web

上面規劃中nginx只提供負載均衡做用,並不實現web訪問功能:算法

[root@master ~]# cat /etc/ansible/hosts|grep "^\[nodes" -A 2
[nodes]
172.16.23.129
172.16.23.130

 查看nginx服務狀態:shell

[root@master ~]# ansible nodes -m shell -a "systemctl status nginx"|grep running
   Active: active (running) since 二 2018-12-18 16:33:04 CST; 12min ago
   Active: active (running) since 二 2018-12-18 16:35:51 CST; 10min ago

 首先nginx服務正常開啓,而後查看後端服務httpd:json

[root@master ~]# cat /etc/ansible/hosts|grep "^\[backend_nodes" -A 2
[backend_nodes]
172.16.23.128
172.16.23.131

 查看httpd服務狀態:後端

[root@master ~]# ansible backend_nodes -m shell -a "systemctl status httpd"|grep running
   Active: active (running) since 二 2018-12-18 16:29:36 CST; 22min ago
   Active: active (running) since 二 2018-12-18 16:30:03 CST; 21min ago

 而後在nginx兩臺服務器上分別測試負載均衡效果:

[root@master ~]# ansible 172.16.23.129 -m get_url -a "url=http://172.16.23.129/index.html dest=/tmp"|grep status_code
    "status_code": 200, 
[root@master ~]# ansible 172.16.23.129 -m shell -a "cat /tmp/index.html"
172.16.23.129 | CHANGED | rc=0 >>
172.16.23.128

[root@master ~]# ansible 172.16.23.129 -m get_url -a "url=http://172.16.23.129/index.html dest=/tmp"|grep status_code
    "status_code": 200, 
[root@master ~]# ansible 172.16.23.129 -m shell -a "cat /tmp/index.html"
172.16.23.129 | CHANGED | rc=0 >>
172.16.23.131

 由上面能夠看出nginx1:172.16.23.129上進行測試返回後端httpd服務的web頁面:172.16.23.128以及172.16.23.131,測試訪問沒有問題,負載均衡沒有問題

[root@master ~]# ansible 172.16.23.130 -m get_url -a "url=http://172.16.23.130/index.html dest=/tmp"|grep status_code
    "status_code": 200, 
[root@master ~]# ansible 172.16.23.130 -m shell -a "cat /tmp/index.html"
172.16.23.130 | CHANGED | rc=0 >>
172.16.23.128

[root@master ~]# ansible 172.16.23.130 -m get_url -a "url=http://172.16.23.130/index.html dest=/tmp"|grep status_code
    "status_code": 200, 
[root@master ~]# ansible 172.16.23.130 -m shell -a "cat /tmp/index.html"
172.16.23.130 | CHANGED | rc=0 >>
172.16.23.131

 由上面能夠看見nginx2服務訪問後端httpd服務也是徹底OK的,因而nginx兩臺服務負載均衡效果達到,如今在nginx兩臺服務器上安裝keepalived服務:

[root@master ~]# ansible nodes -m shell -a "systemctl status keepalived"|grep running
   Active: active (running) since 二 2018-12-18 16:06:38 CST; 52min ago
   Active: active (running) since 二 2018-12-18 16:05:04 CST; 54min ago

 查看VIP信息:發現vip在node1節點上

[root@master ~]# ansible nodes -m shell -a "hostname;ip a|grep ens33|grep -Po '(?<=inet ).*(?=\/)'"
172.16.23.129 | CHANGED | rc=0 >>
node1
172.16.23.129
172.16.23.132

172.16.23.130 | CHANGED | rc=0 >>
node2
172.16.23.130

 能夠看出VIP落在了nginx1也就是node1節點上,而後經過訪問vip看看負載均衡效果:

[root@master ~]# curl http://172.16.23.132
172.16.23.131
[root@master ~]# curl http://172.16.23.132
172.16.23.128

 由上面返回結果看,沒有任何問題,如今摘掉一臺nginx服務器,看看keepalived狀況,以及訪問vip的狀況:

[root@master ~]# ansible 172.16.23.130 -m shell -a "systemctl stop nginx"
172.16.23.130 | CHANGED | rc=0 >>

 查看keepalived服務狀態,查看vip信息:

[root@master ~]# ansible nodes -m shell -a "systemctl status keepalived"|grep running
   Active: active (running) since 二 2018-12-18 16:05:04 CST; 1h 4min ago
   Active: active (running) since 二 2018-12-18 16:06:38 CST; 1h 3min ago

[root@master ~]# ansible nodes -m shell -a "hostname;ip a|grep ens33|grep -Po '(?<=inet ).*(?=\/)'"
172.16.23.130 | CHANGED | rc=0 >>
node2
172.16.23.130

172.16.23.129 | CHANGED | rc=0 >>
node1
172.16.23.129
172.16.23.132

 vip信息沒有漂移,keepalived服務狀態正常,如今訪問vip:

[root@master ~]# curl http://172.16.23.132
172.16.23.128
[root@master ~]# curl http://172.16.23.132
172.16.23.131

 經過vip訪問web服務沒有問題

 

如今將nginx服務開啓,端掉一個節點的keepalived服務:

[root@master ~]# ansible 172.16.23.130 -m shell -a "systemctl start nginx"
172.16.23.130 | CHANGED | rc=0 >>

[root@master ~]# ansible nodes -m shell -a "systemctl status nginx"|grep running
   Active: active (running) since 二 2018-12-18 17:15:48 CST; 18s ago
   Active: active (running) since 二 2018-12-18 16:33:04 CST; 43min ago

 

[root@master ~]# ansible 172.16.23.130 -m shell -a "systemctl stop keepalived"
172.16.23.130 | CHANGED | rc=0 >>

 而後在該節點日誌查看以下:tail -f /var/log/message

Dec 18 17:16:50 node2 systemd: Stopping LVS and VRRP High Availability Monitor...
Dec 18 17:16:50 node2 Keepalived[12981]: Stopping
Dec 18 17:16:50 node2 Keepalived_healthcheckers[12982]: Stopped
Dec 18 17:16:51 node2 Keepalived_vrrp[12983]: Stopped
Dec 18 17:16:51 node2 Keepalived[12981]: Stopped Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Dec 18 17:16:52 node2 systemd: Stopped LVS and VRRP High Availability Monitor.

 

[root@master ~]# ansible nodes -m shell -a "systemctl status keepalived"|grep running
   Active: active (running) since 二 2018-12-18 16:06:38 CST; 1h 10min ago


[root@master ~]# ansible nodes -m shell -a "hostname;ip a|grep ens33|grep -Po '(?<=inet ).*(?=\/)'"
172.16.23.130 | CHANGED | rc=0 >>
node2
172.16.23.130

172.16.23.129 | CHANGED | rc=0 >>
node1
172.16.23.129
172.16.23.132

 因爲斷掉的是nginx2也就是node2節點的keepalived服務,因此vip仍是在node1上,並無漂移在node2,查看node1和node2節點上keepalived服務的配置文件:

[root@master ~]# ansible nodes -m shell -a "cat /etc/keepalived/keepalived.conf"
172.16.23.129 | CHANGED | rc=0 >>
! Configuration File for keepalived

global_defs {
   notification_email {
       346165580@qq.com
   }
   notification_email_from json_hc@163.com
   smtp_server smtp.163.com
   smtp_connect_timeout 30
   router_id test
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 51
    priority 100
    nopreempt           # 非搶佔模式
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.16.23.132/24 dev ens33
    }
}

172.16.23.130 | CHANGED | rc=0 >>
! Configuration File for keepalived

global_defs {
   notification_email {
       346165580@qq.com
   }
   notification_email_from json_hc@163.com
   smtp_server smtp.163.com
   smtp_connect_timeout 30
   router_id test
}

vrrp_instance VI_1 {
    state BACKUP 
    interface ens33
    virtual_router_id 51
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.16.23.132/24 dev ens33 
    }
}

 能夠由配置看出,只有優先級不同以及node1節點設置了nopreempt # 非搶佔模式,如今將node2節點的keepalived服務開啓,而後將node1節點的keepalived服務關掉,看看vip信息:

[root@master ~]# ansible 172.16.23.130 -m shell -a "systemctl start keepalived"
172.16.23.130 | CHANGED | rc=0 >>

 查看node2日誌:

Dec 18 17:23:14 node2 systemd: Starting LVS and VRRP High Availability Monitor...
Dec 18 17:23:14 node2 Keepalived[15994]: Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Dec 18 17:23:14 node2 Keepalived[15994]: Opening file '/etc/keepalived/keepalived.conf'.
Dec 18 17:23:14 node2 Keepalived[15995]: Starting Healthcheck child process, pid=15996
Dec 18 17:23:14 node2 Keepalived_healthcheckers[15996]: Opening file '/etc/keepalived/keepalived.conf'.
Dec 18 17:23:14 node2 Keepalived[15995]: Starting VRRP child process, pid=15997
Dec 18 17:23:14 node2 systemd: Started LVS and VRRP High Availability Monitor.
Dec 18 17:23:14 node2 Keepalived_vrrp[15997]: Registering Kernel netlink reflector
Dec 18 17:23:14 node2 Keepalived_vrrp[15997]: Registering Kernel netlink command channel
Dec 18 17:23:14 node2 Keepalived_vrrp[15997]: Registering gratuitous ARP shared channel
Dec 18 17:23:14 node2 Keepalived_vrrp[15997]: Opening file '/etc/keepalived/keepalived.conf'.
Dec 18 17:23:24 node2 Keepalived_vrrp[15997]: VRRP_Instance(VI_1) removing protocol VIPs.
Dec 18 17:23:24 node2 Keepalived_vrrp[15997]: Using LinkWatch kernel netlink reflector...
Dec 18 17:23:24 node2 Keepalived_vrrp[15997]: VRRP_Instance(VI_1) Entering BACKUP STATE
Dec 18 17:23:24 node2 Keepalived_vrrp[15997]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]

 兩節點keepalived服務狀態,以及vip信息:

[root@master ~]# ansible nodes -m shell -a "systemctl status keepalived"|grep running
   Active: active (running) since 二 2018-12-18 17:23:14 CST; 56s ago
   Active: active (running) since 二 2018-12-18 16:06:38 CST; 1h 17min ago

[root@master ~]# ansible nodes -m shell -a "hostname;ip a|grep ens33|grep -Po '(?<=inet ).*(?=\/)'"
172.16.23.129 | CHANGED | rc=0 >>
node1
172.16.23.129
172.16.23.132

172.16.23.130 | CHANGED | rc=0 >>
node2
172.16.23.130

 如今將node1的keepalived服務停掉看看vip信息:

[root@master ~]# ansible 172.16.23.129 -m shell -a "systemctl stop keepalived"
172.16.23.129 | CHANGED | rc=0 >>

 查看各自節點的日誌信息:

Dec 18 17:27:41 node1 systemd: Stopping LVS and VRRP High Availability Monitor...
Dec 18 17:27:41 node1 Keepalived[24483]: Stopping
Dec 18 17:27:41 node1 Keepalived_vrrp[24485]: VRRP_Instance(VI_1) sent 0 priority
Dec 18 17:27:41 node1 Keepalived_vrrp[24485]: VRRP_Instance(VI_1) removing protocol VIPs.
Dec 18 17:27:41 node1 Keepalived_healthcheckers[24484]: Stopped
Dec 18 17:27:42 node1 Keepalived_vrrp[24485]: Stopped
Dec 18 17:27:42 node1 Keepalived[24483]: Stopped Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Dec 18 17:27:42 node1 systemd: Stopped LVS and VRRP High Availability Monitor.

 

Dec 18 17:27:42 node2 Keepalived_vrrp[15997]: VRRP_Instance(VI_1) Transition to MASTER STATE
Dec 18 17:27:43 node2 Keepalived_vrrp[15997]: VRRP_Instance(VI_1) Entering MASTER STATE
Dec 18 17:27:43 node2 Keepalived_vrrp[15997]: VRRP_Instance(VI_1) setting protocol VIPs.
Dec 18 17:27:43 node2 Keepalived_vrrp[15997]: Sending gratuitous ARP on ens33 for 172.16.23.132
Dec 18 17:27:43 node2 Keepalived_vrrp[15997]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 172.16.23.132
Dec 18 17:27:43 node2 Keepalived_vrrp[15997]: Sending gratuitous ARP on ens33 for 172.16.23.132
Dec 18 17:27:43 node2 Keepalived_vrrp[15997]: Sending gratuitous ARP on ens33 for 172.16.23.132
Dec 18 17:27:43 node2 Keepalived_vrrp[15997]: Sending gratuitous ARP on ens33 for 172.16.23.132
Dec 18 17:27:43 node2 Keepalived_vrrp[15997]: Sending gratuitous ARP on ens33 for 172.16.23.132
Dec 18 17:27:48 node2 Keepalived_vrrp[15997]: Sending gratuitous ARP on ens33 for 172.16.23.132
Dec 18 17:27:48 node2 Keepalived_vrrp[15997]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 172.16.23.132
Dec 18 17:27:48 node2 Keepalived_vrrp[15997]: Sending gratuitous ARP on ens33 for 172.16.23.132
Dec 18 17:27:48 node2 Keepalived_vrrp[15997]: Sending gratuitous ARP on ens33 for 172.16.23.132
Dec 18 17:27:48 node2 Keepalived_vrrp[15997]: Sending gratuitous ARP on ens33 for 172.16.23.132
Dec 18 17:27:48 node2 Keepalived_vrrp[15997]: Sending gratuitous ARP on ens33 for 172.16.23.132

 能夠看到vip漂移的信息切換,如今查看vip信息:

[root@master ~]# ansible nodes -m shell -a "hostname;ip a|grep ens33|grep -Po '(?<=inet ).*(?=\/)'"
172.16.23.130 | CHANGED | rc=0 >>
node2
172.16.23.130
172.16.23.132

172.16.23.129 | CHANGED | rc=0 >>
node1
172.16.23.129

 由上面信息,vip確認漂移到了node2節點,如今將node1節點的keepalived服務開啓,看看vip是否會再次漂移回去到node1節點:

[root@master ~]# ansible 172.16.23.129 -m shell -a "systemctl start keepalived"
172.16.23.129 | CHANGED | rc=0 >>

 查看node1日誌:

Dec 18 17:30:18 node1 systemd: Starting LVS and VRRP High Availability Monitor...
Dec 18 17:30:18 node1 Keepalived[28009]: Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Dec 18 17:30:18 node1 Keepalived[28009]: Opening file '/etc/keepalived/keepalived.conf'.
Dec 18 17:30:18 node1 Keepalived[28010]: Starting Healthcheck child process, pid=28011
Dec 18 17:30:18 node1 Keepalived_healthcheckers[28011]: Opening file '/etc/keepalived/keepalived.conf'.
Dec 18 17:30:18 node1 Keepalived[28010]: Starting VRRP child process, pid=28012
Dec 18 17:30:18 node1 systemd: Started LVS and VRRP High Availability Monitor.
Dec 18 17:30:18 node1 Keepalived_vrrp[28012]: Registering Kernel netlink reflector
Dec 18 17:30:18 node1 Keepalived_vrrp[28012]: Registering Kernel netlink command channel
Dec 18 17:30:18 node1 Keepalived_vrrp[28012]: Registering gratuitous ARP shared channel
Dec 18 17:30:18 node1 Keepalived_vrrp[28012]: Opening file '/etc/keepalived/keepalived.conf'.
Dec 18 17:30:28 node1 Keepalived_vrrp[28012]: VRRP_Instance(VI_1) removing protocol VIPs.
Dec 18 17:30:28 node1 Keepalived_vrrp[28012]: Using LinkWatch kernel netlink reflector...
Dec 18 17:30:28 node1 Keepalived_vrrp[28012]: VRRP_Instance(VI_1) Entering BACKUP STATE
Dec 18 17:30:28 node1 Keepalived_vrrp[28012]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]

 查看node2日誌:

Dec 18 17:30:01 node2 systemd: Started Session 1328 of user root.
Dec 18 17:30:01 node2 systemd: Starting Session 1328 of user root.
Dec 18 17:30:05 node2 systemd-logind: Removed session 1327.
Dec 18 17:31:02 node2 systemd: Started Session 1329 of user root.
Dec 18 17:31:02 node2 systemd: Starting Session 1329 of user root.

 由node2日誌信息顯示vip並無作漂移切換動做,如今查看vip:

[root@master ~]# ansible nodes -m shell -a "hostname;ip a|grep ens33|grep -Po '(?<=inet ).*(?=\/)'"
172.16.23.129 | CHANGED | rc=0 >>
node1
172.16.23.129

172.16.23.130 | CHANGED | rc=0 >>
node2
172.16.23.130
172.16.23.132

 根據上面也能夠驗證到vip並無漂移回來,這正好驗證了nopreempt # 非搶佔模式的功能

根據上面操做說明:

若是不但願keepalived服務再次上線而伴隨vip再次漂移,能夠設置nopreempt # 非搶佔模式,具體配置信息參考上面的例子(只有優先級不一樣,外加上nopreempt # 非搶佔模式)

如今vip在node2節點上,若是node2節點keepalived服務再次掛掉,看看vip是否會漂移:

[root@master ~]# ansible 172.16.23.130 -m shell -a "systemctl stop keepalived"
172.16.23.130 | CHANGED | rc=0 >>

 查看node2日誌:

Dec 18 17:35:59 node2 systemd: Stopping LVS and VRRP High Availability Monitor...
Dec 18 17:35:59 node2 Keepalived[15995]: Stopping
Dec 18 17:35:59 node2 Keepalived_vrrp[15997]: VRRP_Instance(VI_1) sent 0 priority
Dec 18 17:35:59 node2 Keepalived_vrrp[15997]: VRRP_Instance(VI_1) removing protocol VIPs.
Dec 18 17:35:59 node2 Keepalived_healthcheckers[15996]: Stopped
Dec 18 17:36:00 node2 Keepalived_vrrp[15997]: Stopped
Dec 18 17:36:00 node2 Keepalived[15995]: Stopped Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Dec 18 17:36:00 node2 systemd: Stopped LVS and VRRP High Availability Monitor.

 查看node1日誌:

Dec 18 17:36:05 node1 Keepalived_vrrp[28012]: Sending gratuitous ARP on ens33 for 172.16.23.132
Dec 18 17:36:05 node1 Keepalived_vrrp[28012]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 172.16.23.132
Dec 18 17:36:05 node1 Keepalived_vrrp[28012]: Sending gratuitous ARP on ens33 for 172.16.23.132
Dec 18 17:36:05 node1 Keepalived_vrrp[28012]: Sending gratuitous ARP on ens33 for 172.16.23.132
Dec 18 17:36:05 node1 Keepalived_vrrp[28012]: Sending gratuitous ARP on ens33 for 172.16.23.132
Dec 18 17:36:05 node1 Keepalived_vrrp[28012]: Sending gratuitous ARP on ens33 for 172.16.23.132

 能夠看到vip又再次漂移到node1節點上了,這正是目前但願看到的

[root@master ~]# ansible nodes -m shell -a "hostname;ip a|grep ens33|grep -Po '(?<=inet ).*(?=\/)'"
172.16.23.129 | CHANGED | rc=0 >>
node1
172.16.23.129
172.16.23.132

172.16.23.130 | CHANGED | rc=0 >>
node2
172.16.23.130

 由上面測試得出:

當node1優先級高於node2節點,而且node1設置了nopreempt # 非搶佔模式,那麼當node1上面的keepalived服務掛掉並再次上線時,vip不會進行漂移回去,只有當node2上面的keepalived服務掛掉,vip纔會再次漂移到node1節點

 

如今測試後端提供的httpd服務:

若是後端httpd服務掛掉一個,訪問以下:

[root@master ~]# ansible 172.16.23.131 -m shell -a "systemctl stop httpd"
172.16.23.131
| CHANGED | rc=0 >> [root@master ~]# ansible 172.16.23.131 -m shell -a "systemctl status httpd" 172.16.23.131 | FAILED | rc=3 >>

 根據vip訪問以下:

[root@master ~]# curl http://172.16.23.132
172.16.23.128
[root@master ~]# curl http://172.16.23.132
172.16.23.128

 訪問沒有任何問題,如今若是將172.16.23.131這臺的httpd服務開啓進行手動測試,並不提供給vip進行訪問,當測試沒問題後再進行爲vip進行調用:

將兩臺nginx的配置以下進行修改:

    upstream webserver {
        server 172.16.23.128 weight=1;
        server 172.16.23.131 weight=1;
    }

 將server 172.16.23.131 weight=1;這一行進行摘掉,由於nginx兩臺,因此一臺一臺來處理,確保應用不會中斷(在172.16.23.131上線以前操做)

因爲兩臺nginx都只負載到了172.16.23.128上面,因此當172.16.23.131上線了也不會被調度到,若是要將172.16.23.131做爲服務提供,那麼再將nginx一臺一臺進行增長後端節點就行

相關文章
相關標籤/搜索