keepalived的介紹及配置高可用集羣

12月19日任務

18.1 集羣介紹

18.2 keepalived介紹

18.3/18.4/18.5 用keepalived配置高可用集羣

 

集羣介紹

根據功能劃分爲2類：高可用和負載均衡

高可用集羣：一般爲兩臺服務器，一臺工做，另一臺做爲冗餘。當提供服務的機器宕機，冗餘將接替繼續提供服務。按可用的效率衡量高可用，例如「4個9」即99.99%的可用性，在99.99%的時間上不容許服務器宕機，當服務器宕機後做爲冗餘的服務器將當即接替主服務器提供服務，切換的時間間隔也較短，給用戶形成的影響較小。實現高可用的開源軟件有：heartbeat、keepalived。

負載均衡集羣：一般須要有一臺服務器做爲分發器，它負責把用戶的請求分發給後端的服務器集羣進行處理，在這個集羣裏，除了分發器外，就是給用戶提供服務的服務器，而且這些服務器的數量至少爲2臺。實現負載均衡的開源軟件有LVS、keepalived、haproxy、nginx，商業的負載均衡器有F五、Netscaler，優勢爲高併發量、高穩定性。

---

keepalived介紹

keepalived經過VRRP(Virtual Router Redundancy Protocol，虛擬路由冗餘協議)來實現高可用。

在這個協議裏會將多臺功能相同的路由器組成一個小組，這個小組裏會有1個master角色和N個backup角色。master會經過組播的形式向各個backup發送VRRP協議的數據包，當backup收不到master發來的VRRP數據包時，就會認爲master已宕機。此時就須要根據各個backup的優先級來決定哪一個backup成爲新的master。

keepalived有三個模塊，分別是core、check和vrrp。其中core模塊是keepalived的核心，負責主進程的啓動、維護及全局配置文件的加載和解析；check模塊負責健康檢查；vrrp模塊用來實現VRRP協議。

---

keepalived配置

test1:192.168.65.133 test2:192.168.65.134 test1做爲master，test2做爲backup

test1/test2都安裝keepalived和nginx

# 若是主機內已經源碼安裝過nginx就不須要再安裝了
[root@test1 ~]# yum install -y keepalived
[root@test1 ~]# yum install -y nginx

[root@test2 ~]# yum install -y keepalived
[root@test2 ~]# yum install -y nginx

更改配置文件

# 默認安裝keepalived後會有一個keepalived.conf,這裏清空內容後輸入下列的代碼
[root@test1 ~]# > /etc/keepalived/keepalived.conf
[root@test1 ~]# vim /etc/keepalived/keepalived.conf
# 全局定義參數
global_defs {
    # 出現問題時發郵件，郵件地址自定義
   notification_email {
     1245626656@qq.com
   }
   notification_email_from 1245626656@qq.com
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

# 用來檢測服務是否正常
vrrp_script chk_nginx {
    # 自定義腳本：檢測服務是否正常
    script "/usr/local/sbin/check_ng.sh"
    # 檢測間隔3s
    interval 3
}

# 定義相關master信息
vrrp_instance VI_1 {
    # 角色定義：主爲MASTER，從爲BACKUP
    state MASTER
    # 定義發送數據包的網卡
    interface ens33
    # 定義路由器id
    virtual_router_id 51
    # 權重
    priority 100
    advert_int 1
    # 定義認證信息
    authentication {
        auth_type PASS
        auth_pass test1
    }
    # 定義公有ip，正常時master綁定，master宕機後backup綁定
    virtual_ipaddress {
        192.168.188.100
    }
    # 加載檢測腳本，對於上面
    track_script {
        chk_nginx
    }
}

編輯檢測腳本

# 檢測服務是否正常，腳本所在路徑在/etc/keepalived/keepalived.conf內定義
[root@test1 ~]# vim /usr/local/sbin/check_ng.sh
#!/bin/bash
#時間變量，用於記錄日誌
d=`date --date today +%Y%m%d_%H:%M:%S`
#計算nginx進程數量
n=`ps -C nginx --no-heading | wc -l`

#若是進程爲0，則啓動nginx，而且再次檢測nginx進程數量，
#若是還爲0，說明nginx沒法啓動，此時須要關閉keepalived
if [ $n -eq "0" ]; then
        /etc/init.d/nginx start
        n2=`ps -C nginx --no-heading|wc -l`
        # 執行啓動nginx後發現nginx啓動不成功，就關閉keepalived服務，並記錄日誌
        if [ $n2 -eq "0"  ]; then
                echo "$d nginx down,keepalived will stop" >> /var/log/check_ng.log
                systemctl stop keepalived
        fi
fi

[root@test1 ~]# chmod 755 /usr/local/sbin/check_ng.sh
[root@test1 ~]# systemctl stop firewalld
[root@test1 ~]# setenforce 0

若是你master上的nginx是源碼編譯的，那麼對應腳本上啓動nginx的命令要作相應修改，此外master上的防火牆和selinux必須關閉，不然keepalived沒法拉起nginx。

keepalived的日誌文件記錄在/var/log/messages文件內

[root@test1 data]# less /var/log/messages
...
Jan 16 19:58:06 test1 Keepalived_vrrp[5547]: Registering gratuitous ARP shared channel
Jan 16 19:58:06 test1 Keepalived_vrrp[5547]: Opening file '/etc/keepalived/keepalived.conf'.
Jan 16 19:58:06 test1 kernel: show_signal_msg: 53 callbacks suppressed
Jan 16 19:58:06 test1 kernel: keepalived[5547]: segfault at 0 ip           (null) sp 00007fff063621b8 error 14 in libgcc_s-4.8.5-20150702.so.1[7fd2356f4000+15000]
Jan 16 19:58:06 test1 Keepalived_vrrp[5547]: VRRP_Instance(VI_
1) removing protocol VIPs.
Jan 16 19:58:06 test1 Keepalived[4983]: Keepalived_vrrp exited due to segmentation fault (SIGSEGV).
Jan 16 19:58:06 test1 Keepalived[4983]:  Please report a bug at https://github.com/acassen/keepalived/issues
Jan 16 19:58:06 test1 Keepalived[4983]:  and include this log from when keepalived started, what happened
Jan 16 19:58:06 test1 Keepalived[4983]:  immediately before the crash, and your configuration file.
Jan 16 19:58:06 test1 Keepalived[4983]: VRRP child process(5547) died: Respawning
Jan 16 19:58:06 test1 Keepalived[4983]: Starting VRRP child process, pid=5548
Jan 16 19:58:06 test1 Keepalived_vrrp[5548]: Registering Kernel netlink reflector
Jan 16 19:58:06 test1 Keepalived_vrrp[5548]: Registering Kernel netlink command channel
Jan 16 19:58:06 test1 Keepalived_vrrp[5548]: Registering gratuitous ARP shared channel
Jan 16 19:58:06 test1 Keepalived_vrrp[5548]: Opening file '/etc/keepalived/keepalived.conf'.
...

公有ip的查看

[root@test1 data]# ip addr
...
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    ...
    inet 192.168.65.100/32 scope global ens33
       valid_lft forever preferred_lft forever
    ...

配置backup從服務器

修改配置文件

# 配置前關閉防火牆及selinux
[root@test1 ~]# systemctl stop firewalld
[root@test1 ~]# setenforce 0

[root@test2 ~]# cat /etc/keepalived/keepalived.conf 
global_defs {
   notification_email {
     1245626656@qq.com
   }
   notification_email_from 1245626656@qq.com
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}
vrrp_script chk_nginx {
    script "/usr/local/sbin/check_ng.sh"
    interval 3
}
vrrp_instance VI_1 {
    # 配置爲backup機
    state BACKUP
    interface ens33
    # router_id與master一致
    virtual_router_id 51
    # 權重較master要小
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass test2
    }
    # 這樣設置公有ip，master宕機後，backup機綁定
    virtual_ipaddress {
        192.168.65.100
    }
    track_script {
        chk_nginx
    }
}

配置檢測腳本

[root@test2 ~]# vim /usr/local/sbin/check_ng.sh
#!/bin/bash
#時間變量，用於記錄日誌
d=`date --date today +%Y%m%d_%H:%M:%S`
#計算nginx進程數量
n=`ps -C nginx --no-heading | wc -l`

if [ $n -eq "0" ]; then
        # backup機上的nginx使用yum安裝的，啓動目錄不一樣
        systemctl start nginx
        n2=`ps -C nginx --no-heading|wc -l`
        # 執行啓動nginx後發現nginx啓動不成功，就關閉keepalived服務，並記錄日誌
        if [ $n2 -eq "0"  ]; then
                echo "$d nginx down,keepalived will stop" >> /var/log/check_ng.log
                systemctl stop keepalived
        fi
fi

# 配置權限
[root@test1 ~]# chmod 755 /usr/local/sbin/check_ng.sh

區分主從內的nginx

修改master上nginx默認主機的默認網頁內容爲「master web」；backup上nginx默認主頁的內容爲「backup web」;訪問公有ip看顯示的是哪一個服務器的默認網頁。

# 修改默認虛擬主機上的默認網頁內容，這個要根據本身的實際狀況修改
[root@test1 ~]# echo "master web server" > /data/wwwroot/default/index.html

# yum 安裝的nginx的默認網頁路徑是/usr/share/nginx/html/index.html
[root@test2 ~]# echo "backup web server" > /usr/share/nginx/html/index.html

測試高可用

master信息

[root@test1 data]# curl -x127.0.0.1:80 test1.com
master web server
[root@test1 data]# curl -x127.0.0.1:80 test1.com -I
HTTP/1.1 200 OK
Server: nginx/1.12.2
Date: Tue, ... 13:36:35 GMT
Content-Type: text/html
Content-Length: 18
Last-Modified: Tue, ... 13:33:27 GMT
Connection: keep-alive
ETag: "5a5dff27-12"
Accept-Ranges: bytes
# 當前公用ip綁定在test1上
[root@test1 data]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:d0:81:f5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.65.133/24 brd 192.168.65.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.65.100/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::893d:23f6:69dd:1d54/64 scope link 
       valid_lft forever preferred_lft forever

backup信息

[root@test2 ~]# curl -x127.0.0.1:80 test2.com
backup web server
[root@test2 ~]# curl -x127.0.0.1:80 test2.com -I
HTTP/1.1 200 OK
Server: nginx/1.12.2
Date: Tue, ... 13:35:49 GMT
Content-Type: text/html
Content-Length: 18
Last-Modified: Tue, ... 13:35:33 GMT
Connection: keep-alive
ETag: "5a5dffa5-12"
Accept-Ranges: bytes

1. 模擬主宕機：關閉master上的keepalived
2. 模擬主恢復服務：重啓master上的keepalived