Keepalived實現LVS-DR集羣高可用

時間  2019-11-29
標籤 keepalived 實現 lvs 集羣 可用 欄目 負載均衡 简体版
原文   原文鏈接

1、集羣高可用概述html

單純的lvs/nginx反向代理模型作負載集羣應用時，DR（director）存在單點故障隱患，故須要有機制來保證DR的高可用性。經常使用的高可用性方案有Keepalived、corosync，Keepalived主要是由VRRP協議實現了VIPfloating，比較適用於前端DR的高可用性，Corosync通常用於更專業的集羣模型實現Service的高可用。Keepalived起初就是爲了實現LVS集羣director高可用而開發的，本處僅作Keepalived+LVS-DR模型實驗。前端

2、Keepalived原理簡介nginx

Keepalived中優先級高的節點爲MASTER。MASTER其中一個職責就是響應VIP的arp包，將VIP和mac地址映射關係告訴局域網內其 他主機，同時，它還會以多播的形式向局域網中發送VRRP通告，告知BACKUP組本身的優先級。網絡中的全部BACKUP節點只負責 處理MASTER發出的多播包，當發現MASTER的優先級沒本身高（腳本檢測故障觸發自我降級），或者沒收到MASTER的VRRP通告（網絡故障/MASTER宕機）時，BACKUP將本身切換到MASTER狀 態，而後作MASTER該作的事：1.響應arp包，2.發送VRRP通告。vim

3、實驗環境緩存

1.網絡拓補圖bash

2.軟件環境服務器

  • CentOS7.4
  • keepalived.x86_64 1.3.5-6.el7
  • nginx.x86_64 1:1.12.2-2.el7

4、配置流程網絡

（1）兩臺DR配置keepalived.confdom

  1. 配置DR1：curl

    global_defs {
    notification_email {
    root@localhost #此處僅發給本機，更定製化的郵件通知功能通常由zabbix來作。
    }
    notification_email_from keepalived@localhost
    smtp_server 127.0.0.1
    smtp_connect_timeout 30
    router_id keepalivedR1 #定義路由器標識,每臺服務器局域網內惟一就行。
    vrrp_mcast_group4 224.0.0.33 #定義master向backup組播vrrp報文的地址。
    }
    vrrp_script chk_down {
    script "/etc/keepalived/chk_down.sh"
    interval 1 #腳本檢測間隔
    weight -15 #即原有優先級+weight，負值即下降。
    # 注意當weight=0時, 用於變動vrrp_instance的狀態例如腳本檢測失敗, 則vrrp的狀態直接變爲FAULT. (無論有沒有其餘節點存在來接管MASTER)
    fall 2 #執行腳本兩次exit非0則下降優先級
    rise 1 #執行腳本一次exit 0 則還原優先級
    user keepalived_script #默認用戶yum安裝keepalived需自建，不存在則調用root（不推薦）
    }

    vrrp_instance VI_1 {
                 state MASTER    #自定義的state，但若是你的優先級小於backup也不會成爲master。
                 interface ens39       #綁定爲當前虛擬路由器使用的物理接口。
                 virtual_router_id 3  #當前虛擬路由器的唯一標識，範圍是0-255。同實例一致！
                 priority 99         #初始優先級，範圍1-254。
                 advert_int 1        #vrrp通告的時間間隔。
                 authentication {
                     auth_type PASS   #簡單密碼驗證，不超過8位。
                     auth_pass 736w4ib2  #最好使用隨機字符串,同vip實例保持一致！
                 }
                 virtual_ipaddress {
                     192.168.7.120/24 dev ens39
                 }
                 notify_master "/etc/keepalived/notify.sh master" #調用通知腳本
                 notify_backup "/etc/keepalived/notify.sh backup"
                 notify_fault "/etc/keepalived/notify.sh fault"
             }
            virtual_server 192.168.7.120 80 {  #此處可用IP port/fwmark id 標識VIP對應服務
                  delay_loop 2   #服務輪詢的時間間隔2s
                 lb_algo rr
                 lb_kind DR
                 protocol TCP
                 real_server 192.168.7.125 80 {
                     weight 1
                     HTTP_GET {
                     url {
                         path /
                         status_code 200
                     }
                     connect_timeout 2
                     nb_get_retry 3
                     delay_before_retry 3
                     }
                 }
                 real_server 192.168.7.126 80 {
                     weight 1
                     HTTP_GET {
                         url {
                             path /
                             status_code 200
                         }
                         connect_timeout 2
                         nb_get_retry 3
                         delay_before_retry 3
                         }
              }

將上述配置錄入/etc/keepalived/keepalived.conf中，過程以下：

[root@DR1 ~]# cd /etc/keepalived/
[root@DR1 keepalived]# cp keepalived.conf{,.bak} #備份下配置文件
[root@DR1 keepalived]# ls
keepalived.conf  keepalived.conf.bak
[root@DR1 keepalived]# vim keepalived.conf

  • 配置中調用的notify腳本內容以下：

    #!/bin/bash
  #
  contact='root@localhost'

  notify() {
      local mailsubject="$(hostname) to be $1, vip floating"
      local mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
      echo "$mailbody" | mail -s "$mailsubject" $contact
  }

  case $1 in
  master)
      notify master
      ;;
  backup)
      notify backup
      ;;
  fault)
      notify fault
      ;;
  *)
      echo "Usage: $(basename $0) {master|backup|fault}"
      exit 1
      ;;
  esac

  • chk_down.sh只有條件判斷：[[ -f /etc/keepalived/down ]] && exit 1 || exit 0

  1. 配置DR2

DR2，僅需修改以下配置：

  • router_id keepalivedR2
  • state BACKUP
  • priority 90

（2）兩臺RS配置

兩臺RS均yum安裝nginx，啓動服務監聽80端口便可。

須要注意的有以下幾點：

  • 檢查系統是否自帶httpd服務並中止，不然會與nginx發生衝突。

  • 添加vip到網卡輔助接口，並修改arp參數抑制apr報文響應，此處可用腳本以下：
    #!/bin/bash
    vip=192.168.7.120
    /usr/sbin/ip addr add $vip/32 dev lo label lo:0
    echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
    echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

    注意：因客戶端在同網段，添加完RS抑制arp響應後，須要將客戶端的arp緩存清空防止干擾效果。

  • 修改默認主頁，主要是作標記區分RS1/RS2。可採用以下方法：

    編輯默認index.html
    在vim末行模式輸入:%s/nginx/server1/g回車便可

5、測試實驗效果

  1. 測試VIP漂移

    • 先在DR2初始BACKUP開啓keepalived觀察是否有狀態變動：
      [root@DR2 ~]# systemctl start keepalived.service
      [root@DR2 ~]# systemctl status keepalived.service
      ?.keepalived.service - LVS and VRRP High Availability Monitor
      Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
      Active: active (running) since Sun 2018-07-15 13:28:10 CST; 19s ago
      Process: 2620 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
      Main PID: 2621 (keepalived)
      CGroup: /system.slice/keepalived.service
      ?..2621 /usr/sbin/keepalived -D
      ?..2622 /usr/sbin/keepalived -D
      ?..2623 /usr/sbin/keepalived -D

      Jul 15 13:28:14 DR2 Keepalived_vrrp[2623]: Sending gratuitous ARP on ens39 for 192.168.7.120
Jul 15 13:28:14 DR2 Keepalived_vrrp[2623]: Sending gratuitous ARP on ens39 for 192.168.7.120
Jul 15 13:28:14 DR2 Keepalived_vrrp[2623]: Sending gratuitous ARP on ens39 for 192.168.7.120
Jul 15 13:28:14 DR2 Keepalived_vrrp[2623]: Opening script file /etc/keepalived/notify.sh
Jul 15 13:28:19 DR2 Keepalived_vrrp[2623]: Sending gratuitous ARP on ens39 for 192.168.7.120
Jul 15 13:28:19 DR2 Keepalived_vrrp[2623]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens39 for 192.168.7.120
[root@DR2 ~]# mail
Heirloom Mail version 12.5 7/5/10.  Type ? for help.
"/var/spool/mail/root": 6 messages 2 new 2 unread
>N  5 root                  Sun Jul 15 13:28  18/667   "DR2 to be backup, vip floating"
 N  6 root                  Sun Jul 15 13:28  18/667   "DR2 to be master, vip floating"
& 6
Message  6:
From root@DR2.localdomain  Sun Jul 15 13:28:15 2018
Return-Path: <root@DR2.localdomain>
X-Original-To: root@localhost
Delivered-To: root@localhost.localdomain
Date: Sun, 15 Jul 2018 13:28:14 +0800
To: root@localhost.localdomain
Subject: DR2 to be master, vip floating
User-Agent: Heirloom mailx 12.5 7/5/10
Content-Type: text/plain; charset=us-ascii
From: root@DR2.localdomain (root)
Status: R

2018-07-15 13:28:14: vrrp transition, DR2 changed to be master
      能夠看到，由於master沒上線，DR2由backup變爲master。咱們的notify.sh腳本成功發送狀態變動通知到本地郵箱。
    • 測試DR1上線效果：
      [root@DR2 ~]# systemctl status keepalived.service
      Jul 15 14:52:48 DR2 Keepalived_vrrp[1812]: VRRP_Instance(VI_1) Received advert with higher priority 99, ours 90
      Jul 15 14:52:48 DR2 Keepalived_vrrp[1812]: VRRP_Instance(VI_1) Entering BACKUP STATE
      Jul 15 14:52:48 DR2 Keepalived_vrrp[1812]: VRRP_Instance(VI_1) removing protocol VIPs.
      Jul 15 14:52:48 DR2 Keepalived_vrrp[1812]: Opening script file /etc/keepalived/notify.sh
      You have new mail in /var/spool/mail/root
      [root@DR2 ~]# ip addr show
      1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
      valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host
      valid_lft forever preferred_lft forever
      2: ens39: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
      link/ether 00:50:56:3b:a3:7c brd ff:ff:ff:ff:ff:ff
      inet 192.168.7.122/24 brd 192.168.7.255 scope global ens39
      valid_lft forever preferred_lft forever
      inet6 fe80::250:56ff:fe3b:a37c/64 scope link
      valid_lft forever preferred_lft forever
      [root@DR1 ~]# systemctl status keepalived.service
      Jul 15 14:52:49 DR1 Keepalived_vrrp[3471]: Sending gratuitous ARP on ens39 for 192.168.7.120
      Jul 15 14:52:49 DR1 Keepalived_vrrp[3471]: Opening script file /etc/keepalived/notify.sh
      Jul 15 14:52:54 DR1 Keepalived_vrrp[3471]: Sending gratuitous ARP on ens39 for 192.168.7.120
      Jul 15 14:52:54 DR1 Keepalived_vrrp[3471]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens39 for 192.168.7.120
      DR2由於收到了更高優先級的VRRP組播報文而卸載VIP，DR1獲取到VIP。

    • 手工touch生成/etc/keepalived/down文件模擬單機故障也會發生上述VIP漂移現象。
      Jul 15 15:21:51 DR1 Keepalived_vrrp[3907]: WARNING - default user 'keepalived_script' for script execution does not exist - please create.
      # 此處踩坑了，提示conf文件中配置的腳本須要keepalived_script用戶執行
      # 還有個巨坑是參考資料中直接在keepalived.conf中定義script "[[ -f /etc/keepalived/down ]] && # exit 1 || exit 0" 可是我屢次測試不成功，將條件判斷放入腳本，conf文件僅引用路徑才成功......
      Jul 15 15:54:24 DR1 Keepalived_vrrp[4438]: /etc/keepalived/chk_down.sh exited with status 1
      Jul 15 15:54:24 DR1 Keepalived_vrrp[4438]: VRRP_Script(chk_down) failed
      Jul 15 15:54:25 DR1 Keepalived_vrrp[4438]: VRRP_Instance(VI_1) Changing effective priority from 99 to 84
      Jul 15 15:54:25 DR1 Keepalived_vrrp[4438]: /etc/keepalived/chk_down.sh exited with status 1
      Jul 15 15:54:25 DR1 Keepalived_vrrp[4438]: VRRP_Instance(VI_1) Received advert with higher priority 90, ours 84
      Jul 15 15:54:25 DR1 Keepalived_vrrp[4438]: VRRP_Instance(VI_1) Entering BACKUP STATE
      Jul 15 15:54:25 DR1 Keepalived_vrrp[4438]: VRRP_Instance(VI_1) removing protocol VIPs.
      Jul 15 15:54:25 DR1 Keepalived_vrrp[4438]: Opening script file /etc/keepalived/notify.sh

  2. 測試RS故障檢測
    • 先測試RS是否正常被輪詢 for i in {1,5};do curl http://192.168.7.120 ;done
    • 手動關停一臺rs的nginx
      [root@RS2 ~]# systemctl stop nginx
      [root@DR1 ~]# journalctl -xe
      Jul 15 16:59:04 DR1 Keepalived_healthcheckers[4437]: Error connecting server [192.168.7.126]:80.
      [root@DR1 ~]# systemctl status keepalived
      Jul 15 16:59:05 DR1 Keepalived_healthcheckers[4437]: Error connecting server [192.168.7.126]:80.
      Jul 15 16:59:06 DR1 Keepalived_healthcheckers[4437]: Error connecting server [192.168.7.126]:80.
      Jul 15 16:59:07 DR1 Keepalived_healthcheckers[4437]: Error connecting server [192.168.7.126]:80.
      Jul 15 16:59:07 DR1 Keepalived_healthcheckers[4437]: Check on service [192.168.7.126]:80 failed after 3 retry.
      Jul 15 16:59:07 DR1 Keepalived_healthcheckers[4437]: Removing service [192.168.7.126]:80 from VS [192.168.7.120]:80
      Jul 15 16:59:07 DR1 Keepalived_healthcheckers[4437]: Remote SMTP server [127.0.0.1]:25 connected.
      Jul 15 16:59:07 DR1 Keepalived_healthcheckers[4437]: SMTP alert successfully sent.
      [root@DR1 ~]# mail
      N 16 keepalived@localhost Sun Jul 15 16:59 17/646 "[keepalivedR1] Realserver [192.168.7.126]:80 - DOWN"
      • 從新啓動nginx Jul 15 17:02:55 DR1 Keepalived_healthcheckers[4437]: HTTP status code success to [192.168.7.126]:80 url(1). Jul 15 17:02:55 DR1 Keepalived_healthcheckers[4437]: Remote Web server [192.168.7.126]:80 succeed on service. Jul 15 17:02:55 DR1 Keepalived_healthcheckers[4437]: Adding service [192.168.7.126]:80 to VS [192.168.7.120]:80 Jul 15 17:02:55 DR1 Keepalived_healthcheckers[4437]: Remote SMTP server [127.0.0.1]:25 connected. Jul 15 17:02:55 DR1 Keepalived_healthcheckers[4437]: SMTP alert successfully sent. You have new mail in /var/spool/mail/root 從新上線的RS能夠順利被檢測到並上線服務，至此，本次實驗結束。
相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程