keepalived+haproxy 高可用負載均衡

時間  2019-11-13
標籤 keepalived+haproxy keepalived haproxy 可用 負載 均衡 欄目 負載均衡 简体版
原文   原文鏈接

      因爲在生產環境使用了mysqlcluster,須要實現高可用負載均衡,這裏提供了keepalived+haproxy來實現. html

      keepalived主要功能是實現真實機器的故障隔離及負載均衡器間的失敗切換.可在第3,4,5層交換.它經過VRRPv2(Virtual Router Redundancy Protocol) stack實現的. mysql

      Layer3:Keepalived會按期向服務器羣中的服務器.發送一個ICMP的數據包（既咱們平時用的Ping程序）,若是發現某臺服務的IP地址沒有激活，Keepalived便報告這臺服務器失效，並將它從服務器羣中剔除，這種狀況的典型例子是某臺服務器被非法關機。Layer3的方式是以服務器的IP地址是否有效做爲服務器工做正常與否的標準。 linux

     Layer4:主要以TCP端口的狀態來決定服務器工做正常與否。如web server的服務端口通常是80，若是Keepalived檢測到80端口沒有啓動，則Keepalived將把這臺服務器從服務器羣中剔除。 web

     Layer5:在網絡上佔用的帶寬也要大一些。Keepalived將根據用戶的設定檢查服務器程序的運行是否正常，若是與用戶的設定不相符，則Keepalived將把服務器從服務器羣中剔除。 redis

Software Design sql


keepalived啓動後會有單個進程 shell

8352 ?        Ss     0:00 /usr/sbin/keepalived
 8353 ?        S      0:00  \_ /usr/sbin/keepalived
 8356 ?        S      0:01  \_ /usr/sbin/keepalived

父進程:內存管理,子進程管理等等 數據庫

子進程:VRRP子進程 vim

子進程:Healthchecking 子進程 bash


實例

2臺mysqlcluster 10.1.6.203 master  10.1.6.205 backup

vip 10.1.6.173 

目的訪問10.1.6.173 3366端口 分別輪詢經過haproxy轉發到10.1.6.203 3306 和10.1.6.205 3306

mysqlcluster搭建參照以前博客,這裏在2臺機上安裝keepalived 

root@10.1.6.203:~# apt-get install keepalived
root@10.1.6.203:~# cat /etc/keepalived/keepalived.conf 
vrrp_script chk_haproxy {
        script "killall -0 haproxy"   # verify the pid existance
        interval 2                    # check every 2 seconds
        weight -2                    # add 2 points of prio if OK
}

vrrp_instance VI_1 {
        interface eth1                # interface to monitor
        state MASTER                  
        virtual_router_id 51          # Assign one ID for this route
        priority 101                  # 101 on master, 100 on backup
        nopreempt
        debug

        virtual_ipaddress {
                10.1.6.173
        }

        track_script {       #注意大括號空格
                chk_haproxy
        }

       notify_master /etc/keepalived/scripts/start_haproxy.sh  #表示當切換到master狀態時,要執行的腳本
       notify_fault  /etc/keepalived/scripts/stop_keepalived.sh #故障時執行的腳本
       notify_stop   /etc/keepalived/scripts/stop_haproxy.sh # keepalived中止運行前運行notify_stop指定的腳本 }

VRRPD配置包括三個類:

VRRP同步組(synchroization group)

VRRP實例(VRRP Instance)

VRRP腳本

這裏使用了 VRRP實例, VRRP腳本


注意配置選項: 

stat:指定instance(Initial)的初始狀態，就是說在配置好後，這臺服務器的初始狀態就是這裏指定的，但這裏指定的不算，仍是得要經過競選經過優先級來肯定，裏若是這裏設置爲master，但如若他的優先級不及另一臺，那麼這臺在發送通告時，會發送本身的優先級，另一臺發現優先級不如本身的高，那麼他會就回搶佔爲master

interface：實例綁定的網卡，由於在配置虛擬IP的時候必須是在已有的網卡上添加的

priority 101：設置本節點的優先級，優先級高的爲master

debug：debug級別

nopreempt:設置爲不搶佔

vrrp_script chk_haproxy {
        script "killall -0 haproxy"   # verify the pid existance
        interval 2                    # check every 2 seconds 腳本執行間隔
        weight -2                    # add 2 points of prio if OK 腳本結果致使的優先級變動：2表示優先級+2；-2則表示優先級-2
}

而後在實例(vrrp_instance)裏面引用，有點相似腳本里面的函數引用同樣：先定義，後引用函數名
        track_script {
                chk_haproxy
        }
注意：VRRP腳本(vrrp_script)和VRRP實例(vrrp_instance)屬於同一個級別 

root@10.1.6.203:scripts# cat start_haproxy.sh 
#!/bin/bash

sleep 5
get=`ip addr  |grep 10.1.6.173 |wc -l`
echo $get >> /etc/keepalived/scripts/start_ha.log

if [ $get -eq 1 ]
then
        echo "`date +%c` success to get vip" >> /etc/keepalived/scripts/start_ha.log
        /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg
else
        echo "`date +%c` can not get vip" >> /etc/keepalived/scripts/start_ha.log
fi
root@10.1.6.203:scripts# cat stop_keepalived.sh 
#!/bin/bash

pid=`pidof keepalived`
if [ $pid == "" ]
then
	echo "`date +%c` no keepalived process id"  >> /etc/keepalived/scripts/stop_keep.log
else
	echo "`date +%c` will stop keepalived "  >> /etc/keepalived/scripts/stop_keep.log
	/etc/init.d/keepalived  stop
fi

/etc/init.d/keepalived  stop

root@10.1.6.203:scripts# cat stop_haproxy.sh 
#!/bin/bash

pid=`pidof haproxy`
echo "`date +%c` stop haproxy" >> /etc/keepalived/scripts/stop_ha.log
kill -9 $pid

同理配置10.1.6.205 

root@10.1.6.205:~# cat /etc/keepalived/keepalived.conf 
vrrp_script chk_haproxy {
   script "killall -0 haproxy"   # verify the pid existance
   interval 2                    # check every 2 seconds
   weight 2                      # add 2 points of prio if OK
}
 
vrrp_instance VI_1 {
   interface eth1               # interface to monitor
   state BACKUP
   virtual_router_id 51          # Assign one ID for this route
   priority 100                 # 101 on master, 100 on backup
   virtual_ipaddress {
       10.1.6.173
   }

   track_script {
       chk_haproxy
   }

notify_master /etc/keepalived/scripts/start_haproxy.sh
notify_fault  /etc/keepalived/scripts/stop_keepalived.sh
notify_stop /etc/keepalived/scripts/stop_haproxy.sh

}

下面再介紹下haproxy

       HAProxy是一款基於TCP(第四層)和HTTP(第七層)應用的代理軟件,它也可做爲負載均衡器.能夠支持數以萬計的併發鏈接.同時能夠保護服務器不暴露到網絡上,經過端口映射.它還自帶監控服務器狀態的頁面.

      安裝haproxy 

wget -O/tmp/haproxy-1.4.22.tar.gz http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.22.tar.gz
tar xvfz /tmp/haproxy-1.4.22.tar.gz -C /tmp/
cd /tmp/haproxy-1.4.22
make TARGET=linux26
make install

      haproxy須要對每個mysqlcluster服務器進行健康檢查

1.在2臺主機分別配置haproxy.cfg 

root@10.1.6.203:scripts# cat /etc/haproxy/haproxy.cfg 
global
        maxconn 51200  #默認最大鏈接數 
        #uid 99
        #gid 99
        daemon        #之後臺形式運行haproxy
        #quiet
        nbproc 1      #進程數量(能夠設置多個進程提升性能) 
        pidfile /etc/haproxy/haproxy.pid  #haproxy的pid存放路徑,啓動進程的用戶必須有權限訪問此文件 

defaults
        mode tcp            #所處理的類別 (#7層 http;4層tcp  ) 
        option redispatch   #serverId對應的服務器掛掉後,強制定向到其餘健康的服務器 
        option abortonclose #當服務器負載很高的時候，自動結束掉當前隊列處理比較久的鏈接 
        timeout connect 5000s   #鏈接超時
        timeout client 50000s  #客戶端超時
        timeout server 50000s   #服務器超時
        log 127.0.0.1 local0   #錯誤日誌記錄
        balance roundrobin    #默認的負載均衡的方式,輪詢方式 

listen proxy
        bind 10.1.6.173:3366   #監聽端口 
        mode tcp               #http的7層模式
        option  httpchk        #心跳檢測的文件
        server db1 10.1.6.203:3306  weight 1 check port 9222 inter 12000 rise 3 fall 3      #服務器定義,check inter 12000是檢測心跳頻率 rise 3是3次正確認爲服務器可用， fall 3是3次失敗認爲服務器不可用，weight表明權重 
        server db2 10.1.6.205:3306  weight 1 check port 9222 inter 12000 rise 3 fall 3

listen  haproxy_stats
        mode http
        bind 10.1.6.173:8888
        option httplog
        stats refresh 5s   
        stats uri /status #網站健康檢測URL，用來檢測HAProxy管理的網站是否能夠用，正常返回200，不正常返回503 
        stats realm Haproxy Manager
        stats auth admin:p@a1SZs24 #帳號密碼
root@10.1.6.205:~$ cat /etc/haproxy/haproxy.cfg 
global
        maxconn 51200
        #uid 99
        #gid 99
        daemon
        #quiet
        nbproc 1
        pidfile /etc/haproxy/haproxy.pid

defaults
        mode tcp
        option redispatch   
        option abortonclose
        timeout connect 5000s
        timeout client 50000s
        timeout server 50000s
        log 127.0.0.1 local0
        balance roundrobin 

listen proxy
        bind 10.1.6.173:3366
        mode tcp
        option  httpchk
        server db1 10.1.6.203:3306  weight 1 check port 9222 inter 12000 rise 3 fall 3
        server db2 10.1.6.205:3306  weight 1 check port 9222 inter 12000 rise 3 fall 3

listen  haproxy_stats
        mode http
        bind 10.1.6.173:8888
        option httplog
        stats refresh 5s   
        stats uri /status  
        stats realm Haproxy Manager
        stats auth admin:p@a1SZs24

2.安裝xinetd 

root@10.1.6.203:~# apt-get install xinetd

3.在每一個節點添加xinetd服務腳本和mysqlchk端口號 

root@10.1.6.203:~# vim /etc/xinetd.d/mysqlchk 
# default: on
# description: mysqlchk
service mysqlchk                  #須要在servive定義
{
        flags           = REUSE
        socket_type     = stream
        port            = 9222
        wait            = no
        user            = nobody
        server          = /opt/mysqlchk  
        log_on_failure  += USERID
        disable         = no
        per_source      = UNLIMITED
        bind            = 10.1.6.173
}

root@10.1.6.203:~# vim /etc/services 
mysqlchk        9222/tcp                        # mysqlchk
4.編寫mysqlchk監控服務腳本 
root@10.1.6.203:~# ls -l /opt/mysqlchk 
-rwxr--r-- 1 nobody root 1994 2013-09-17 11:27 /opt/mysqlchk
root@10.1.6.203:~# cat /opt/mysqlchk 
#!/bin/bash
#
# This script checks if a mysql server is healthy running on localhost. It will
# return:
# "HTTP/1.x 200 OK\r" (if mysql is running smoothly)
# - OR -
# "HTTP/1.x 500 Internal Server Error\r" (else)
#
# The purpose of this script is make haproxy capable of monitoring mysql properly
#

MYSQL_HOST="localhost"
MYSQL_SOCKET="/var/run/mysqld/mysqld.sock" 
MYSQL_USERNAME="mysqlchkusr"     #該帳戶密碼須要在mysql裏添加
MYSQL_PASSWORD="secret"
MYSQL_OPTS="-N -q -A"
TMP_FILE="/dev/shm/mysqlchk.$$.out"
ERR_FILE="/dev/shm/mysqlchk.$$.err"
FORCE_FAIL="/dev/shm/proxyoff"
MYSQL_BIN="/opt/mysqlcluster/mysql-cluster-gpl-7.2.6-linux2.6-x86_64/bin/mysql"
CHECK_QUERY="select 1"

preflight_check()
{
    for I in "$TMP_FILE" "$ERR_FILE"; do
        if [ -f "$I" ]; then
            if [ ! -w $I ]; then
                echo -e "HTTP/1.1 503 Service Unavailable\r\n"
                echo -e "Content-Type: Content-Type: text/plain\r\n"
                echo -e "\r\n"
                echo -e "Cannot write to $I\r\n"
                echo -e "\r\n"
                exit 1
            fi
        fi
    done
}

return_ok()
{
    echo -e "HTTP/1.1 200 OK\r\n"
    echo -e "Content-Type: text/html\r\n"
    echo -e "Content-Length: 43\r\n"
    echo -e "\r\n"
    echo -e "<html><body>MySQL is running.</body></html>\r\n"
    echo -e "\r\n"
    rm $ERR_FILE $TMP_FILE
    exit 0
}
return_fail()
{
    echo -e "HTTP/1.1 503 Service Unavailable\r\n"
    echo -e "Content-Type: text/html\r\n"
    echo -e "Content-Length: 42\r\n"
    echo -e "\r\n"
    echo -e "<html><body>MySQL is *down*.</body></html>\r\n"
    sed -e 's/\n$/\r\n/' $ERR_FILE
    echo -e "\r\n"
    rm $ERR_FILE $TMP_FILE
    exit 1
}
preflight_check
if [ -f "$FORCE_FAIL" ]; then
        echo "$FORCE_FAIL found" > $ERR_FILE
        return_fail;
fi
$MYSQL_BIN $MYSQL_OPTS --host=$MYSQL_HOST --socket=$MYSQL_SOCKET --user=$MYSQL_USERNAME --password=$MYSQL_PASSWORD -e "$CHECK_QUERY" > $TMP_FILE 2> $ERR_FILE
if [ $? -ne 0 ]; then
        return_fail;
fi
return_ok;


測試

2個節點開啓keepalived(主節點會得到vip,自動拉起haproxy),xinetd 

root@10.1.6.203:~# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
    link/ether 00:26:b9:36:0f:81 brd ff:ff:ff:ff:ff:ff
    inet 211.151.105.186/26 brd 211.151.105.191 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:26:b9:36:0f:83 brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.203/24 brd 10.1.6.255 scope global eth1
    inet 10.1.6.173/32 scope global eth1
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:26:b9:36:0f:85 brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:26:b9:36:0f:87 brd ff:ff:ff:ff:ff:ff
root@10.1.6.203:~# netstat -tunlp | grep ha
tcp        0      0 10.1.6.173:3366     0.0.0.0:*               LISTEN      1042/haproxy    
tcp        0      0 10.1.6.203:8888     0.0.0.0:*               LISTEN      1042/haproxy    
udp        0      0 0.0.0.0:56562           0.0.0.0:*                           1042/haproxy    
root@10.1.6.203:~# netstat  -tunlp | grep xine
tcp        0      0 10.1.6.203:9222     0.0.0.0:*               LISTEN      30897/xinetd    
root@10.1.6.203:~# ps -ef | grep haproxy
root      1042     1  0 Sep17 ?        00:00:00 /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg


測試:

經過vip10.1.6.173 3366訪問cluster數據庫(注意帳戶dave權限須要加3個ip10.1.6.203,10.1.6.205,10.1.6.173) 

root@10.1.6.203:mgm# mysql -udave -p -h 10.1.6.173 -P 3366
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1344316
Server version: 5.5.22-ndb-7.2.6-gpl-log MySQL Cluster Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema | 
| dave              | 
| test               | 
+--------------------+
3 rows in set (0.01 sec)

mysql>

手動分別使keepalive,haproxy,數據庫掛掉.vip10.1.6.173會自動漂到10.1.6.205從上，並不影響vip的訪問


經過vip,haproxy查看各節點狀態

http://10.1.6.173:8888/status



參考:

http://www.keepalived.org/

http://haproxy.1wt.eu/

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程