使用LVS和Keepalived搭建高可用WEB服務

by: 白馬公園/naritechlinux

 

         本文的主旨在於配置一個儘量簡單的高可用WEB服務系統,幫助讀者理清keepalived、LVS等軟件的配置和使用,爲進一步用好LVS樹立信心,夯實基礎。nginx

本文只涉及使用LVS和Keepalived解決WEB服務的高可用和高併發,對於WEB集羣中的session共享即數據一致性問題不作討論,對WEB集羣中的session共享問題感興趣的朋友能夠參考個人另一篇文章《nginx、tomcat、redis配置session共享》web

         網上關於LVS和keepalived搭建高可用WEB服務的文章不少,本文的不一樣之處在於筆者在配置的過程當中遇到了不少問題,於是走了很多彎路。因此本文的重點將放在這些問題的講述上。redis

 

1、  系統架構算法

全系統包括五臺服務器,均爲虛擬機,安裝Centos6.5操做系統,其中tomcat

hadoop01   192.168.75.88              安裝keepalived,做爲主負載調度器使用服務器

hadoop02   192.168.75.89              安裝keepalived,做爲備用負載調度器使用session

 

hadoop03   192.168.75.90              安裝tomcat,做爲真實Web服務器使用架構

hadoop04   192.168.75.91              安裝tomcat,做爲真實Web服務器使用併發

hadoop05   192.168.75.92              安裝tomcat,做爲真實Web服務器使用

 

         VIP設定爲 192.168.75.188,正常運行時該IP被設置在hadoop01上,接受客戶端請求並按照負載分配策略挑選出一臺真實的服務器,並將請求分配給真實的服務器進行處理。當hadoop01上的keepalived退出運行時,VIP被設置到hadoop02上,hadoop02接管此前hadoop01的任務,繼續接受客戶端請求並提供負載調度服務

 

2、  keepalived的安裝配置(hadoop01和hadoop02均須要安裝配置)

 

2.1    keepalived版本選擇

         最新的keepalived1.2.24對應Centos6.5操做系統在安裝上有些問題,make不能成功,不推薦使用Keepalived1.2.6版本安裝後運行異常,跟蹤日誌文件後沒法定位問題,也不推薦使用,這裏選擇keepalived1.2.19,安裝介質能夠在keepalived官網上下載

 

2.2    keepalived安裝前的準備

         一、須要安裝C++編譯器。

                   C++編譯器的安裝請參考《nginx、tomcat、redis配置session共享》

         二、須要安裝openSSL

                   rpm -ivh zlib-1.2.3-29.el6.x86_64.rpm

rpm -ivh zlib-devel-1.2.3-29.el6.x86_64.rpm

rpm -ivh libsepol-devel-2.0.41-4.el6.x86_64.rpm

rpm -ivh pkgconfig-0.23-9.1.el6.x86_64.rpm

rpm -ivh libcom_err-devel-1.41.12-18.el6.x86_64.rpm

rpm -ivh keyutils-libs-devel-1.4-4.el6.x86_64.rpm

rpm -ivh libselinux-devel-2.0.94-5.3.el6_4.1.x86_64.rpm

rpm -ivh krb5-devel-1.10.3-10.el6_4.6.x86_64.rpm

rpm -ivh openssl-1.0.1e-15.el6.x86_64.rpm

                   rpm -ivh openssl-devel-1.0.1e-15.el6.x86_64.rpm

 

三、須要安裝kernel-devel開發包

         rpm -ivh kernel-devel-2.6.32-431.el6.x86_64.rpm

 

2.3    keepalived的configure

        

./configure --prefix=/usr/local/keepalived --with-kernel-dir=/usr/src/kernels/2.6.32-431.el6.x86_64 --disable-fwmark

         當安裝了kernel-devel開發包以後,系統就有了/usr/src/kernels/2.6.32-431.el6.x86_64這樣的相似目錄,直接找到這個目錄便可

 

         Configure成功後會顯示出如下內容:

 

         Keepalived configuration

------------------------

Keepalived version       : 1.2.19

Compiler                 : gcc

Compiler flags           : -g -O2

Extra Lib                : -lssl -lcrypto -lcrypt

Use IPVS Framework       : Yes

IPVS sync daemon support : Yes

IPVS use libnl           : No

fwmark socket support    : No

Use VRRP Framework       : Yes

Use VRRP VMAC            : Yes

SNMP support             : No

SHA1 support             : No

Use Debug flags          : No

        

特別要注意:Use IPVS Framework、IPVS sync daemon support、Use VRRP Framework、Use VRRP VMAC這四項必須爲Yes

 

2.4    keepalived的make

         介質路徑下直接鍵入make,注意使用root用戶

         make成功以後會顯示如下內容:

 

Make complete

make[1]: Leaving directory `/users/oracle/software/keepalived-1.2.19/genhash'

 

Make complete

 

2.5    keepalived的make install

         介質路徑下直接鍵入make install,注意使用root用戶

         make install成功以後會顯示如下內容:

        

make[1]: Leaving directory `/users/oracle/software/keepalived-1.2.19/keepalived'

make -C genhash install

make[1]: Entering directory `/users/oracle/software/keepalived-1.2.19/genhash'

install -d /usr/local/keepalived/bin

install -m 755 ../bin/genhash /usr/local/keepalived/bin/

install -d /usr/local/keepalived/share/man/man1

install -m 644 ../doc/man/man1/genhash.1 /usr/local/keepalived/share/man/man1

make[1]: Leaving directory `/users/oracle/software/keepalived-1.2.19/genhash'

 

2.6    安裝後的系統級設置

         cp     /usr/local/keepalived/sbin/keepalived  /usr/sbin/

         cp      /usr/local/keepalived/etc/sysconfig/keepalived  /etc/sysconfig/

         cp     /usr/local/keepalived/etc/rc.d/init.d/keepalived  /etc/init.d/

         chmod 777 /etc/init.d/keepalived

         cd     /etc/init.d

         chkconfig --add keepalived

         chkconfig keepalived on

         mkdir –p /etc/keepalived

         touch /etc/keepalived/keepalived.conf

 

2.7    keepalived的配置

 

! Configuration File for keepalived

 

global_defs {               

   notification_email {

     sadgump@163.com                                //能夠是虛假的email地址,但要配置

   }

   notification_email_from  sadgump@163.com          //能夠是虛假的email地址,但要配置

   smtp_server 192.168.200.1                        //能夠是虛假的IP地址,但必定要配置

   smtp_connect_timeout 30                          //默認

   router_id LVS_DEVEL                                            //默認

}

 

vrrp_instance VI_1 {                       //定義一個VRRP實力對象  VI_1

    state MASTER                                  //hadoop01上設置爲MASTER,hadoop02上設置爲BACKUP

    interface eth0                                  //定義設置VIP地址的網卡名

    virtual_router_id 51                //默認

    priority 100           //hadoop01設爲100,hadoop02設爲99,數越小,級別越低

    advert_int 1          //默認

    authentication {            

        auth_type PASS               //默認

        auth_pass 1111              //默認

    }

    virtual_ipaddress {

        192.168.75.188               //VIP地址

    }

}

 

//VIP地址的監聽端口,該端口要和實際服務器的監聽端口一致,不然會不能正常轉發

virtual_server 192.168.75.188 8080 {     

    delay_loop 6                           //默認

    lb_algo rr                                //負載調度算法設置

    lb_kind DR                    //負載均衡轉發:使用效率最高的DR算法

    nat_mask 255.255.255.0        //默認

    persistence_timeout 50                   //默認

    protocol TCP                                     //轉發支持的協議

 

    real_server 192.168.75.90 8080 {             //實時服務器一的地址及端口

        weight 1                                                      //權重

        TCP_CHECK {                                     

            connect_timeout 3                    //默認

            nb_get_retry 3                           //默認

            delay_before_retry 3                 //默認

        }

    }

 

    real_server 192.168.75.91 8080 {             //實時服務器二的地址及端口

        weight 1                                                      //權重

        TCP_CHECK {                                     

            connect_timeout 3                    //默認

            nb_get_retry 3                           //默認

            delay_before_retry 3                 //默認

        }

    }

   

    real_server 192.168.75.92 8080 {             //實時服務器三的地址及端口

        weight 1                                                      //權重

        TCP_CHECK {                                     

            connect_timeout 3                    //默認

            nb_get_retry 3                           //默認

            delay_before_retry 3                 //默認

        }

    }

}

3、  實際服務器的軟件安裝和配置

實際服務器有三臺即hadoop0三、hadoop0四、hadoop05,均須要安裝和配置

3.1    tomcat的安裝

         略

         在三臺實際服務器上啓動tomcat,監聽端口爲8080

 

3.2    IPVS相關配置

編輯一個realserver.sh腳本,chmod 777 realserver.sh

內容:

 

SNS_VIP=192.168.75.188

source /etc/rc.d/init.d/functions

case "$1" in

start)

ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP up

/sbin/route add -host $SNS_VIP dev lo:0

echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

sysctl -p >/dev/null 2>&1

echo "RealServer Start OK"

;;

stop)

ifconfig lo:0 down

/sbin/route del $SNS_VIP >/dev/null 2>&1

echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce

echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore

echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce

;;

*)

        echo "Usage:$0 {start|stop}"

        exit 1

esac

exit 0

 

在三臺機器上逐一使用realserver.sh start啓動該腳本

 

4、  測試

使用http://192.168.75.188:8080能夠訪問到tomcat,中止hadoop01或者hadoop02中的任意一臺機器都不影響http://192.168.75.188:8080這個入口,中止hadoop03、hadoop0四、hadoop05中的任意一臺或者任意兩臺都不影響web服務的訪問

 

5、  補充

 

5.1    keepalived的調試

         Keepalived的安裝相對而言難一點,遇到的問題較多。安裝並不困難,困難在啓動以後會遇到一些異常,在排查這些異常的時候,須要使用一些適合的方法,能夠從/var/log/message文件中查看keepalived啓動後的輸出日誌,從而判斷是否運行正常

         Keepalived啓動以後,可使用ip addr命令查看VIP是否已經正確加載到了對應的網卡上,注意不要使用ifconfig –a命令,ifconfig命令沒法查看到VIP的信息

         Keepalived啓動異常的緣由有很大多是keepalived.conf配置文件配置不正確,能夠經過精簡配置文件來逐一排查。好比先去掉virtualserver的部分配置,只保留VIP的配置,以下:

 

! Configuration File for keepalived

 

global_defs {               

   notification_email {

     sadgump@163.com                                //能夠是虛假的email地址,但要配置

   }

   notification_email_from  sadgump@163.com          //能夠是虛假的email地址,但要配置

   smtp_server 192.168.200.1                        //能夠是虛假的IP地址,但必定要配置

   smtp_connect_timeout 30                          //默認

   router_id LVS_DEVEL                                            //默認

}

 

vrrp_instance VI_1 {                       //定義一個VRRP實力對象  VI_1

    state MASTER                                  //hadoop01上設置爲MASTER,hadoop02上設置爲BACKUP

    interface eth0                                  //定義設置VIP地址的網卡名

    virtual_router_id 51                //默認

    priority 100           //hadoop01設爲100,hadoop02設爲99,數越小,級別越低

    advert_int 1          //默認

    authentication {            

        auth_type PASS               //默認

        auth_pass 1111              //默認

    }

    virtual_ipaddress {

        192.168.75.188               //VIP地址

    }

}

 

使用這個配置文件來運行keepalived,測試VIP是否能被正確添加到網卡上,若是能夠,再加上virtualserver的配置部分,經過這種手段能夠快速定位異常和錯誤

相關文章
相關標籤/搜索