by: 白馬公園/naritechlinux
本文的主旨在於配置一個儘量簡單的高可用WEB服務系統,幫助讀者理清keepalived、LVS等軟件的配置和使用,爲進一步用好LVS樹立信心,夯實基礎。nginx
本文只涉及使用LVS和Keepalived解決WEB服務的高可用和高併發,對於WEB集羣中的session共享即數據一致性問題不作討論,對WEB集羣中的session共享問題感興趣的朋友能夠參考個人另一篇文章《nginx、tomcat、redis配置session共享》web
網上關於LVS和keepalived搭建高可用WEB服務的文章不少,本文的不一樣之處在於筆者在配置的過程當中遇到了不少問題,於是走了很多彎路。因此本文的重點將放在這些問題的講述上。redis
1、 系統架構算法
全系統包括五臺服務器,均爲虛擬機,安裝Centos6.5操做系統,其中tomcat
hadoop01 192.168.75.88 安裝keepalived,做爲主負載調度器使用服務器
hadoop02 192.168.75.89 安裝keepalived,做爲備用負載調度器使用session
hadoop03 192.168.75.90 安裝tomcat,做爲真實Web服務器使用架構
hadoop04 192.168.75.91 安裝tomcat,做爲真實Web服務器使用併發
hadoop05 192.168.75.92 安裝tomcat,做爲真實Web服務器使用
VIP設定爲 192.168.75.188,正常運行時該IP被設置在hadoop01上,接受客戶端請求並按照負載分配策略挑選出一臺真實的服務器,並將請求分配給真實的服務器進行處理。當hadoop01上的keepalived退出運行時,VIP被設置到hadoop02上,hadoop02接管此前hadoop01的任務,繼續接受客戶端請求並提供負載調度服務
2、 keepalived的安裝配置(hadoop01和hadoop02均須要安裝配置)
2.1 keepalived版本選擇
最新的keepalived1.2.24對應Centos6.5操做系統在安裝上有些問題,make不能成功,不推薦使用Keepalived1.2.6版本安裝後運行異常,跟蹤日誌文件後沒法定位問題,也不推薦使用,這裏選擇keepalived1.2.19,安裝介質能夠在keepalived官網上下載
2.2 keepalived安裝前的準備
一、須要安裝C++編譯器。
C++編譯器的安裝請參考《nginx、tomcat、redis配置session共享》
二、須要安裝openSSL
rpm -ivh zlib-1.2.3-29.el6.x86_64.rpm
rpm -ivh zlib-devel-1.2.3-29.el6.x86_64.rpm
rpm -ivh libsepol-devel-2.0.41-4.el6.x86_64.rpm
rpm -ivh pkgconfig-0.23-9.1.el6.x86_64.rpm
rpm -ivh libcom_err-devel-1.41.12-18.el6.x86_64.rpm
rpm -ivh keyutils-libs-devel-1.4-4.el6.x86_64.rpm
rpm -ivh libselinux-devel-2.0.94-5.3.el6_4.1.x86_64.rpm
rpm -ivh krb5-devel-1.10.3-10.el6_4.6.x86_64.rpm
rpm -ivh openssl-1.0.1e-15.el6.x86_64.rpm
rpm -ivh openssl-devel-1.0.1e-15.el6.x86_64.rpm
三、須要安裝kernel-devel開發包
rpm -ivh kernel-devel-2.6.32-431.el6.x86_64.rpm
2.3 keepalived的configure
./configure --prefix=/usr/local/keepalived --with-kernel-dir=/usr/src/kernels/2.6.32-431.el6.x86_64 --disable-fwmark
當安裝了kernel-devel開發包以後,系統就有了/usr/src/kernels/2.6.32-431.el6.x86_64這樣的相似目錄,直接找到這個目錄便可
Configure成功後會顯示出如下內容:
Keepalived configuration
------------------------
Keepalived version : 1.2.19
Compiler : gcc
Compiler flags : -g -O2
Extra Lib : -lssl -lcrypto -lcrypt
Use IPVS Framework : Yes
IPVS sync daemon support : Yes
IPVS use libnl : No
fwmark socket support : No
Use VRRP Framework : Yes
Use VRRP VMAC : Yes
SNMP support : No
SHA1 support : No
Use Debug flags : No
特別要注意:Use IPVS Framework、IPVS sync daemon support、Use VRRP Framework、Use VRRP VMAC這四項必須爲Yes
2.4 keepalived的make
介質路徑下直接鍵入make,注意使用root用戶
make成功以後會顯示如下內容:
Make complete
make[1]: Leaving directory `/users/oracle/software/keepalived-1.2.19/genhash'
Make complete
2.5 keepalived的make install
介質路徑下直接鍵入make install,注意使用root用戶
make install成功以後會顯示如下內容:
make[1]: Leaving directory `/users/oracle/software/keepalived-1.2.19/keepalived'
make -C genhash install
make[1]: Entering directory `/users/oracle/software/keepalived-1.2.19/genhash'
install -d /usr/local/keepalived/bin
install -m 755 ../bin/genhash /usr/local/keepalived/bin/
install -d /usr/local/keepalived/share/man/man1
install -m 644 ../doc/man/man1/genhash.1 /usr/local/keepalived/share/man/man1
make[1]: Leaving directory `/users/oracle/software/keepalived-1.2.19/genhash'
2.6 安裝後的系統級設置
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
chmod 777 /etc/init.d/keepalived
cd /etc/init.d
chkconfig --add keepalived
chkconfig keepalived on
mkdir –p /etc/keepalived
touch /etc/keepalived/keepalived.conf
2.7 keepalived的配置
! Configuration File for keepalived
global_defs {
notification_email {
sadgump@163.com //能夠是虛假的email地址,但要配置
}
notification_email_from sadgump@163.com //能夠是虛假的email地址,但要配置
smtp_server 192.168.200.1 //能夠是虛假的IP地址,但必定要配置
smtp_connect_timeout 30 //默認
router_id LVS_DEVEL //默認
}
vrrp_instance VI_1 { //定義一個VRRP實力對象 VI_1
state MASTER //hadoop01上設置爲MASTER,hadoop02上設置爲BACKUP
interface eth0 //定義設置VIP地址的網卡名
virtual_router_id 51 //默認
priority 100 //hadoop01設爲100,hadoop02設爲99,數越小,級別越低
advert_int 1 //默認
authentication {
auth_type PASS //默認
auth_pass 1111 //默認
}
virtual_ipaddress {
192.168.75.188 //VIP地址
}
}
//VIP地址的監聽端口,該端口要和實際服務器的監聽端口一致,不然會不能正常轉發
virtual_server 192.168.75.188 8080 {
delay_loop 6 //默認
lb_algo rr //負載調度算法設置
lb_kind DR //負載均衡轉發:使用效率最高的DR算法
nat_mask 255.255.255.0 //默認
persistence_timeout 50 //默認
protocol TCP //轉發支持的協議
real_server 192.168.75.90 8080 { //實時服務器一的地址及端口
weight 1 //權重
TCP_CHECK {
connect_timeout 3 //默認
nb_get_retry 3 //默認
delay_before_retry 3 //默認
}
}
real_server 192.168.75.91 8080 { //實時服務器二的地址及端口
weight 1 //權重
TCP_CHECK {
connect_timeout 3 //默認
nb_get_retry 3 //默認
delay_before_retry 3 //默認
}
}
real_server 192.168.75.92 8080 { //實時服務器三的地址及端口
weight 1 //權重
TCP_CHECK {
connect_timeout 3 //默認
nb_get_retry 3 //默認
delay_before_retry 3 //默認
}
}
}
3、 實際服務器的軟件安裝和配置
實際服務器有三臺即hadoop0三、hadoop0四、hadoop05,均須要安裝和配置
3.1 tomcat的安裝
略
在三臺實際服務器上啓動tomcat,監聽端口爲8080
3.2 IPVS相關配置
編輯一個realserver.sh腳本,chmod 777 realserver.sh
內容:
SNS_VIP=192.168.75.188
source /etc/rc.d/init.d/functions
case "$1" in
start)
ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP up
/sbin/route add -host $SNS_VIP dev lo:0
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p >/dev/null 2>&1
echo "RealServer Start OK"
;;
stop)
ifconfig lo:0 down
/sbin/route del $SNS_VIP >/dev/null 2>&1
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
;;
*)
echo "Usage:$0 {start|stop}"
exit 1
esac
exit 0
在三臺機器上逐一使用realserver.sh start啓動該腳本
4、 測試
使用http://192.168.75.188:8080能夠訪問到tomcat,中止hadoop01或者hadoop02中的任意一臺機器都不影響http://192.168.75.188:8080這個入口,中止hadoop03、hadoop0四、hadoop05中的任意一臺或者任意兩臺都不影響web服務的訪問
5、 補充
5.1 keepalived的調試
Keepalived的安裝相對而言難一點,遇到的問題較多。安裝並不困難,困難在啓動以後會遇到一些異常,在排查這些異常的時候,須要使用一些適合的方法,能夠從/var/log/message文件中查看keepalived啓動後的輸出日誌,從而判斷是否運行正常
Keepalived啓動以後,可使用ip addr命令查看VIP是否已經正確加載到了對應的網卡上,注意不要使用ifconfig –a命令,ifconfig命令沒法查看到VIP的信息
Keepalived啓動異常的緣由有很大多是keepalived.conf配置文件配置不正確,能夠經過精簡配置文件來逐一排查。好比先去掉virtualserver的部分配置,只保留VIP的配置,以下:
! Configuration File for keepalived
global_defs {
notification_email {
sadgump@163.com //能夠是虛假的email地址,但要配置
}
notification_email_from sadgump@163.com //能夠是虛假的email地址,但要配置
smtp_server 192.168.200.1 //能夠是虛假的IP地址,但必定要配置
smtp_connect_timeout 30 //默認
router_id LVS_DEVEL //默認
}
vrrp_instance VI_1 { //定義一個VRRP實力對象 VI_1
state MASTER //hadoop01上設置爲MASTER,hadoop02上設置爲BACKUP
interface eth0 //定義設置VIP地址的網卡名
virtual_router_id 51 //默認
priority 100 //hadoop01設爲100,hadoop02設爲99,數越小,級別越低
advert_int 1 //默認
authentication {
auth_type PASS //默認
auth_pass 1111 //默認
}
virtual_ipaddress {
192.168.75.188 //VIP地址
}
}
使用這個配置文件來運行keepalived,測試VIP是否能被正確添加到網卡上,若是能夠,再加上virtualserver的配置部分,經過這種手段能夠快速定位異常和錯誤