1、爲何要使用負載均衡技術?html
一、系統高可用性node
二、 系統可擴展性nginx
三、 負載均衡能力web
LVS+keepalived能很好的實現以上的要求,LVS提供負載均衡,keepalived提供健康檢查,故障轉移,提升系統的可用性!採用這樣的架構之後很容易對現有系統進行擴展,只要在後端添加或者減小realserver,只要更改lvs的配置文件,並能實現無縫配置變動!算法
2、LVS+Keepalived介紹vim
一、 LVS後端
LVS是一個開源的軟件,能夠實現LINUX平臺下的簡單負載均衡。LVS是Linux Virtual Server的縮寫,意思是Linux虛擬服務器。目前有三種IP負載均衡技術(VS/NAT、VS/TUN和VS/DR);八種調度算法(rr,wrr,lc,wlc,lblc,lblcr,dh,sh)。瀏覽器
二、 keepalivedbash
Keepalived 是運行在lvs 之上,它的主要功能是實現真實機的故障隔離及負載均衡器間的失敗切換,提升系統的可用性服務器
實驗環境:Centos6
服務器地址分配:Master Server:192.168.214.129 VIP:192.168.214.10
Backup Server :192.168.214.130 VIP:192.168.214.10
Web Server 1 : 192.168.214.131
Web Server 2 : 192.168.214.132
三:負載層安裝:
負載層使用LVS實現負載均衡功能,Keepalived實現主備高可用功能,所以在Master Server, Backup Server 上同時安裝LVS,Keepalied。LVS,Keepalied的安裝包能夠到官網下載獲取。也可使用yum安裝,實驗使用yum安裝。
1. Master Server, Backup Server 上安裝 ipvsadm:
yum install ipvsadm -y
2. Master Server, Backup Server 上安裝Keepalived(keepalived官網:http://www.keepalived.org/,能夠編譯安裝,實驗使用yum安裝)
yum install keepalived -y
當LVS與keepalived結合使用時,就須要用ipvsadm指令添加負載均衡的規則 ,直接在keepalived.conf的配置文件中配置便可,很是方便。
四:Web server端的配置:
負載均衡層安裝好後遲點再配置,先把後端服務安裝好後再配置keepalived,這裏Web Server 只安裝nginx測試。過程略過。
須要注意的是,Web server端也須要進行虛擬ip的配置。而且須要抑制APR通訊。提供以下執行腳本realserver.sh:
#!/bin/bash # description: Config realserver lo and apply noarp SNS_VIP=192.168.214.10 /etc/rc.d/init.d/functions case "$1" in start) ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP /sbin/route add -host $SNS_VIP dev lo:0 echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce sysctl -p >/dev/null 2>&1 echo "RealServer Start OK" ;; stop) ifconfig lo:0 down route del $SNS_VIP >/dev/null 2>&1 echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce echo "RealServer Stoped" ;; *) echo "Usage: $0 {start|stop}" exit 1 esac exit 0
上面的腳本分別在Web Server上執行。執行後查看
ifconfig指令顯示以下:
[root@anan ~]# ifconfig eth0 Link encap:Ethernet HWaddr 00:0C:29:2A:64:95 inet addr:192.168.214.132 Bcast:192.168.214.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fe2a:6495/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:22128 errors:0 dropped:0 overruns:0 frame:0 TX packets:9775 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3186564 (3.0 MiB) TX bytes:2538536 (2.4 MiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) lo:0 Link encap:Local Loopback inet addr:192.168.214.10 Mask:255.255.255.255 UP LOOPBACK RUNNING MTU:65536 Metric:1
關於arp_ignore和arp_announce的做用:(屬於內核參數,存放路徑:/proc/sys/net/ipv4/conf/all/)
arp_ignore和arp_announce做用:(存放路徑:/proc/sys/net/ipv4/conf/all/)
arp_ignore:定義接收到arp請求時的響應級別
(默認)0:只要本地有相應地址就響應,不論在哪一個接口上
1:僅在請求目的地址匹配請求到達的接口上的時候才響應
arp_announce:定義將本身地址向外通告時的通告級別
(默認) 0:將本機上任意接口上的任意地址向外通告
1:試圖將目標網絡通告與其網絡匹配的地址
2:僅向與本地接口上匹配的網絡進行通告
能夠將上面的腳本設置開機自動執行:在 /etc/rc.d/rc.local 中添加執行命令:
vim /etc/rc.d/rc.local #!/bin/sh # # This script will be executed *after* all the other init scripts. # You can put your own initialization stuff in here if you don't # want to do the full Sys V style init stuff. touch /var/lock/subsys/local sh /etc/init.d/realserver.sh start 2>&1 > /dev/null
查看ARP設置:
[root@anan ~]# cat /proc/sys/net/ipv4/conf/lo/arp_ignore 1 [root@anan ~]# cat /proc/sys/net/ipv4/conf/lo/arp_announce 2 [root@anan ~]#
測試後端服務是否正常:
五:配置keepalied.conf 配置文件:
vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { acassen@firewall.loc ##配置管理員郵箱,用於報警,一行設置一個,能夠設置多個,系統須要開啓sendmail 服務 failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc #設置郵件的發送地址 smtp_server 192.168.200.1 #設置smtp server地址 smtp_connect_timeout 30 #設置鏈接smtp服務超時時間 router_id LVS_DEVEL #路由ID,和BACKUP 要不一樣,否則會報錯 } vrrp_instance VI_1 { state MASTER #指定Keepalived的角色,MASTER表示此主機是主用服務器,BACKUP表示是備用服務器。 interface eth0 #指定檢測網絡的接口
virtual_router_id 51 #虛擬路由標識,即同一個vrrp_instance下,MASTER和BACKUP必須是一致的。 priority 100 #定義優先級,數字越大,優先級越高,在一個vrrp_instance下,MASTER的優先級必須大於BACKUP的優先級。 advert_int 1 #設定MASTER與BACKUP負載均衡器之間同步檢查的時間間隔,單位是秒。 authentication { #設定驗證類型和密碼
auth_type PASS #設置驗證類型,主要有PASS和AH兩種
auth_pass 1111 #設置驗證密碼,在一個vrrp_instance下,MASTER與BACKUP必須使用相同的密碼才能正常通訊。
}
}
virtual_ipaddress {
192.168.214.10 #設置虛擬IP地址,能夠設置多個虛擬IP地址,每行一個。
}
}
virtual_server 192.168.214.10 80 { #設置虛擬服務器,須要指定虛擬ip地址和服務端口,ip與端口之間用空格隔開。
delay_loop 6 #設置健康檢查時間,單位是秒。
lb_algo rr #設置負載調度算法,這裏設置爲rr,即輪詢算法
lb_kind DR #設置LVS實現負載均衡的機制,能夠有NAT、TUN和DR三個模式可選。
nat_mask 255.255.255.0
persistence_timeout 2 #會話保持時間,單位是秒,默認是50,這個會話保持功能,用戶的請求會被一直分發到某個服務節點,直到超過這個會話保持時間。需 要注意的是,這個會話保持時間,是最大無響應超時時間,也就是說用戶在操做動態頁面時,若是在2秒內沒有執行任何操 做,那麼接下來的操做會被分發到另外節點,可是若是一直在操做動態頁面,則不受2秒的時間限制。因爲在本次屬於實驗性 質,須要看到輪詢功能的狀況,因此將此值調整得2秒這麼小。
protocol TCP #指定轉發協議類型,有tcp和udp兩種。
real_server 192.168.214.131 80 { #配置服務節點1,須要指定realserver的真實IP地址和端口,ip與端口之間用空格隔開。
weight 1 #配置服務節點的權值,權值大小用數字表示,數字越大,權值越高,設置權值的大小能夠爲不一樣性能的服務器分配不一樣的負載,可 以對性能高的服務器設置較高的權值,而對性能較低的服務器設置相對較低的權值,這樣就合理的利用和分配了系統資源。
HTTP_GET { #realserve的狀態檢測設置部分,單位是秒
url {
path /index.html #檢查首頁文件
status_code 200 #請求狀態碼是200
}
connect_timeout 3 #請求超時時間 3 秒
nb_get_retry 3 #重試次數
delay_before_retry 3 #重試間隔時間
}
}
real_server 192.168.214.132 80 { #配置服務節點2
weight 1
HTTP_GET {
url {
path /index.html
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
保存退出。將改配置文件cp到backup server中,使用修改上面標紅兩處便可。
將 state MASTER 改成 BACKUP (字母必需要大寫)
將 priority 100 改成 99 (值必需要比MASTER的值小)
六:啓動測試:
MASTER 和 BACKUP 分別啓動keepalived服務
service keepalived start
而後查看LVS狀態:
[root@booktest ~]# ipvsadm -l -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.214.10:80 rr persistent 2 -> 192.168.214.131:80 Route 1 0 0 -> 192.168.214.132:80 Route 1 0 0 [root@booktest ~]#
狀態正常,查看keepalived啓動的日誌
[root@booktest ~]# tail /var/log/messages
Oct 21 15:36:37 booktest Keepalived_vrrp[2095]: Opening file '/etc/keepalived/keepalived.conf'.
Oct 21 15:36:37 booktest Keepalived_vrrp[2095]: Configuration is using : 63498 Bytes
Oct 21 15:36:37 booktest Keepalived_vrrp[2095]: Using LinkWatch kernel netlink reflector...
Oct 21 15:36:37 booktest Keepalived_vrrp[2095]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Oct 21 15:36:38 booktest Keepalived_vrrp[2095]: VRRP_Instance(VI_1) Transition to MASTER STATE
Oct 21 15:36:39 booktest Keepalived_vrrp[2095]: VRRP_Instance(VI_1) Entering MASTER STATE
Oct 21 15:36:39 booktest Keepalived_vrrp[2095]: VRRP_Instance(VI_1) setting protocol VIPs.
Oct 21 15:36:39 booktest Keepalived_vrrp[2095]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.214.10
Oct 21 15:36:39 booktest Keepalived_healthcheckers[2094]: Netlink reflector reports IP 192.168.214.10 added
Oct 21 15:36:44 booktest Keepalived_vrrp[2095]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.214.10
[root@booktest ~]#
能夠看到啓動成功,而且狀態爲MASTER狀態。接下來執行service keepalived start 啓動BACKUP SERVER而且查看日誌信息:
[root@node1 ~]# tail /var/log/messages
Oct 21 23:41:36 node1 Keepalived_vrrp[3055]: Netlink reflector reports IP 192.168.214.130 added
Oct 21 23:41:36 node1 Keepalived_vrrp[3055]: Netlink reflector reports IP fe80::20c:29ff:fe93:4905 added
Oct 21 23:41:36 node1 Keepalived_vrrp[3055]: Registering Kernel netlink reflector
Oct 21 23:41:36 node1 Keepalived_vrrp[3055]: Registering Kernel netlink command channel
Oct 21 23:41:36 node1 Keepalived_vrrp[3055]: Registering gratuitous ARP shared channel
Oct 21 23:41:36 node1 Keepalived_vrrp[3055]: Opening file '/etc/keepalived/keepalived.conf'.
Oct 21 23:41:36 node1 Keepalived_vrrp[3055]: Configuration is using : 63500 Bytes
Oct 21 23:41:36 node1 Keepalived_vrrp[3055]: Using LinkWatch kernel netlink reflector...
Oct 21 23:41:36 node1 Keepalived_vrrp[3055]: VRRP_Instance(VI_1) Entering BACKUP STATE
Oct 21 23:41:36 node1 Keepalived_vrrp[3055]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
能夠看到BACKUP SERVER主機啓動服務後自動成爲備用狀態。說明keepalived.conf的vrrp_instance配置是正確的。
這時在瀏覽器輸入VIP地址:192.168.214.10測試
在進行負載均衡測試的時候遇到了一個問題:就是不停的刷新頁面頁面都沒有變化,沒有調用web server 2 的頁面,要過很長時間纔會輪詢一次。這樣顯然不合理。網上查了一下是LVS 有持久鏈接的特性。就是keepalived.conf中 persistence_timeout 2 指定的值。上面解釋中提到這參數的做用。
同時還須要查看LVS的默認超時時間:
[root@booktest ~]# ipvsadm -l --timeout Timeout (tcp tcpfin udp): 900 120 300 [root@booktest ~]#
900 120 300這三個數值分別是TCP TCPFINUDP的時間.也就是說一條tcp的鏈接通過lvs後,lvs會把這臺記錄保存15分鐘,就是由於這個時間過長,
因此大部分人都會發現作好LVS DR以後輪詢現象並無發生,查看不多文檔並無提到這一點,有點暈。由於是實驗性質,因此將此數值調整爲很是小,
以便看到輪詢效果。
執行指令調整這個值:
[root@booktest ~]# ipvsadm --set 1 2 1
再查看
[root@booktest ~]# ipvsadm -l --timeout
Timeout (tcp tcpfin udp): 1 2 1
[root@booktest ~]#
保存一下:
[root@booktest ~]# ipvsadm --save
-A -t 192.168.214.10:http -s rr -p 2
-a -t 192.168.214.10:http -r 192.168.214.131:http -g -w 1
-a -t 192.168.214.10:http -r 192.168.214.132:http -g -w 1
接下來繼續測試:
隔兩秒刷新頁面以下:
反覆幾回輪詢調用正常。
七:模擬故障
故障前先看一下ipvsadm狀態:
[root@booktest ~]# ipvsadm -l -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.214.10:80 rr persistent 2 -> 192.168.214.131:80 Route 1 0 0 -> 192.168.214.132:80 Route 1 0 0 [root@booktest ~]#
將192.168.214.131這臺Web Server down掉模擬故障。直接粗暴點殺掉nginx:pkill nginx
頁面正常訪問。再看ipvsadm狀態:
[root@booktest ~]# ipvsadm -l -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.214.10:80 rr persistent 2 -> 192.168.214.131:80 Route 1 0 0 [root@booktest ~]#
能夠看到192.168.21.132節點的Web Server已經掉線了
接下來,測試負載均衡的主備切換。先關掉MASTER的keepalived服務模擬故障,而後查看BACKUP 的狀態
[root@booktest ~]# service keepalived stop 中止 keepalived: [肯定] [root@booktest ~]# tail /var/log/messages Oct 21 16:10:43 booktest Keepalived_healthcheckers[2094]: Error connecting server [192.168.214.132]:80. Oct 21 16:10:43 booktest Keepalived_healthcheckers[2094]: Removing service [192.168.214.132]:80 from VS [192.168.214.10]:80 Oct 21 16:10:43 booktest Keepalived_healthcheckers[2094]: Remote SMTP server [192.168.200.1]:25 connected. Oct 21 16:11:04 booktest Keepalived_healthcheckers[2094]: Error reading data from remote SMTP server [192.168.200.1]:25. Oct 21 16:19:27 booktest kernel: IPVS: __ip_vs_del_service: enter Oct 21 16:19:27 booktest Keepalived_vrrp[2095]: VRRP_Instance(VI_1) sending 0 priority Oct 21 16:19:27 booktest Keepalived_vrrp[2095]: VRRP_Instance(VI_1) removing protocol VIPs. Oct 21 16:19:27 booktest Keepalived[2093]: Stopping Keepalived v1.2.13 (03/19,2015) Oct 21 16:19:27 booktest Keepalived_healthcheckers[2094]: Netlink reflector reports IP 192.168.214.10 removed Oct 21 16:19:27 booktest Keepalived_healthcheckers[2094]: Removing service [192.168.214.131]:80 from VS [192.168.214.10]:80 [root@booktest ~]#
接着查看BACKUP的狀態:
[root@node1 ~]# tail /var/log/messages Oct 22 00:11:03 node1 Keepalived_healthcheckers[3053]: Error reading data from remote SMTP server [192.168.200.1]:25. Oct 22 00:16:29 node1 dhclient[989]: DHCPREQUEST on eth0 to 192.168.214.254 port 67 (xid=0x37dffe90) Oct 22 00:16:29 node1 dhclient[989]: DHCPACK from 192.168.214.254 (xid=0x37dffe90) Oct 22 00:16:31 node1 dhclient[989]: bound to 192.168.214.130 -- renewal in 806 seconds. Oct 22 00:19:26 node1 Keepalived_vrrp[3055]: VRRP_Instance(VI_1) Transition to MASTER STATE Oct 22 00:19:27 node1 Keepalived_vrrp[3055]: VRRP_Instance(VI_1) Entering MASTER STATE Oct 22 00:19:27 node1 Keepalived_vrrp[3055]: VRRP_Instance(VI_1) setting protocol VIPs. Oct 22 00:19:27 node1 Keepalived_healthcheckers[3053]: Netlink reflector reports IP 192.168.214.10 added Oct 22 00:19:27 node1 Keepalived_vrrp[3055]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.214.10 Oct 22 00:19:32 node1 Keepalived_vrrp[3055]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.214.10 [root@node1 ~]#
能夠看到原來BACKUP已經接替了MASTER,成爲了MASTER。
而後再從新啓動MASTER的keepalived服務,重啓的MASTER會自動成爲MASTER,而原來的BACKUP會從MASTER變爲BACKUP
[root@booktest ~]# service keepalived start 正在啓動 keepalived: [肯定] [root@booktest ~]# tail /var/log/messages Oct 21 16:29:43 booktest Keepalived_healthcheckers[2256]: Using LinkWatch kernel netlink reflector... Oct 21 16:29:43 booktest Keepalived_healthcheckers[2256]: Activating healthchecker for service [192.168.214.131]:80 Oct 21 16:29:43 booktest Keepalived_healthcheckers[2256]: Activating healthchecker for service [192.168.214.132]:80 Oct 21 16:29:43 booktest Keepalived_vrrp[2257]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)] Oct 21 16:29:44 booktest Keepalived_vrrp[2257]: VRRP_Instance(VI_1) Transition to MASTER STATE Oct 21 16:29:44 booktest Keepalived_vrrp[2257]: VRRP_Instance(VI_1) Received lower prio advert, forcing new election Oct 21 16:29:45 booktest Keepalived_vrrp[2257]: VRRP_Instance(VI_1) Entering MASTER STATE Oct 21 16:29:45 booktest Keepalived_vrrp[2257]: VRRP_Instance(VI_1) setting protocol VIPs. Oct 21 16:29:45 booktest Keepalived_healthcheckers[2256]: Netlink reflector reports IP 192.168.214.10 added Oct 21 16:29:45 booktest Keepalived_vrrp[2257]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.214.10 [root@booktest ~]#
再看原來BACKUP主機狀態:
[root@node1 ~]# tail /var/log/messages Oct 22 00:19:27 node1 Keepalived_healthcheckers[3053]: Netlink reflector reports IP 192.168.214.10 added Oct 22 00:19:27 node1 Keepalived_vrrp[3055]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.214.10 Oct 22 00:19:32 node1 Keepalived_vrrp[3055]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.214.10 Oct 22 00:29:42 node1 Keepalived_vrrp[3055]: VRRP_Instance(VI_1) Received higher prio advert Oct 22 00:29:42 node1 Keepalived_vrrp[3055]: VRRP_Instance(VI_1) Entering BACKUP STATE Oct 22 00:29:42 node1 Keepalived_vrrp[3055]: VRRP_Instance(VI_1) removing protocol VIPs. Oct 22 00:29:42 node1 Keepalived_healthcheckers[3053]: Netlink reflector reports IP 192.168.214.10 removed Oct 22 00:29:57 node1 dhclient[989]: DHCPREQUEST on eth0 to 192.168.214.254 port 67 (xid=0x37dffe90) Oct 22 00:29:57 node1 dhclient[989]: DHCPACK from 192.168.214.254 (xid=0x37dffe90) Oct 22 00:29:59 node1 dhclient[989]: bound to 192.168.214.130 -- renewal in 793 seconds. [root@node1 ~]#
BACKUP主機從新從MASTER狀態變成BACKUP狀態。繼續刷新頁面正常。
Lvs+Keepalived高可用負載均衡實驗成功。接下來將會對web server進行部署優化。