日期及版本:2014.5.4v1.0javascript
架構圖php
實驗目的:css
1.Haproxy+Keepalived雙主雙機高可用模型,keepalived爲Haproxy主從提供高可用保證haproxy-master若掛掉haproxy-backup能無縫接管,haproxy爲後端Web提供負載均衡,緩解併發壓力,實現WEB站點負載均衡+高可用性;html
2. Haproxy反代web作動靜分離;前端
3. Haproxy反代mysql 算法leastconn和roundrobin的不一樣效果;java
系統環境:mysql
OS:centos6.4 X64web
apache: httpd-2.2.15-26.el6.centos.x86_64redis
haproxy: haproxy-1.4.22-3.el6.x86_64算法
keepalived:keepalived-1.2.7
基礎配置:
Role |
Out-IP |
In-IP |
Host |
haproxy VIP: | 192.168.1.10 | ||
haproxy-master(haproxy1): | 192.168.1.11 | 172.16.100.254 | ha1.example.com |
haproxy-backup(haproxy2): | 192.168.1.12 | 172.16.100.253 | ha2.example.com |
apache1: | 172.16.100.13 | web1.example.com | |
apache2: | 172.16.100.14 | web2.example.com |
實驗步驟:由於都是雙機配置,重複內容合併
1.一、配置後端web1,web2按IP修改
[root@web1 ~]# hostname web1
[root@web1 ~]# ifconfig eth0 172.16.100.13/24 up
[root@web1 ~]# route add default gw 172.16.100.254
[root@web1 ~]# route -n
[root@web1 ~]# yum -y install httpd
[root@web1 ~]# echo '<h1>web1.example.com</h1>' > /var/www/html/index.html
[root@web1 ~]# service httpd start
[root@web1 ~]# curl http://172.16.100.13
1.二、安裝php
[root@web1 ~]# yum -y install php
[root@ web1 ~]# cd /var/www/html/
[root@ web1 html]# cp index.html index.php
[root@web1 html]# cat >> index.php <<EOF
<?php phpinfo(); ?> EOF
[root@web1 html]# service httpd restart
二、配置Haproxy反向代理
[root@station11 ~]# hostname ha1
[root@ha1 ~]# ifconfig eth0 192.168.1.11/24 up
[root@ha1 ~]# ifconfig eth1 172.16.100.254/24 up
[root@ha1 ~]# route add default gw 192.168.1.1
[root@ha1 ~]# sed -i '/net.ipv4.ip_forward/ s/\(.*= \).*/\11/' /etc/sysctl.conf#打開轉發功能
net.ipv4.ip_forward = 1
[root@ha1 ~]# yum -y install haproxy
[root@ha1 ~]# cd /etc/haproxy/
[root@ha1 haproxy]# cp haproxy.cfg haproxy.cfg.bak
用rsyslog的理由:
1.防止系統崩潰沒法獲取系統日誌分享崩潰緣由,用rsyslog能夠把日誌傳輸到遠程的日誌服務器上
2.使用rsyslog日誌能夠減輕系統壓力,由於使用rsyslog能夠有效減輕系統的磁盤IO
3.rsyslog使用tcp傳輸很是可靠,能夠對日誌進行過濾,提取出有效的日誌,rsyslog是輕量級的日誌軟件,在大量日誌寫的狀況下,系統負載基本上在0.1如下
[root@ha1 haproxy]# yum -y install rsyslog
[root@ ha1 haproxy ~]# vim /etc/rsyslog.conf
啓用接受遠程日誌功能 去註釋 # Provides UDP syslog reception $ModLoad imudp $UDPServerRun 514 基於UDP接收日誌請求 rsyslog默認會把以下信息發往 *.info;mail.none;authpriv.none;cron.none; /var/log/messages 修改上行爲下行,添加local2.none不記錄到 *.info;mail.none;authpriv.none;cron.none;local2.none; /var/log/messages 添加一行,日誌記錄到 local2.* /var/log/haproxy.log
重啓服務
[root@ha1 haproxy]# service rsyslog restart
三、雙機公共設置ha1與ha2相同設置
[root@ha1 haproxy]# vim haproxy.cfg
global log 127.0.0.1 local2 #以日誌服務器形式發往本機地址,設施名local2 chroot /var/lib/haproxy #haproxy工做目錄切換到假根目錄,這裏默認 pidfile /var/run/haproxy.pid maxconn 4000 #每一個haproxy單進程所接受的最大併發鏈接數 user haproxy group haproxy daemon #haproxy以守護進程的方式工做於後臺 stats socket /var/lib/haproxy/stats #統計數據保存位置 defaults mode http #指定frontend和backend工做模式{tcp|http|health},代理後端web站點用http模式 log global #日誌使用全局中定義的日誌參數 option httplog #啓用http的log,啓用對http請求會話計時捕獲到cookie的日誌,默認原格式簡陋,後面還可跟參數[clf] clf格式 option dontlognull #不記錄空信息,保證HAProxy不記錄上級負載均衡發送過來的用於檢測狀態沒有數據的心跳包。 option http-server-close #啓用http鏈接在服務器端關閉功能,支持客戶端一側長鏈接 option forwardfor except 127.0.0.0/8 # forwardfor將用戶請求轉發後端服時,在HTTP請求報文中添加"X-Forwarded-For"特殊首部,以便後端服記錄真實發起請求的客戶端IP地址,而不是代理服務器內網卡地址。 option redispatch #當原分配用戶請求的後端服故障時,容許把用戶請求從新分發給其餘後端服 option abortonclose #當Haproxy服務器負載很高的時候,自動結束掉當前隊列處理比較久的鏈接 retries 3 #對後端服鏈接失敗後的重連次數 timeout http-request 10s #http請求超時時間10秒 timeout queue 1m #後端有多個服務器,當每一個後端服務器都達到最大鏈接上限,haproxy等待發送到對應後端服務器的隊列已滿或請求已入列但未處理的超時時間1分種 timeout connect 10s #haproxy向後端服務器請求創建鏈接的超時時間 timeout client 1m #發起鏈接請求的前端客戶端鏈接處於非活動態的最大超時時間,過期斷開,至關於Apache的timeout keepalive timeout server 1m #服務器端鏈接處於非活動態的最大超時時間 timeout http-keep-alive 10s #長鏈接超時時間 timeout check 10s #作健康狀態檢測的超時時間 maxconn 3000 #最大鏈接數 listen stats mode http bind *:8080 #綁定在特殊端口8080 stats enable #啓動status管理界面 stats hide-version #status隱藏haproxy版本信息 stats uri /haproxyadmin?stats #status訪問路徑 stats realm Haproxy\ Statistics #status登錄驗證信息 stats auth admin:admin #status頁面登錄用戶名或密碼 stats admin if TRUE #經過驗證才能管理後端,必須加判斷條件,不然語法錯誤 frontend webservers bind *:80 mode http log global option httpclose #每次請求完畢後主動關閉http通道 option logasap #傳輸大文件時能夠提早記錄日誌 option dontlognull #不記錄空信息,保證HAProxy不記錄上級負載均衡發送過來的用於檢測狀態沒有數據的心跳包 #################HAProxy的日誌記錄內容設置################### capture request header Host len 20 #只記錄Host這首部值的前20字節 capture request header X-Forwarded-For len 15 #記錄發起請求客戶端的IP地址,IP地址通常就15個字節。 capture request header Referer len 60 #Referrer記錄點擊連接時所在頁面的引用位置 ################url_static配合if使用實現動靜分離######################### acl url_static path_beg -i /static /p_w_picpaths /javascript /stylesheets #path_begin路徑以XX開頭,-i忽略大小寫 acl url_static path_end -i .html .jpg .jpeg .gif .png .css .js #path_end路徑以XX結尾,同名acl邏輯或關係 use_backend static_servs if url_static #知足ACL url_static 使用backend static_servs default_backend dynamic_servs #調度算法動靜區別在於調整配置文件haproxy.cfg,動態調整reload生效,靜態調整restart生效 #輪調平均分配訪問到後端服,訪問動態頁面須要保持會話所以source consistent源地址一致性hash算法把來自於同一客戶端請求始終轉發於同一臺後端服 backend static_servs balance roundrobin #動態輪調 server static1 172.16.100.13:80 check maxconn 6000 #此處server名static1是後端服別名,關鍵是IP地址 server static2 172.16.100.14:80 check maxconn 6000 backend dynamic_servs balance source #對源IP地址進行哈希,hash-type決定動態靜態 hash-type consistent #hash類型 map-based圖位靜態 consistent-hash一致性hash動態 server dynamic1 172.16.100.13:80 check maxconn 1000 #根據服務器內容自定義最大鏈接數 server dynamic2 172.16.100.14:80 check maxconn 1000
在web1,web2服務器配置中日誌格式添加"X-Forwarded-For"特殊首部,以便後端服務器記錄真實發起請求的客戶端IP地址
[root@web1 ~]# vim /etc/httpd/conf/httpd.conf
搜LogFormat 第一個字節段插入%{X-Forwarded-For}i
LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
[root@web1 ~]# sed -i '/^LogFormat.*combined/ s#"#"\%{X-Forwarded-For}\i#' /etc/httpd/conf/httpd.conf
[root@web1 html]# service httpd restart
[root@web2 html]# service httpd restart
三、驗證靜態頁面的負載均衡,動態頁面長鏈接保持於同一後端服務器,動靜分離。
Haproxy反代日誌會詳細顯示前端客戶端IP,Haproxy反代IP,根據動靜分離ACL選擇不一樣後端服務器,GET到不一樣頁面。後端upstream服務器別名
[root@ha1 haproxy]# tail /var/log/haproxy.log
May 4 01:15:25 localhost haproxy[1819]: 192.168.1.108:50854 [04/May/2014:01:15:25.571] webservers dynamic_servs/dynamic2 24/0/1/2/+27 200 +177 - - ---- 2/2/2/2/0 0/0 {192.168.1.11||http://192.168.1.11/} "GET /index.php?=PHPE9568F34-D428-11d2-A769-00AA001ACF42 HTTP/1.1" May 4 01:15:25 localhost haproxy[1819]: 192.168.1.108:50855 [04/May/2014:01:15:25.572] webservers dynamic_servs/dynamic2 25/0/0/2/+27 200 +177 - - ---- 1/1/1/1/0 0/0 {192.168.1.11||http://192.168.1.11/} "GET /index.php?=PHPE9568F35-D428-11d2-A769-00AA001ACF42 HTTP/1.1" May 4 01:15:28 localhost haproxy[1819]: 192.168.1.108:50856 [04/May/2014:01:15:28.552] webservers static_servs/static1 5/0/1/1/+7 304 +147 - - ---- 3/3/1/1/0 0/0 {192.168.1.11||} "GET /index.html HTTP/1.1" May 4 01:15:29 localhost haproxy[1819]: 192.168.1.108:50857 [04/May/2014:01:15:28.556] webservers static_servs/static2 799/0/1/1/+801 200 +265 - - ---- 2/2/1/1/0 0/0 {192.168.1.11||} "GET /index.html HTTP/1.1"
[root@ha2 haproxy]# tail /var/log/haproxy.log
May 4 01:38:30 localhost haproxy[1733]: 192.168.1.108:51208 [04/May/2014:01:38:30.436] webservers dynamic_servs/dynamic2 24/0/0/4/+28 200 +177 - - ---- 2/2/2/2/0 0/0 {192.168.1.12||http://192.168.1.12/} "GET /index.php?=PHPE9568F34-D428-11d2-A769-00AA001ACF42 HTTP/1.1" May 4 01:38:30 localhost haproxy[1733]: 192.168.1.108:51209 [04/May/2014:01:38:30.436] webservers dynamic_servs/dynamic2 24/0/0/4/+28 200 +177 - - ---- 2/2/2/1/0 0/0 {192.168.1.12||http://192.168.1.12/} "GET /index.php?=PHPE9568F35-D428-11d2-A769-00AA001ACF42 HTTP/1.1" May 4 01:39:04 localhost haproxy[1733]: 192.168.1.108:51219 [04/May/2014:01:39:04.178] webservers static_servs/static1 88/0/1/1/+90 200 +265 - - ---- 3/3/1/1/0 0/0 {192.168.1.12||} "GET /index.html HTTP/1.1" May 4 01:39:38 localhost haproxy[1733]: 192.168.1.108:51229 [04/May/2014:01:39:38.161] webservers static_servs/static2 8/0/2/0/+10 200 +265 - - ---- 3/3/1/1/0 0/0 {192.168.1.12||} "GET /index.html HTTP/1.1"
各個後端服務器日誌記錄前端客戶端IP,Haproxy反代內網卡(內網關)IP,Get到不一樣頁面,動態頁面還有haproxy反代外網卡IP
[root@web1 ~]# tail /var/log/httpd/access_log
192.168.1.108172.16.100.254 - - [04/May/2014:01:14:01 +0800] "GET /index.html HTTP/1.1" 304 - "-"
[root@web2 ~]# tail /var/log/httpd/access_log
192.168.1.108172.16.100.253 - - [04/May/2014:01:38:29 +0800] "GET /index.php?=PHPE9568F34-D428-11d2-A769-00AA001ACF42 HTTP/1.1" 200 2524 "http://192.168.1.12/" 192.168.1.108172.16.100.253 - - [04/May/2014:01:38:29 +0800] "GET /index.php?=PHPE9568F35-D428-11d2-A769-00AA001ACF42 HTTP/1.1" 200 2146 "http://192.168.1.12/"
不一樣http://192.168.1.1x:8080/haproxyadmin?stats訪問同一偵聽stats主機的指定url,端口通過身份驗證,顯示管理頁面。
配置keepalived主從服務器,主從相同實例高低相反
[root@ha1 ~]# yum -y install keepalived
[root@ha1 ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
global_defs { notification_email { #keepalived切換時發送到對象,一行一個 root@localhost } notification_email_from keepadmin@localhost #發件人 smtp_connect_timeout 3 #指定smtp超時時間 smtp_server 127.0.0.1 #指定smtp服務器地址 router_id LVS_DEVEL #VRRP組名,同屬一組的兩節點必須設置同樣 } vrrp_script chk_haproxy { script "killall -0 haproxy" interval 2 #腳本執行間隔時間 weight 2 #腳本結果致使的優先級變動:2表示優先級+2;-2則表示優先級-2 } vrrp_instance VI_1 { interface eth0 #綁定VIP的網卡 state MASTER #設置主節點MASTER priority 100 #主節點優先級(1-254),主必定要比備優先級高 virtual_router_id 173 #設置VRID,兩節點必須一致 garp_master_delay 1 #主從切換時間秒 authentication { auth_type PASS #驗證類型 auth_pass 1111 #驗證密碼 } track_interface { #設置額外的監控,裏面那個網卡出現問題都會切換 eth0 } virtual_ipaddress { 192.168.1.88/24 dev eth0 #設置VIP,兩節點必須一致 } track_script { chk_haproxy #腳本跟蹤監測 } } vrrp_instance VI_2 { interface eth0 #綁定VIP的網卡 state BACKUP #設置備節點BACKUP priority 99 #備節點優先級(1-254),主必定要比備優先級高 virtual_router_id 174 #設置VRID,兩節點必須一致 garp_master_delay 1 #主從切換時間秒 authentication { auth_type PASS #驗證類型 auth_pass 11111 #驗證密碼 } track_interface { #設置額外的監控,裏面那個網卡出現問題都會切換 eth0 } virtual_ipaddress { 192.168.1.188/24 dev eth0 #設置VIP,兩節點必須一致 } }
[root@ha1 keepalived]# service keepalived start
[root@ha2 keepalived]# vim keepalived.conf
global_defs { notification_email { root@localhost } notification_email_from keepadmin@localhost smtp_connect_timeout 3 smtp_server 127.0.0.1 router_id LVS_DEVEL } vrrp_script chk_haproxy { script "killall -0 haproxy" interval 2 weight 2 #腳本結果致使的優先級變動:2表示優先級+2;-2則表示優先級-2 } vrrp_instance VI_1 { interface eth0 #綁定VIP的網卡 state BACKUP #設置備節點 priority 99 #備節點優先級(1-254),主必定要比備優先級高 virtual_router_id 173 #設置VRID,兩節點必須一致 garp_master_delay 1 #主從切換時間秒 authentication { auth_type PASS auth_pass 1111 } track_interface { eth0 } virtual_ipaddress { 192.168.1.88/24 dev eth0 } track_script { chk_haproxy } } vrrp_instance VI_2 { interface eth0 state MASTER priority 100 virtual_router_id 174 garp_master_delay 1 authentication { auth_type PASS auth_pass 11111 } track_interface { eth0 } virtual_ipaddress { 192.168.1.188/24 dev eth0 } }
[root@ha1 ~]# ip addr | grep eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 inet 192.168.1.11/24 brd 192.168.1.255 scope global eth0 inet 192.168.1.88/24 scope global secondary eth0
[root@ha2 ~]# ip addr | grep eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 inet 192.168.1.12/24 brd 192.168.1.255 scope global eth0 inet 192.168.1.188/24 scope global secondary eth0
[root@ha1 ~]# tail /var/log/messages
May 4 04:49:26 station11 Keepalived_vrrp[3973]: VRRP_Instance(VI_1) Transition to MASTER STATE May 4 04:49:27 station11 Keepalived_vrrp[3973]: VRRP_Instance(VI_1) Entering MASTER STATE May 4 04:49:35 station11 Keepalived_vrrp[3973]: VRRP_Instance(VI_2) Entering BACKUP STATE [root@ha2 ~]# tail /var/log/messages May 4 04:49:35 station12 Keepalived_vrrp[3594]: VRRP_Instance(VI_1) Entering BACKUP STATE May 4 04:49:35 station12 Keepalived_vrrp[3594]: VRRP_Instance(VI_2) Transition to MASTER STATE May 4 04:49:36 station12 Keepalived_vrrp[3594]: VRRP_Instance(VI_2) Entering MASTER STATE
中止ha1服務器haproxy服務,keepalived進行從新選舉,ha2提高VI_1從備到主,VIP切換到備用服務器
[root@ha1 ~]# service haproxy stop
[root@ha2 ~]# tail /var/log/messages
May 4 04:56:00 station12 Keepalived_vrrp[3594]: VRRP_Instance(VI_1) forcing a new MASTER election May 4 04:56:01 station12 Keepalived_vrrp[3594]: VRRP_Instance(VI_1) Transition to MASTER STATE May 4 04:56:02 station12 Keepalived_vrrp[3594]: VRRP_Instance(VI_1) Entering MASTER STATE
[root@ha1 ~]# ip addr | grep eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 inet 192.168.1.11/24 brd 192.168.1.255 scope global eth0
VIP偏移到Keepalived backup機上
[root@ha2 ~]# ip addr | grep eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 inet 192.168.1.12/24 brd 192.168.1.255 scope global eth0 inet 192.168.1.188/24 scope global secondary eth0 inet 192.168.1.88/24 scope global secondary eth0
啓動ha1服務器haproxy服務,VIP從新切換回主服務器
[root@ha1 ~]# service haproxy start
[root@ha1 ~]# ip addr | grep eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 inet 192.168.1.11/24 brd 192.168.1.255 scope global eth0 inet 192.168.1.88/24 scope global secondary eth0
[root@ha2 ~]# ip addr | grep eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 inet 192.168.1.12/24 brd 192.168.1.255 scope global eth0 inet 192.168.1.188/24 scope global secondary eth0
[root@ha1 ~]# tail /var/log/messages
May 4 05:07:41 ha1 Keepalived_vrrp[3973]: VRRP_Instance(VI_1) forcing a new MASTER election May 4 05:07:42 ha1 Keepalived_vrrp[3973]: VRRP_Instance(VI_1) Transition to MASTER STATE May 4 05:07:43 ha1 Keepalived_vrrp[3973]: VRRP_Instance(VI_1) Entering MASTER STATE
[root@ha2 ~]# tail /var/log/messages
May 4 05:07:41 station12 Keepalived_vrrp[3594]: VRRP_Instance(VI_1) Entering BACKUP STATE
三、反代mysql 算法leastconn和roundrobin的不一樣效果?
[root@web1 ~]# yum -y install mysql mysql-server mysql-devel
[root@web1 ~]# service mysqld start
[root@web1 ~]# mysqladmin -u root password redhat
[root@web1 ~]# mysql -uroot -predhat
mysql> USE mysql; mysql> INSERT INTO user (Host,User) values ('172.16.100.254','mysqlchecker'); ##INSERT INTO user (Host,User) values ('<ip_of_haproxy>','<username>'); mysql> grant select on *.* to mysqlchecker@'172.16.100.254' identified by 'redhat'; mysql> flush privileges;
station14相同
[root@ha1 haproxy]# vim haproxy.cfg
註銷 defaults #option httplog #option http-server-close #option forwardfor except 127.0.0.0/8 添加 listen mysqlservers mode tcp #模式 TCP bind *:3306 #代理端口 balance roundrobin 輪調算法 option mysql-check user mysqlchecker #mysql健康檢查 mysqlchecker爲mysql登陸用戶名 server myserv1 172.16.100.13:3306 check inter 1500 rise 3 fall 3 weight 1 server myserv2 172.16.100.14:3306 check inter 1500 rise 3 fall 3 weight 1 #服務器定義:check inter 1500是檢測心跳頻率,rise 3是3次正確認爲服務器可用,fall 3是3次失敗認爲服務器不可用,weight表明權重
[root@ha1 haproxy]# service haproxy reload
[root@ha2 ~]# mysql -umysqlchecker -h192.168.1.11
mysql> select @@hostname;
| station14.example.com |
[root@ha2 ~]# mysql -umysqlchecker -h192.168.1.11
mysql> select @@hostname;
| station13.example.com |
[root@ha1 haproxy]# vim haproxy.cfg
balance leastconn 修改算法爲最少鏈接數,只考慮活動鏈接數,哪一個後端服務器的鏈接最少就調度到哪臺。
[root@ha1 haproxy]# service haproxy reload
用tcpdump捕捉來自不一樣源ip的tcp請求
[root@web1 ~]# tcpdump -i eth0 -s 0 -l -w out.log dst port 3306 | strings
[root@web2 ~]# tcpdump -i eth0 -s 0 -l -w out.log dst port 3306 | strings
使用不一樣客戶端訪問,
[root@station21 ~]# mysql -umysqlchecker -h192.168.1.11 -e "select @@hostname;"
| station13.example.com |
在每一個客戶端執行命令後馬上在對應mysql端運行以下命令
[root@web1 ~]# strings out.log | grep -i 'select @@hostmame'| wc -l
1
[root@station22 ~]# mysql -umysqlchecker -h192.168.1.11 -e "select @@hostname;"
| station14.example.com |
[root@web2 ~]# strings out.log | grep -i 'select @@hostmame'| wc -l
1
[root@station23 ~]# mysql -umysqlchecker -h192.168.1.11 -e "select @@hostname;"
| station13.example.com |
[root@web1 ~]# strings out.log | grep -i 'select @@hostmame'| wc -l
2
[root@station24 ~]# mysql -umysqlchecker -h192.168.1.11 -e "select @@hostname;"
| station13.example.com |
[root@web1 ~]# strings out.log | grep -i 'select @@hostmame'| wc -l
3
[root@station25 ~]# mysql -umysqlchecker -h192.168.1.11 -e "select @@hostname;"
| station14.example.com |
[root@web2 ~]# strings out.log | grep -i 'select @@hostmame'| wc -l
2
[root@station26 ~]# mysql -umysqlchecker -h192.168.1.11 -e "select @@hostname;"
| station14.example.com |
[root@web2 ~]# strings out.log | grep -i 'select @@hostmame'| wc -l
3
會獲得執行語句當時的遞增次數,最少鏈接算法會把鏈接請求調度到當時活動鏈接最少的那臺服務器。