Nginx+Keepalived簡單構建高可用集羣

轉自：http://laoguang.blog.51cto.com/6013350/1099103

之前一直用heartbeat或corosync+pacemaker構建高可用集羣，如今發現keepalived實現起來更簡單。
keepalived的master向backup發送廣播，當backup一段時間收不到對方傳來的VRRP廣播時，backup會經過競選一個master,master就會從新持有資源。具體的理論知識參見http://bbs.ywlm.net/thread-790-1-1.html

實驗目標:2臺Nginx+Keepalived 2臺Lamp構建高可用Web集羣

規劃：

ng1.laoguang.me 192.168.1.22 ng1
ng2.laoguang.me 192.168.1.23 ng2
lamp1.laoguang.me   192.168.1.24 lamp1
lamp2.laoguang.me   192.168.1.25 lamp2

拓撲：

一.基本環境準備
ng1,ng2上安裝nginx
lamp1,lamp2 上構建LAMP或只安裝httpd，我只安裝了Httpd，這裏不給你們演示了，有須要請看個人其它博文，更改lamp1,lamp2的 index.html的內容分別爲lamp1和lamp2,以容易區分，實際集羣中內容應該是一致的，由共享存儲提供。

二.ng1,ng2上安裝配置keepalived
下載地址:http://www.keepalived.org/download.html
2.1 安裝keepalived

tar xvf keepalived-1.2.7.tar.gz
cd keepalived-1.2.7
./configure --prefix=/usr/local/keepalived
##可能會提示安裝popt-devel包，yum便可
make && make install

2.2 整理配置文件與腳本

mkdir /etc/keepalived
##keepalived默認配置文件從/etc/keepalived下讀取
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
##就一個二進制文件，直接拷貝過去便可，多的話就更改PATH吧
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
##腳本的額外配置文件讀取位置
cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
##啓動腳本你懂得
cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
##咱們關鍵的keepalived配置文件

2.3 修改ng1的/etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
  notification_email {
    ibuler@qq.com         ##出故障發送郵件給誰
  }
  notification_email_from keepalived@localhost ##故障用哪一個郵箱發送郵件
  smtp_server 127.0.0.1   ##SMTP_Server IP
  smtp_connect_timeout 30 ##超時時間
  router_id LVS_DEVEL     ##服務器標識
}
vrrp_instance VI_1 {
   state BACKUP
##狀態，都爲BACKUP，它們會推選Master，若是你寫MASTER，它就會是Master，
    ##當Master故障時Backup會成爲Master,當原來的Master恢復後，原來的Master會成爲Master
   interface eth0        ##發送VRRP的接口，仔細看你的是否是eth0
   virtual_router_id 51  ##虛擬路由標識，同一個組應該用一個，即Master與Backup同一個
   priority 100   ##重要的優先級哦
   nopreempt      ##不搶佔，一個故障時，重啓後恢復後不搶佔意資源
   advert_int 1   ##同步間隔時長
   authentication {              ##認證
       auth_type PASS            ##認證方式
       auth_pass www.laoguang.me ##密鑰
   }
   virtual_ipaddress {
       192.168.1.18/24 dev eth0              ##VIP
   }
}
##後面的刪除吧，LVS上纔有用

拷貝到ng2上一份，只修改priority 90 便可

scp /etc/keepalived/keepalived.conf 192.168.1.23:/etc/keepalived/
##Ng2上
vi /etc/keepalived/keepalived.conf  priority 90   ##其它一致

2.4 ng1,ng2上啓動keepalived

service keepalived start

查看日誌

tail /var/log/messages
Nov 27 08:07:54 localhost Keepalived_vrrp[41871]: VRRP_Instance(VI_1) Entering BACKUP STATE
Nov 27 08:07:54 localhost Keepalived_vrrp[41871]: VRRP sockpool: [ifindex(2), proto(112), fd(11,12)]
Nov 27 08:07:54 localhost Keepalived_healthcheckers[41870]: Using LinkWatch kernel netlink reflector...
Nov 27 08:07:54 localhost Keepalived_vrrp[41871]: VRRP_Instance(VI_1) forcing a new MASTER election
Nov 27 08:07:55 localhost Keepalived_vrrp[41871]: VRRP_Instance(VI_1) Transition to MASTER STATE
Nov 27 08:07:56 localhost Keepalived_vrrp[41871]: VRRP_Instance(VI_1) Entering MASTER STATE
Nov 27 08:07:56 localhost Keepalived_vrrp[41871]: VRRP_Instance(VI_1) setting protocol VIPs.
Nov 27 08:07:56 localhost Keepalived_healthcheckers[41870]: Netlink reflector reports IP 192.168.1.18 added
Nov 27 08:07:56 localhost Keepalived_vrrp[41871]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.18
Nov 27 08:08:01 localhost Keepalived_vrrp[41871]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.18

查看vip綁定到哪臺機器上了

ip addr     ##ng1上
....省略
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
   link/ether 00:0c:29:e8:90:0b brd ff:ff:ff:ff:ff:ff
   inet 192.168.1.22/24 brd 192.168.1.255 scope global eth0
   inet 192.168.1.18/32 scope global eth0
   inet6 fe80::20c:29ff:fee8:900b/64 scope link
      valid_lft forever preferred_lft forever

由此可知vip綁定到ng1上了
三，Keepalived測試

3.1 關閉ng1上的keepalived或者直接關閉ng1 查看vip轉移狀況

service keepalived stop
ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
   link/ether 00:0c:29:e8:90:0b brd ff:ff:ff:ff:ff:ff
   inet 192.168.1.22/24 brd 192.168.1.255 scope global eth0
   inet6 fe80::20c:29ff:fee8:900b/64 scope link
      valid_lft forever preferred_lft forever

3.2 查看ng2上是否綁定了vip

ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
   link/ether 00:0c:29:dd:00:77 brd ff:ff:ff:ff:ff:ff
   inet 192.168.1.23/24 brd 192.168.1.255 scope global eth0
   inet 192.168.1.18/32 scope global eth0
   inet6 fe80::20c:29ff:fedd:77/64 scope link
      valid_lft forever preferred_lft forever

由此可知ip轉移正常，keepalived設置成功

四.配置Nginx作反向代理

4.1 修改nginx配置文件

vi /etc/nginx/nginx.conf
user  nginx nginx;   ##運行nginx的用戶和組
worker_processes  2; ##啓動進程數
error_log /var/log/nginx/error.log  notice;  ##錯誤日誌記錄
pid        /tmp/nginx.pid;                   ##pid存放位置
worker_rlimit_nofile 65535;                  ##線程最大打開文件數，須配合ulimit -SHn使用
events {
   use epoll;                 ##工做模型
   worker_connections  65536; ##單進程最大鏈接數
}
http {                         ##http模塊
   include       mime.types;  ##包含進來
   default_type  application/octet-stream; ##默認類型
   log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                     '$status $body_bytes_sent "$http_referer" '
                     '"$http_user_agent" "$http_x_forwarded_for"';
##日誌格式
   access_log  /var/logs/nginx/http.access.log  main; ##訪問日誌
   client_max_body_size 20m;  ##最大請求文件大小
   client_header_buffer_size 16k; ##來自客戶端請求header_buffer大小
   large_client_header_buffers 4 16k; ##較大請求緩衝個數與大小
   sendfile       on;                 ##內核空間直接發送到tcp隊列
   tcp_nopush     on;
   tcp_nodelay    on;
   keepalive_timeout  65;     ##長鏈接時長
   gzip  on;                  ##啓用壓縮
   gzip_min_length 1k;        ##最小壓縮大小
   gzip_buffers 4 16k;        ##壓縮緩衝
   gzip_http_version 1.1;     ##支持協議
   gzip_comp_level 2;         ##壓縮等級
   gzip_types text/plain application/x-javascript text/css application/xml;      ##壓縮類型
   gzip_vary on;              ##前端緩存服務器能夠緩存壓縮過的頁面
   upstream laoguang.me {     ##用upstream模塊定義集羣與RS
       server 192.168.1.24:80 max_fails=3fail_timeout=10s;   ##RS的地址，最大錯誤數與超時時間，超過了自動剔除
       server 192.168.1.25:80 max_fails=3fail_timeout=10s;
}
server {
       listen       80;           ##監聽端口
       server_name  192.168.1.18; ##servername
       root   html;               ##根目錄
       index  index.html index.htm; ##你懂得
       #charset koi8-r;
       access_log  logs/192.168.1.18.access.log  main;
  ##這個server的訪問日誌
       location / {
               proxy_pass http://laoguang.me;  ##反向代理
               proxy_redirect off;
               proxy_set_header X-Real-IP $remote_addr;
 ##真實客戶ip告訴後端
               proxy_set_header X-Forwarded-For Proxy_add_x_forwarded_for;
       }
       location /nginx {
               access_log off;
               stub_status on;  ##狀態頁面
       }
       error_page   500 502 503 504  /50x.html;
location = /50x.html {
           root   html;
       }
   }
}

4.2 拷貝到ng2上一份

scp /etc/nginx/nginx.conf 192.168.1.23:/etc/nginx/

4.3 測試反向代理可否負載均衡

lamp1,lamp2啓動httpd

service httpd start

ng1重啓nginx

service nginx restart

用RealIp訪問測試可否輪詢
http://192.168.1.22

一樣測試ng2,若是都能實現負載均衡，那麼繼續

五.測試keepalived與nginx配合運行

如今192.168.1.18在 ng2上，        訪問 http://192.168.1.18 測試可否輪詢
ng2上 service keepalived stop     訪問測試 http://192.168.1.18 可否輪詢
關閉lamp1上的service httpd stop   訪問測試http://192.168.1.18 是否會報錯

到此高可用webserver構建完畢，沒有單點故障，任何一點故障不影響業務。