keepalived+nginx實現nginx的高可用

keepalived+nginx實現nginx的高可用

=================================

nginx的高可用

nginx實現後端realserver的負載均衡

==================================

實驗環境：

OS：Centos 6.4(redhat 6.4)

yum源:

1 2 3 4 5 6 7 8 9 10 11

[centos] name=sohu-centos baseurl=http: //mirrors .sohu.com /centos/ $releasever /os/ $basearch gpgcheck=1 enable =0 gpgkey=http: //mirrors .sohu.com /centos/RPM-GPG-KEY-CentOS-6 [epel] name=sohu-epel baseurl=http: //mirrors .sohu.com /fedora-epel/ $releasever/$basearch/ enable =1 gpgcheck=0

拓撲圖：

拓撲圖的規劃：

	IP地址	軟件
Master	172.16.22.1(VIP：172.16.22.100)	keepalived+nginx
Backup	172.16.22.2(VIP：172.16.22.100)	keepalived+nginx
apache1	172.16.22.3	httpd
apache2	172.16.22.4	httpd

此架構需考慮的問題

1）、Master沒掛，則Master佔有vip且nginx運行在Master上

2）、Master掛了，則backup搶佔vip且在backup上運行nginx服務

3）、若是master服務器上的nginx服務掛了，則vip資源轉移到backup服務器上

4）、檢測後端服務器的健康狀態

Master和Backup兩邊都開啓nginx服務，不管Master仍是Backup，當其中的一個keepalived服務中止後，vip都會漂移到keepalived服務還在的節點上，

若是要想使nginx服務掛了，vip也漂移到另外一個節點，則必須用腳本或者在配置文件裏面用shell命令來控制。

首先必須明確後端服務器的健康狀態檢測keepalived在這種架構上是沒法檢測的，後端服務器的健康狀態檢測是有nginx來判斷的，可是nginx的檢測機制有必定的缺陷，後端服務器某一個宕機以後，nginx仍是會分發請求給它，在必定的時間內後端服務響應不了，nginx則會發給另一個服務器，而後當客戶的請求來了，nginx會一段時間內不會把請求分發給已經宕機的服務器，可是過一段時間後，nginx仍是會把分發請求發給宕機的服務器上。

1、安裝keepalived+nginx

Master：

一、安裝keepalived和編譯安裝nginx

[root@jie1 ~]# yum -y install keepalived
[root@jie1 ~]#tar xf nginx-1.4.2.tar.gz
[root@jie1 ~]#yum -y groupinstall "Development tools" "Server  Platform Development"
[root@jie1 ~]#yum -y install pcre-devel
[root@jie1 ~]# cd nginx-1.4.2
[root@jie1 nginx-1.4.2]# groupadd nginx
[root@jie1 nginx-1.4.2]# useradd -r -g nginx nginx
[root@jie1 nginx-1.4.2]#./configure \
--prefix=/usr\
--sbin-path=/usr/sbin/nginx\
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/var/run/nginx/nginx.pid  \
--lock-path=/var/lock/nginx.lock \
--user=nginx \
--group=nginx \
--with-http_ssl_module \
--with-http_flv_module \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--http-client-body-temp-path=/var/tmp/nginx/client/\
--http-proxy-temp-path=/var/tmp/nginx/proxy/\
--http-fastcgi-temp-path=/var/tmp/nginx/fcgi/\
--http-uwsgi-temp-path=/var/tmp/nginx/uwsgi\
--http-scgi-temp-path=/var/tmp/nginx/scgi\
--with-pcre
[root@jie1 nginx-1.4.2]# make && make install

二、提供nginx的system V服務腳本文件

[root@jie1 nginx-1.4.2]# vim /etc/rc.d/init.d/nginx
#!/bin/sh
#
# nginx - this script starts and stops the nginx daemon
#
# chkconfig:   - 85 15
# description:  Nginx is an HTTP(S) server, HTTP(S) reverse \
#               proxy and IMAP/POP3 proxy server
# processname: nginx
# config:      /etc/nginx/nginx.conf
# config:      /etc/sysconfig/nginx
# pidfile:     /var/run/nginx.pid
# Source function library.
. /etc/rc.d/init.d/functions
# Source networking configuration.
. /etc/sysconfig/network
# Check that networking is up.
[ "$NETWORKING"= "no"] && exit0
nginx="/usr/sbin/nginx"
prog=$(basename$nginx)
NGINX_CONF_FILE="/etc/nginx/nginx.conf"
[ -f /etc/sysconfig/nginx] && . /etc/sysconfig/nginx
lockfile=/var/lock/subsys/nginx
make_dirs() {
# make required directories
user=`nginx -V 2>&1 | grep"configure arguments:"| sed's/[^*]*--user=\([^ ]*\).*/\1/g'-`
options=`$nginx -V 2>&1 | grep'configure arguments:'`
foropt in$options; do
if[ `echo$opt | grep'.*-temp-path'` ]; then
value=`echo$opt | cut-d "="-f 2`
if[ ! -d "$value"]; then
# echo "creating" $value
mkdir-p $value && chown-R $user $value
fi
fi
done
}
start() {
[ -x $nginx ] || exit5
[ -f $NGINX_CONF_FILE ] || exit6
make_dirs
echo-n $"Starting $prog: "
daemon $nginx -c $NGINX_CONF_FILE
retval=$?
echo
[ $retval -eq0 ] && touch$lockfile
return$retval
}
stop() {
echo-n $"Stopping $prog: "
killproc $prog -QUIT
retval=$?
echo
[ $retval -eq0 ] && rm-f $lockfile
return$retval
}
restart() {
configtest || return$?
stop
sleep1
start
}
reload() {
configtest || return$?
echo-n $"Reloading $prog: "
killproc $nginx -HUP
RETVAL=$?
echo
}
force_reload() {
restart
}
configtest() {
$nginx -t -c $NGINX_CONF_FILE
}
rh_status() {
status $prog
}
rh_status_q() {
rh_status >/dev/null2>&1
}
case"$1"in
start)
rh_status_q && exit0
$1
;;
stop)
rh_status_q || exit0
$1
;;
restart|configtest)
$1
;;
reload)
rh_status_q || exit7
$1
;;
force-reload)
force_reload
;;
status)
rh_status
;;
condrestart|try-restart)
rh_status_q || exit0
;;
*)
echo$"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
exit2
esac
[root@jie1 nginx-1.4.2]# chmod +x /etc/rc.d/init.d/nginx
[root@jie1 nginx-1.4.2]# service nginx start
Starting nginx:                                            [  OK  ]
[root@jie1 nginx-1.4.2]# scp -p /etc/rc.d/init.d/nginx  172.16.22.2:/etc/rc.d/init.d    #把nginx的服務腳本複製到backup上，-p是保持原有的權限

三、修改配置文件

[root@jie1 ~]# cd /etc/keepalived/
[root@jie1 keepalived]# vim keepalived.conf
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from admin@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LTT
}
vrrp_script chk_nginx {  #檢測nginx服務是否在運行有不少方式，好比進程，用腳本檢測等等
   script "killall -0 nginx"  #用shell命令檢查nginx服務是否存在
   interval 1  #時間間隔爲1秒檢測一次
   weight -2   #當nginx的服務不存在了，就把當前的權重-2
   fall 2      #測試失敗的次數
   rise 1      #測試成功的次數
}
vrrp_instance IN_1 {
    state MASTER
    interface eth0
    virtual_router_id 22
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass aaaa
    }
    virtual_ipaddress {
        172.16.22.100
    }
   track_script {
    chk_nginx   #引用上面的vrrp_script定義的腳本名稱
}
}
[root@jie1 keepalived]#scp keepalived.conf 172.16.22.2:/etc/keepalived #把配置文件copy到Backup服務器上，copy以前要保證Backup服務器上面已經安裝了keepalived

四、開啓keepalived和nginx的服務

[root@jie1 keepalived]# service keepalived start
Starting keepalived:                                       [  OK  ]
[root@jie1 keepalived]# chkconfig --add keepalived
[root@jie1 keepalived]# chkconfig keepalived on
[root@jie1 ~]# service nginx start
Starting nginx:                                            [  OK  ]
[root@jie1 ~]# chkconfig --add nginx
[root@jie1 ~]# chkconfig  nginx on

Backup：

一、安裝keepalived和編譯安裝nginx

[root@jie2 ~]# yum -y install keepalived
[root@jie2 ~]#tar xf nginx-1.4.2.tar.gz
[root@jie2 ~]#yum -y groupinstall "Development tools" "Server  Platform Development"
[root@jie2 ~]#yum -y install pcre-devel
[root@jie2 ~]# cd nginx-1.4.2
[root@jie2 nginx-1.4.2]# groupadd nginx
[root@jie2 nginx-1.4.2]# useradd -r -g nginx nginx
[root@jie2 nginx-1.4.2]#./configure \
  --prefix=/usr \
  --sbin-path=/usr/sbin/nginx \
  --conf-path=/etc/nginx/nginx.conf \
  --error-log-path=/var/log/nginx/error.log \
  --http-log-path=/var/log/nginx/access.log \
  --pid-path=/var/run/nginx/nginx.pid  \
  --lock-path=/var/lock/nginx.lock \
  --user=nginx \
  --group=nginx \
  --with-http_ssl_module \
  --with-http_flv_module \
  --with-http_stub_status_module \
  --with-http_gzip_static_module \
  --http-client-body-temp-path=/var/tmp/nginx/client/ \
  --http-proxy-temp-path=/var/tmp/nginx/proxy/ \
  --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ \
  --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi \
  --http-scgi-temp-path=/var/tmp/nginx/scgi \
  --with-pcre
[root@jie2 nginx-1.4.2]# make && make install

二、以前 已經從Master複製了nginx的system V服務腳本文件，啓動nginx服務

[root@jie2 ~]# service nginx start
Starting nginx:                                            [  OK  ]
[root@jie2 ~]# chkconfig --add nginx
[root@jie2 ~]# chkconfig  nginx on

三、修改配置文件

[root@jie2 ~]# cd /etc/keepalived/
[root@jie2 keepalived]# vim keepalived.conf   #此配置文件是從Master服務器上copy過來，只需小小改動
state BACKUP  #把這裏原先的MASTER改爲BACKUP
priority 99   #把這裏原先的100改爲99

四、開啓服務

[root@jie2 keepalived]# service keepalived start
Starting keepalived:                                       [  OK  ]
[root@jie2 keepalived]# chkconfig --add keepalived
[root@jie2 keepalived]# chkconfig keepalived on

apache1：

一、安裝(博主這裏用rpm包安裝，各位朋友能夠用源碼編譯安裝)

[root@jie3 ~]# yum -y install httpd

二、創建測試網頁文件

[root@jie3 ~]# cd /var/www/html/
[root@jie3 html]# cat index.html #建一個測試網頁
<h1>this is apache1</h1>

三、開啓服務

[root@jie3 html]# service httpd start
Starting httpd:                 [  OK  ]
[root@jie3 html]# chkconfig --add httpd
[root@jie3 html]# chkconfig  httpd on

apache2：

一、安裝(博主這裏用rpm包安裝，各位朋友能夠用源碼編譯安裝)

[root@jie4 ~]# yum -y install httpd

二、創建測試網頁文件

[root@jie4 ~]# cd /var/www/html/
[root@jie4 html]# cat index.html #建一個測試網頁
<h1>this is apache2</h1>

三、開啓服務

[root@jie4 html]# service httpd start
Starting httpd:                 [  OK  ]
[root@jie4 html]# chkconfig --add httpd
[root@jie4 html]# chkconfig  httpd on

此致全部安裝已經完成。

2、nginx實現後端realserver的負載均衡，因爲兩邊的配置文件必須保持一致，因此在Master配置完後直接copy到Backup上

Master：

[root@jie1 ~]# cd /etc/nginx/
[root@jie1 nginx]# grep -v "#" nginx.conf | grep -v "^$"
worker_processes  1;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    upstream apacheweb {  #定義負載均衡的模塊
        server 172.16.22.3:80 max_fails=3 fail_timeout=2s;
        server 172.16.22.4:80 max_fails=3 fail_timeout=2s;
     }
    server {
        listen       80;
        server_name  localhost;
        location / {
            root   html;
            index  index.html index.htm;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
        location ~ \.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$ {
            root        /var/www/html;  #此處定義後端服務器網頁存放路徑
            proxy_pass   http://apacheweb;
        }
    }
}
[root@jie1 nginx]# scp nginx.conf 172.16.22.2:/etc/nginx

兩邊分別重啓服務

[root@jie2 nginx]# service nginx restart
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
Stopping nginx:                                            [  OK  ]
Starting nginx:                                            [  OK  ]
[root@jie2 nginx]# service nginx restart
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
Stopping nginx:                                            [  OK  ]
Starting nginx:                                            [  OK  ]

驗證負載均衡：

本博客只是簡單的用nginx作了方向代理和靜態頁面的負載均衡，keepalived+nginx實現高可用的nginx的動靜分離，讀寫分離，後續會持續更新