基於nginx+lua+redis高性能api應用實踐

時間 2019-11-13

標籤基於 nginx+lua+redis nginx lua redis 高性能 api 應用實踐欄目 Nginx 简体版

原文原文鏈接

基於nginx+lua+redis高性能api應用實踐

前言

比較傳統的服務端程序（PHP、FAST CGI等），大多都是經過每產生一個請求，都會有一個進程與之相對應，請求處理完畢後相關進程自動釋放。因爲進程建立、銷燬對資源佔用比較高，因此不少語言都經過常駐進程、線程等方式下降資源開銷。即便是資源佔用最小的線程，當併發數量超過1k的時候，操做系統的處理能力就開始出現明顯降低，由於有太多的CPU時間都消耗在系統上下文切換。php

lua-nginx-module模塊將lua嵌入到nginx，讓nginx高效執行lua腳本，高併發，非阻塞的處理各類請求。Lua內建協程，這樣就能夠很好的將異步回調轉換成順序調用的形式。ngx_lua在Lua中進行的IO操做都會委託給Nginx的事件模型，從而實現非阻塞調用。html

每一個NginxWorker進程持有一個Lua解釋器或者LuaJIT實例，被這個Worker處理的全部請求共享這個實例。每一個請求的Context會被Lua輕量級的協程分割，從而保證各個請求是獨立的。 ngx_lua採用「one-coroutine-per-request」的處理模型，對於每一個用戶請求，ngx_lua會喚醒一個協程用於執行用戶代碼處理請求，當請求處理完成這個協程會被銷燬。每一個協程都有一個獨立的全局環境（變量空間），繼承於全局共享的、只讀的「comman data」。因此，被用戶代碼注入全局空間的任何變量都不會影響其餘請求的處理，而且這些變量在請求處理完成後會被釋放，這樣就保證全部的用戶代碼都運行在一個「sandbox」（沙箱），這個沙箱與請求具備相同的生命週期。得益於Lua協程的支持，ngx_lua在處理10000個併發請求時只須要不多的內存。根據測試，ngx_lua處理每一個請求只須要2KB的內存，若是使用LuaJIT則會更少。因此ngx_lua很是適合用於實現可擴展的、高併發的服務。nginx

nginx+lua安裝

環境需求：git

須要lua或luajit支持

Lua和Luajit的區別github

Lua是一個可擴展的輕量級腳本語言，它是用C語言編寫的。Lua的設計目是爲了嵌入應用程序中，從而爲應用程序提供靈活的擴展和定製功能。Lua代碼簡潔優美，幾乎在全部操做系統和平臺上均可以編譯、運行。redis

一個完整的Lua解釋器不過200kjson

LuaJIT是採用C語言寫的Lua的解釋器。LuaJIT被設計成全兼容標準Lua 5.1, 所以LuaJIT代碼的語法和標準Lua的語法沒多大區別。LuaJIT和Lua的一個區別是，LuaJIT的運行速度比標準Lua快數十倍，能夠說是一個lua的高效率版本。vim

官網
www.lua.org
http://luajit.org/download.html

安裝luajit

wget -c http://luajit.org/download/LuaJIT-2.0.4.tar.gz
tar zxf LuaJIT-2.0.4.tar.gz
cd LuaJIT-2.0.4
make && make install

or指定安裝位置
make install PREFIX=/usr/local/luajit2.0.4

下載ngx_devel_kit (NDK) module 模塊,不須要安裝

https://github.com/simpl/ngx_devel_kit/tags

下載nginx的lua模塊,不須要安裝

HttpLuaModule ：http://wiki.nginx.org/HttpLuaModule
https://github.com/openresty/lua-nginx-module#installation
https://github.com/openresty/lua-nginx-module/tags
wget -c https://github.com/openresty/lua-nginx-module/archive/v0.10.7.tar.gz

編譯nginx(傳統編譯)

導入環境變量，告訴nginx編譯系統,在哪查找luajit或lua
若是luajit使用默認安裝，會在如下路徑找到後端

# export LUAJIT_LIB=/usr/local/lib
# export LUAJIT_INC=/usr/local/include/luajit-2.0

# tell nginx's build system where to find LuaJIT 2.0:
 export LUAJIT_LIB=/path/to/luajit/lib
 export LUAJIT_INC=/path/to/luajit/include/luajit-2.0

 # tell nginx's build system where to find LuaJIT 2.1:
 export LUAJIT_LIB=/path/to/luajit/lib
 export LUAJIT_INC=/path/to/luajit/include/luajit-2.1

 # or tell where to find Lua if using Lua instead:
 #export LUA_LIB=/path/to/lua/lib
 #export LUA_INC=/path/to/lua/include

 # Here we assume Nginx is to be installed under /opt/nginx/.
 ./configure --prefix=/opt/nginx \
         --with-ld-opt="-Wl,-rpath,/path/to/luajit-or-lua/lib" \
         --add-module=/path/to/ngx_devel_kit \
         --add-module=/path/to/lua-nginx-module

編譯參數實例
 ./configure --prefix=/usr/local/nginx-1.10.2 --with-http_stub_status_module --with-http_sub_module --with-http_ssl_module --with-pcre=../pcre-8.39 --with-http_realip_module --with-http_gzip_static_module --with-zlib=../zlib-1.2.8 --with-openssl=../openssl-1.0.2h --with-ld-opt="-Wl,-rpath,/usr/local/luajit/lib"  --add-module=/home/wwwroot/ngx_devel_kit-0.3.0/ --add-module=/home/wwwroot/lua-nginx-module-0.10.7/


 ./configure --prefix=/usr/local/nginx-1.7.3-lua --with-http_stub_status_module --with-http_sub_module --with-http_ssl_module --with-pcre=../pcre-8.39 --with-http_realip_module --with-http_gzip_static_module --with-zlib=../zlib-1.2.8 --with-openssl=../openssl-1.0.2h --with-ld-opt="-Wl,-rpath,/usr/local/luajit2.0.4/lib" --add-module=../ngx_devel_kit-0.3.0 --add-module=../lua-nginx-module-0.10.7 

 make -j2
 make install

編譯nginx動態模塊(和以上方式二選一)

nginx從1.9.11版本開始，開始支持編譯動態模塊，經過在./configure命令使用--add-dynamic-module=PATH選項替代--add-module=PATH選項。同時在nginx配置文件頂層經過load_module來加載模塊，例如：api

./configure --add-dynamic-module=PATH 編譯動態模塊
make modules

編譯nginx動態庫，須要先安裝pcre庫，不然會報錯

pcre相關網址
http://www.pcre.org/
ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/
https://sourceforge.net/projects/pcre/files/pcre/

wget -c ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.40.tar.gz
使用如下源下載，速度更快
wget -c https://sourceforge.net/projects/pcre/files/pcre/8.39/pcre-8.39.tar.gz/download
tar zxf pcre-8.39.tar.gz
cd pcre-8.39

./configure
make
make install

同時在編譯nginx的時候，加上--with-ld-opt="-lpcre -Wl,-rpath,/usr/local/lib" 參數

編譯nginx動態模塊實例

注意！編譯動態模塊時，使用編譯參數須要和當前環境的nginx編譯參數相同、nginx版本一致，不然加載動態模塊時，有可能會報不兼容錯誤。使用nginx -V查看當前編譯參數。

導入luajit環境變量
export LUAJIT_LIB=/usr/local/luajit2.0.4/lib
export LUAJIT_INC=/usr/local/luajit2.0.4/include/luajit-2.0/


./configure --prefix=/usr/local/nginx-1.10.2 --with-http_stub_status_module --with-http_sub_module --with-http_ssl_module --with-pcre=../pcre-8.34 --with-http_realip_module --with-http_gzip_static_module  --with-ld-opt="-lpcre -Wl,-rpath,/usr/local/luajit2.0.4/lib" --add-dynamic-module=../ngx_devel_kit-0.3.0 --add-dynamic-module=../lua-nginx-module-0.10.7

./configure --prefix=/usr/local/nginx-1.10.1 --with-http_stub_status_module --with-http_sub_module --with-http_ssl_module --with-pcre=../pcre-8.39 --with-http_realip_module --with-http_gzip_static_module --with-zlib=../zlib-1.2.8 --with-openssl=../openssl-1.0.2h --with-ld-opt="-lpcre -Wl,-rpath,/usr/local/luajit2.0.4/lib" --add-dynamic-module=../ngx_devel_kit-0.3.0 --add-dynamic-module=../lua-nginx-module-0.10.7


./configure --with-pcre=../pcre-8.39 \
--with-openssl=../openssl-1.0.2h \
--with-zlib=../zlib-1.2.8 --with-http_ssl_module \
--with-ld-opt="-Wl,-rpath,/usr/local/luajit2.0.4/lib" \
--add-dynamic-module=../ngx_devel_kit-0.3.0 \
--add-dynamic-module=../lua-nginx-module-0.10.7

make modules

查看剛編譯的模塊
cd objs 

拷備so文件到nginx目錄
mkdir -p /usr/local/nginx/modules
cp ndk_http_module.so ngx_http_lua_module.so /usr/local/nginx/modules/

而後在nginx.conf配置文件中（配置環境main），經過load_module來加載動態模塊

load_module modules/ndk_http_module.so;
load_module modules/ngx_http_lua_module.so;

錯誤處理

啓動NGINX報以下錯誤
[root@695c1860c6f7 nginx-1.10.2]# nginx -t

nginx: error while loading shared libraries: libluajit-5.1.so.2: cannot open shared object file: No such file or directory

解決方法：（根據luajit安裝路徑）
默認安裝

# ln -s /usr/local/lib/libluajit-5.1.so.2 /lib64/libluajit-5.1.so.2
luajit已指定安裝路徑
ln -s /usr/local/luajit2.0.4/lib/libluajit-5.1.so.2 /lib64/libluajit-5.1.so.2

[root@695c1860c6f7 nginx-1.10.2]# nginx -t

nginx: the configuration file /usr/local/nginx-1.10.2/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx-1.10.2/conf/nginx.conf test is successful

nginx加載動態模塊，報錯

nginx: [emerg] dlopen() "/usr/local/nginx-1.10.1/modules/ngx_http_lua_module.so" failed (/usr/local/nginx-1.10.1/modules/ngx_http_lua_module.so: undefined symbol: pcre_dfa_exec) in /usr/local/nginx-1.10.1/conf/nginx.conf:13

解決方法
ngx_http_lua_module，使用了pcre庫，須要安裝pcre庫

pcre相關網址
http://www.pcre.org/
ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/
https://sourceforge.net/projects/pcre/files/pcre/


wget -c ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.40.tar.gz

使用如下源下載，速度更快
wget -c https://sourceforge.net/projects/pcre/files/pcre/8.39/pcre-8.39.tar.gz/download
tar zxf pcre-8.39.tar.gz
cd pcre-8.39

./configure
make
make install

同時在編譯nginx的時候，加上--with-ld-opt="-lpcre -Wl,-rpath,/usr/local/lib" 參數

至此nginx+lua環境安裝成功

在nginx配置文件，server中，加入以下配置，進行測試,curl http://localhost/lua

location = /lua {
    default_type 'text/plain';
    content_by_lua_block {
        ngx.say('hello lua')
    }
}

安裝lua的擴展包，以支持redis,cson解析

1.下載nginx lua redis包

git clone https://github.com/openresty/lua-resty-redis.git
tar解壓到某個目錄便可，稍後在lua程序中調用

2.下載lua cjson包，用於json解析

https://openresty.org/cn/lua-cjson-library.html
git clone https://github.com/openresty/lua-cjson/
wget -c https://www.kyne.com.au/~mark/software/download/lua-cjson-2.1.0.tar.gz

3.安裝lua cjson包

tar zxf lua-cjson-2.1.0.tar.gz
cd lua-cjson-2.1.0

vim Makefile 

能夠用lua5.1或luajit進行編譯，安裝的是luajit，這裏在PREFIX指定luajit的安裝路徑，LUA_INCLUDE_DIR爲包含lua.h的路徑

##### Build defaults #####
LUA_VERSION =       5.1
TARGET =            cjson.so
PREFIX =            /usr/local/luajit2.0.4
#CFLAGS =            -g -Wall -pedantic -fno-inline
CFLAGS =            -O3 -Wall -pedantic -DNDEBUG
CJSON_CFLAGS =      -fpic
CJSON_LDFLAGS =     -shared
LUA_INCLUDE_DIR =   $(PREFIX)/include/luajit-2.0
LUA_CMODULE_DIR =   $(PREFIX)/lib/lua/$(LUA_VERSION)
LUA_MODULE_DIR =    $(PREFIX)/share/lua/$(LUA_VERSION)
LUA_BIN_DIR =       $(PREFIX)/bin

最後make install
或是make,而後手動拷備
cp cjson.so /usr/local/luajit2.0.4/lib/lua/5.1/

配置nginx.conf,支持lua解析

vim nginx.conf，加入以下配置

http{
    #指定剛下載的redis擴展程序存放目錄
    lua_package_path "/home/wwwroot/luacode/vendor/?.lua;;";
    #指定so模式的lua擴展包,基於c編譯的,如cjson包
    lua_package_cpath '/usr/local/luajit2.0.4/lib/lua/5.1/?.so;;';
    #lua nginx worker共享緩存
    lua_shared_dict data 100m;
    init_by_lua_file /home/wwwroot/luacode/init.lua;    
}

upstream backend{
    server 10.101.35.51:8800;
}

server{
    location /api {
        default_type 'text/plain';
        #access_by_lua_file /home/wwwroot/luacode/auth.lua;
        #GET方式的請求，經過lua解析
        if ($request_method = "GET") {
            content_by_lua_file /home/wwwroot/luacode/content.lua;
        }
        if ($request_method != "GET") {
            proxy_pass http://backend;
        }
    }

    location ~ /backend/(.*) {
        internal;
        rewrite /backend/(.*) /index.php?$1 last;
        #rewrite /backend/(.*) $1 break;
        #proxy_pass http://backend;
    }
}

部署lua代碼

vim init.lua

config = {}
config["redis"] = {
    host = "10.99.206.208",
    port = "8379",
    db   = 6,
    timeout = "1000",
    keepalive = {idle = 10000, size = 100},
}
config['nginx'] = {
    ngx_shared_timeout = 120
}

vim content.lua

-- author ljh
-- version 1.0
local redis = require("resty.redis")
local cjson = require("cjson")
local ngx_log = ngx.log
local ngx_ERR = ngx.ERR
local ngx_exit = ngx.exit
local ngx_print = ngx.print
-- local ngx_re_match = ngx.re.match
local ngx_var = ngx.var
local ngx_shared_data = ngx.shared.data
local red = redis:new()

-- 響應輸出內容
-- body   http輸出body內容
-- status http狀態碼
-- header http響應頭,table格式
local function response(body,status,header)
    ngx.status = status
    if header then
        for key, val in pairs(header) do
            ngx.header[key] = val
        end
    end
    ngx_print(body)
    ngx_exit(ngx.status)
end

-- 經過http回後端請求數據
local function read_http(id)
    ngx_log(ngx_ERR, "request http uri :", id)
    local resp = ngx.location.capture("/backend/"..id)
    if not resp then
        ngx_log(ngx_ERR, "request error :", err)
        return 
    end
    response(resp.body,resp.status,resp.header)
    -- return resp
end

--關閉redis鏈接
local function close_redis(red)
    if not red then
        return
    end
    local pool_max_idle_time = config.redis.keepalive.idle
    local pool_size = config.redis.keepalive.size
    -- Basically if your NGINX handle n concurrent requests and your NGINX has m workers, then the connection pool size should be configured as n/m
    -- redis鏈接放入鏈接池
    local ok, err = red:set_keepalive(pool_max_idle_time, pool_size)
    if not ok then
        ngx_log(ngx_ERR, "set redis keepalive error : ", err)
    end
end

-- 驗證access token是否有效
local function validToken(data)
    if not data then
        return false
    end
    if data == ngx.null then
        return false
    end
    local json = cjson.decode(data)
    if 'table' ~= type(json) then
        return false
    end
    local expire_time = json.expire_time
    local current_time = os.time()
    if ((expire_time > 0) and (current_time > expire_time)) then
        return false
    end
    return true
end

-- get access token from http request (header or query params)
local function getAccessToken()
    --str = ngx.req.get_headers()["Authorization"]
    --for i in string.gmatch(str, "%S+") do
    --  ngx.say(i)
    --end
    local access_token = nil
    -- get access_token from header
    local auth_code = ngx.req.get_headers()["Authorization"]
    if auth_code then
        -- the header is Authorization:bearer xxxx
        access_token = string.sub(auth_code,8)
    else
        -- get access token from GET MEquery params
        access_token = ngx.var.arg_access_token
    end
    return access_token
end

-- 驗證http請求,若是經過返回token,不然返回false
local function auth()
    local access_token = getAccessToken()
    if not access_token then
        return false
    end
    key = "access-token-key-"..ngx.md5(access_token)
    local token = red:get(key)
    if not validToken(token) then
        return false
    end
    return cjson.decode(token)
end

-- main function
local function main()
    local status = 200
    local header = {}
    local content = nil
    local resp = nil
    local client_id = nil
    header['content_type'] = 'application/json'
    -- 鏈接redis,失敗轉後端處理
    red:set_timeout(config.redis.timeout)
    local ok, err = red:connect(config.redis.host, config.redis.port)
    if not ok then
        ngx_log(ngx_ERR, "connect to redis error : ", err)
        read_http(ngx_var.request_uri)
    end
    -- select redis db,失敗轉後端處理
    local ok, err = red:select(config.redis.db)
    if not ok then
        ngx_log("failed to select redis db: ", err)
        read_http(ngx_var.request_uri)
    end
    -- 驗證token,失敗回後端(這裏是經過redis驗證,考慮redis失效等狀況)
    local token = auth()
    if not token or not token.client_id then
        read_http(ngx_var.request_uri)
    end
    -- 獲取client_id,結合request_uri組成redis緩存key
    client_id = token.client_id
    -- cache_key,request_uri md5 key
    local cache_key = 'api_clientid_'..client_id..'_request_uri_'..ngx.md5(ngx_var.request_uri)
    -- 從nginx的共享內存中取數據(減小redis的tcp鏈接)
    local content = ngx_shared_data:get(cache_key)
    -- nginx共享內存有數據,直接返回
    if content  then
        response(content,status,header)
    end
    -- nginx共享內存沒有數據，則請求redis緩存
    if not content or content == ngx.null then 
        ngx_log(ngx_ERR, "nginx shared memory not found content, back to reids, id : ", cache_key)
        content = red:get(cache_key)
    end 
    -- redis 沒有數據,將請求轉發到後端
    if not content or content == ngx.null then 
        -- ngx.say('no redis data')
        ngx_log(ngx_ERR, "redis not found content, back to http, request_uri : ", ngx_var.request_uri)
        read_http(ngx_var.request_uri)
    else
        close_redis(red)
        -- 加入nginx共享緩存，worker共享
        ngx_shared_data:set(cache_key,content,config.nginx.ngx_shared_timeout)
        response(content,status,header)
    end
end
main()