據說你的Nginx還不會記錄Response Body?

時間 2020-05-09

標籤據說 nginx 還不記錄 response body 欄目 Nginx 简体版

原文原文鏈接

相信你們都遇到過在排查線上問題或Debug的時候，在某一瞬間，特別想開啓Nginx的Response Body日誌，來幫助本身快速的定位問題；但找半天發現只有request_body/upstream_addr/upstream_response_time這些相近變量可用；這個時候不要慌... 其實Nginx默認不開啓Response Body的日誌記錄是有緣由的，由於在Response Body過大，好比返回一個靜態頁面等內容時，不只形成大量的資源佔用，更會形成整個日誌文件的可讀性降低；但假如咱們明確Response Body的格式，或想捕獲Body裏的關鍵字內容的時候，能夠經過結合lua模塊來實現這一需求。html

Nginx編譯安裝

將Nginx和Lua結合可使用OpenResty，也可使用淘寶開源的Tengine，Tengine徹底兼容Nginx；下面咱們以Nginx編譯lua模塊爲例:nginx

個人系統環境：Ubuntu 14.04git

# 先安裝pcre庫，Redhat系的機器對應的包可能爲pcre-devel；也能夠本身手動編譯
aptitude -y install libpcre3-dev
# 下載並編譯安裝Lua解釋器-LuaJIT
wget -c https://github.com/openresty/luajit2/archive/v2.1-20200102.tar.gz
tar -zxvf v2.1-20200102.tar.gz
cd ./luajit2-2.1-20200102
make && make install
 # 告訴操做系統luajit的路徑
export LUAJIT_LIB=/usr/local/lib
export LUAJIT_INC=/usr/local/include/luajit-2.1
# 寫進配置文件，爲luajit配置動態連接庫
echo '/usr/local/lib/' >> /etc/ld.so.conf.d/luajit_so.conf
ldconfig
 # 下載並解壓Nginx編譯所需的模塊
# OpenResty Lua 模塊
wget -c wget -c https://github.com/openresty/lua-nginx-module/archive/v0.10.14.tar.gz
tar -zxvf v0.10.14.tar.gz
 # ngx_devel_kit 模塊
wget -c https://github.com/vision5/ngx_devel_kit/archive/v0.3.1.tar.gz
tar -zxvf v0.3.1.tar.gz
 # 下載Nginx源碼並編譯安裝
wget -c https://nginx.org/download/nginx-1.17.8.tar.gz
tar -zxvf nginx-1.17.8.tar.gz
cd nginx-1.17.8
# 簡單編譯示例
./configure --user=www --group=www --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module --add-module=../ngx_devel_kit-0.3.1 --add-module=../lua-nginx-module-0.10.16rc5

make && make install
複製代碼

Nginx版本和模塊信息查看

可使用-V選項查看編譯好的Nginx版本和模塊信息github

/usr/local/nginx/sbin/nginx -V

nginx version: nginx/1.17.8
built by gcc 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3)
built with OpenSSL 1.0.1f 6 Jan 2014
TLS SNI support enabled
configure arguments: --user=www --group=www --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module --add-module=../ngx_devel_kit-0.3.1 --add-module=../lua-nginx-module-0.10.14
複製代碼

Nginx配置與lua的配置

下面是一個Nginx的簡單配置示例，爲使Nginx Conf的結構更清晰，咱們使用include關鍵字，將log/lua/server幾部分的配置拆分紅單個獨立的配置文件；下面看下每部分配置的示例：shell

Context HTTP示例

cd /usr/local/nginx/conf
cat nginx.conf

user  www www;
worker_processes  auto;

error_log  /data/nginx/logs/error.log warn;
pid        /data/nginx/nginx.pid;
events {
    worker_connections  4096;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    gzip  on;
    charset utf-8;
    # 日誌格式配置文件
    include common/log_formats.conf;
    # 虛機主機配置
    include vhost/iyue_test.conf;

}

複製代碼

Context Server示例

這裏咱們用nginx代理GitHub的API接口來做爲測試；自定義變量 resp_body(默認爲空)用來存放Response Body的值，並經過lua功能模塊來爲其賦值。ubuntu

cat vhost/iyue_test.conf

server {

    listen       80;
    server_name test.iyue.pub;

    proxy_ignore_client_abort on;
    fastcgi_ignore_client_abort on;

    root /data/htdocs/iyue;
    # 指定日誌輸出文件和日誌格式爲iyue_log
    access_log /data/nginx/logs/iyue_access.log iyue_log;
    # 自定義一個resp_body的變量，用來存放Response Body信息
    set $resp_body "";
    # 也可使用body_filter_by_lua指令，將lua代碼直接展現在這裏；推薦採用文件的配置形式，方便後期維護
    body_filter_by_lua_file conf/lua/get_resp_code.lua;

    # 經過nginx配置代理github api，用於測試結果展現
    location /github/api/ {
        proxy_pass https://api.github.com/;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        #proxy_set_header Host $host;
        proxy_set_header Host api.github.com;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}
複製代碼

lua代碼示例

獲取Respondy Body的值，並賦值給變量$resp_body:api

cat lua/get_resp.lua

local chunk = string.sub(ngx.arg[1], 1, 1000)
ngx.ctx.buffered = (ngx.ctx.buffered or "") .. chunk
if ngx.arg[2] then
    local resp = ngx.ctx.buffered
    ngx.var.resp_body = resp
end
複製代碼

這裏ngx.arg[2]是一個bool類型；經過lua獲取到整個Response Body的信息後，賦值給變量$resp_body；後面能夠在log中打印該變量的值；bash

log_format示例

定義日誌格式，並在log中打印自定義變量$resp_body；這裏將其放到最後方便咱們觀察：app

cat common/log_formats.conf

log_format  iyue_log  '$remote_addr $http_x_forwarded_for [$time_local] $request_method $uri $args $request_body $server_protocol $status $body_bytes_sent $http_referer $http_user_agent $request_length rspt:$upstream_response_time rqtt:$request_time up:$upstream_addr $remote_port $host $resp_body';

複製代碼

結果展現

發起測試請求curl

curl http://test.iyue.pub/github/api/
複製代碼

從輸出日誌中，咱們能夠看到nginx已正常記錄了Response Body的信息：

192.168.10.11 - [04/May/2020:17:35:55 +0800] GET /github/api/ - - HTTP/1.1 200 2178 - curl/7.54.0 88 rspt:0.884 rqtt:0.880 up:140.82.113.5:443 64502 test.iyue.pub {\x22current_user_url\x22:\x22https://api.github.com/user\x22,\x22current_user_authorizations_html_url\x22:\x22https://github.com/settings/connections/applications{/client_id}\x22,\x22authorizations_url\x22:\x22https://api.github.com/authorizations\x22,\x22code_search_url\x22:\x22https://api.github.com/search/code?q={query}{&page,per_page,sort,order}\x22,\x22commit_search_url\x22:\x22https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}\x22,\x22emails_url\x22:\x22https://api.github.com/user/emails\x22,\x22emojis_url\x22:\x22https://api.github.com/emojis\x22,\x22events_url\x22:\x22https://api.github.com/events\x22,\x22feeds_url\x22:\x22https://api.github.com/feeds\x22,\x22followers_url\x22:\x22https://api.github.com/user/followers\x22,\x22following_url\x22:\x22https://api.github.com/user/following{/target}\x22,\x22gists_url\x22:\x22https://api.github.com/gists{/gist_id}\x22,\x22hub_url\x22:\x22https://api.github.com/hub\x22,\x22issue_search_url\x22:\x22https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}\x22,\x22issues_url\x22:\x22https://api.github.com/issues\x22,\x22keys_url\x22:\x22https://api.github.com/user/keys\x22,\x22label_searc
複製代碼

Response Body過濾

經過上面的日誌輸出結果，咱們看到日誌中的帶有'x22'的字樣，這實際上是雙引號的16進制ASCII碼；因此你們也能看出，這種打印出全部Response Body內容的方式，對特殊字符或中文輸出不是很友好；這裏建議的方案是，只輸出你須要的Response Body中的某一片斷，且輸出結果儘可能爲英文；咱們的lua代碼能夠進一步加下過濾輸出，好比咱們只輸出匹配Response Body中'current_user_url'的結果，lua代碼可變動爲：

cat lua/get_resp.lua

local chunk = string.sub(ngx.arg[1], 1, 1000)
ngx.ctx.buffered = (ngx.ctx.buffered or "") .. chunk
if ngx.arg[2] then
    local resp = ngx.ctx.buffered
    local user_url = ""
    user_url = string.match(resp, "\x22current_user_url\x22: \x22(.[^,]+)\x22")
    ngx.var.resp_body = user_url
end

複製代碼

優化後的日誌輸出結果爲:

192.168.10.11 - [04/May/2020:18:05:40 +0800] GET /github/api/ - - HTTP/1.1 200 2308 - curl/7.54.0 88 rspt:0.900 rqtt:0.897 up:140.82.112.6:443 50340 test.iyue.pub https://api.github.com/user
複製代碼

能夠看到日誌格式更加整齊，且獲取到了"current_user_url"對應的結果並正確打印。