利用nginx內置ngx_http_mirror_module模塊實現流量複製及流量放大

0. 需求

複製線上真實流量，在不影響真實業務前提下，利用複製流量來作故障分析、性能定位、遷移評估等功能。具體功能包含：

• 支持或禁止post請求複製
• 記錄複製(鏡像)日誌

mirror： 中文爲鏡像的意思，這裏指流量複製的目的地。

1. ngx_http_mirror_module模塊特性

• nginx 1.13.4及後續版本內置ngx_http_mirror_module模塊，提供流量鏡像(複製)的功能。
• 支持流量放大，作法爲：配置多份相同鏡像。
• 相比tcp-copy的優點：無需錄製流量，實時可用；配置至關簡單。
• 源站請求，直接原路返回；正常配置下，mirror請求不影響源站請求及響應，源站nginx-server將流量複製到mirror站後，二者再也不有任何交集。

2. 安裝及配置

正常安裝nginx 1.13.4及後續版本。下面配置在1.14.1中驗證經過。

2.1 複製get及post請求

server {
        listen       80;
        server_name  web1.www.com;
        # 源站配置
        location / {
                access_log  /data/nginx/1.14.1/logs/web1/access.log  accesslog;
                mirror /mirror;
                mirror_request_body on;# Indicates whether the client request body is mirrored. default value is on.
                proxy_pass http://web1.upstream.name;
        }
        # 鏡像站點配置
        location /mirror {
                internal; # 內部配置
                proxy_pass http://mirror.web1.upstream.name$request_uri;
                proxy_pass_request_body on; # Indicates whether the original request body is passed to the proxied server. default value is on
                proxy_set_header X-Original-URI $request_uri; # reset uri
        }
}

2.2 不容許複製post請求

默認是支持post複製的，須要經過配置來禁止

server {
        listen       80;
        server_name  web1.www.com;

        # 源站配置
        location / {
                access_log  /data/nginx/1.14.1/logs/web1/access.log  accesslog;
                mirror /mirror;
                mirror_request_body off;# Indicates whether the client request body is mirrored. default value is on.
                proxy_pass http://web1.upstream.name;
        }

        # 鏡像站點配置
        location /mirror {
                # 判斷請求方法，不是GET返回403
                if ($request_method != GET) {
                    return 403;
                }
                internal; # 內部配置
                proxy_pass http://mirror.web1.upstream.name$request_uri;
                proxy_pass_request_body off; # Indicates whether the original request body is passed to the proxied server. default value is on
                proxy_set_header Content-Length ""; # mirror_request_body/proxy_pass_request_body都設置爲off，則Conten-length須要設置爲""，不然有坑
                proxy_set_header X-Original-URI $request_uri; # 使用真實的url重置url
        }
}

3. 流量放大配置方法

配置多分mirror

server {
        listen       80;
        server_name  web1.www.com;
        # 源站配置
        location / {
                access_log  /data/nginx/1.14.1/logs/web1/access.log  accesslog;
                mirror /mirror;
                # 多加一份mirror，流量放大一倍
                mirror /mirror;
                mirror_request_body on;# Indicates whether the client request body is mirrored. default value is on.
                proxy_pass http://web1.upstream.name;
        }
        # 鏡像站點配置
        location /mirror {
                internal; # 內部配置
                proxy_pass http://mirror.web1.upstream.name$request_uri;
                proxy_pass_request_body on; # Indicates whether the original request body is passed to the proxied server. default value is on
                proxy_set_header X-Original-URI $request_uri; # reset uri
        }
}

4. mirror日誌

mirror中不支持配置access_log，解決方法：mirror-location跳轉到server，在server中配置accesslog.

server {
        listen       80;
        server_name  web1.www.com;

        # 源站配置
        location / {
                access_log  /data/nginx/1.14.1/logs/web1/access.log  accesslog;
                mirror /mirror;
                mirror_request_body off;# Indicates whether the client request body is mirrored. default value is on.
                proxy_pass http://web1.upstream.name;
        }

        # 鏡像站點配置
        location /mirror {
                internal; # 內部配置
                # 跳轉到下面的內部server
                proxy_pass http://127.0.0.1:10901$request_uri;
                proxy_pass_request_body off; # Indicates whether the original request body is passed to the proxied server. default value is on
                # Content-Length必須配置在mirror中不然無效
                proxy_set_header Content-Length ""; # mirror_request_body/proxy_pass_request_body都設置爲off，則Conten-length須要設置爲""，不然有坑
                proxy_set_header X-Original-URI $request_uri; # 使用真實的url重置url
        }
}

server {
    # server無法設置爲內部
    listen 127.0.0.1:10901;
    location / {
        # 判斷放在server，使得post請求日誌能夠記錄
        if ($request_method != GET) {
            return 403;
        }
        access_log /data/nginx/1.14.1/logs/web1/access.log  accesslog;
        proxy_pass http://mirror.web1.upstream.name;
    }
}

5. 性能評估

利用jemeter，在相同環境，使用30個併發，各請求3000次get或post方法，參數同樣，一組爲有mirror配置，另外一組爲沒mirror配置，整體評估，mirror性能損失在5%之內，具體以下：

97. 坑

97.1 鏡像配置不正確缺乏相關日誌

鏡像配置不正確，致使複製操做沒正常執行，此時nginx可能缺乏錯誤日誌，嚴重影響調試，因此很是建議配置鏡像日誌，配置方法如4. mirror日誌。部分錯誤配置下錯誤信息在error日誌中。

97.2 mirror_request_body/proxy_pass_request_body與Content-Length需配置一致

若是mirror_request_body或者proxy_pass_request_body設置爲off，則Content-Length必須設置爲""，由於nginx(mirror_request_body)或tomcat(mirror_request_body)處理post請求時，會根據Content-Length獲取請求體，若是Content-Length不爲空，而因爲mirror_request_body或者proxy_pass_request_body設置爲off，處理方覺得post有內容，當request_body中沒有，處理方會一直等待至超時，則前者爲off，nginx會報upstream請求超時；後者爲off，tomcat會報以下錯誤：

"2018-11-08T17:26:36.803+08:00" "331632b86ec64b829672066a96fc6324"      "department"        "group"   "project_name"        "hostname"    "127.0.0.1"     ""      "/post" "p=11"  "-"     "PostmanRuntime/7.1.1"  "ERROR" "xxx.GlobalControllerAdvice"       "operateExp"    "-"     "26"    "xxxx.GlobalControllerAdvice"       "unknown"       "org.springframework.http.converter.HttpMessageNotReadableException"    "I/O error while reading input message; nested exception is java.net.SocketTimeoutException"    "GlobalControllerAdvice中捕獲全局異常"  "org.springframework.http.converter.HttpMessageNotReadableException: I/O error while reading input message; nested exception is java.net.SocketTimeoutException
        at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodArgumentResolver.readWithMessageConverters(AbstractMessageConverterMethodArgumentResolver.java:229)
        at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.readWithMessageConverters(RequestResponseBodyMethodProcessor.java:150)
        at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.resolveArgument(RequestResponseBodyMethodProcessor.java:128)
        at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:121)
        at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:158)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:128)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)

98. 引文

• Module ngx_http_mirror_module
• 流量搬運工，Nginx模塊ngx_http_mirror_module
• How to log nginx mirror requests