聊聊nginx與tomcat的5xx

時間 2019-11-07

標籤聊聊 nginx tomcat 5xx 欄目 Nginx 简体版

原文原文鏈接

序

本文主要講述一下nginx與tomcat的50二、50四、503錯誤及其常見的產生緣由。html

502

定義

502 Bad Gateway : 做爲網關或者代理工做的服務器嘗試執行請求時，從上游服務器接收到無效的響應。前端

常見緣由

後端服務掛了的狀況,直接502
後端服務在重啓

實例

將後端服務關掉，而後向nginx發送請求後端接口，日誌以下：

access.log

127.0.0.1 - - [22/Dec/2017:20:44:38 +0800] "GET /timeout/long-write HTTP/1.1" 502 537 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36"

error.log

2017/12/22 20:45:12 [error] 1481#0: *3 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 127.0.0.1, server: localhost, request: "GET /timeout/long-write HTTP/1.1", upstream: "http://[::1]:8080/timeout//long-write", host: "localhost:8888"

504

定義

504 Gateway Timeout : 做爲網關或者代理工做的服務器嘗試執行請求時，未能及時從上游服務器（URI標識出的服務器，例如HTTP、FTP、LDAP）或者輔助服務器（例如DNS）收到響應。注意：某些代理服務器在DNS查詢超時時會返回400或者500錯誤java

常見緣由

該接口太耗時，後端服務接收到請求，開始執行，未能在設定時間返回數據給nginx
後端服務器總體負載過高，接受到請求以後，因爲線程繁忙，未能安排給請求的接口，致使未能在設定時間返回數據給nginx

實例

前端返回

<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>openresty/1.9.15.1</center>
</body>
</html>
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->

access.log

192.168.99.1 - - [22/Dec/2017:21:58:20 +0800] "GET /timeout/long-resp HTTP/1.1" 504 591 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36" "-" "-"

error.log

2017/12/22 21:58:20 [error] 5#5: *7 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.99.1, server: , request: "GET /timeout/long-resp HTTP/1.1", upstream: "http://192.168.99.100:8080/timeout//long-resp", host: "192.168.99.100:8686"

nginx.conf

location /timeout/long-resp {
            proxy_connect_timeout    30;
            proxy_read_timeout       100;
            proxy_send_timeout       10;
            proxy_pass http://192.168.99.100:8080/timeout/long-resp ;
        }

java代碼

@GetMapping("/timeout/long-resp")
    public String longResp() throws InterruptedException {
        TimeUnit.SECONDS.sleep(120);
        return "finish";
    }

服務器接受請求一直沒有返回,nginx在等待100秒後報Connection timed out,返回504；可是後端繼續執行，在第120秒才執行完。

503(`相對少見`)

定義

503 Service Unavailable : 表示服務器當前處於暫時不可用狀態，不管是有意仍是無心，當服務器端處於沒法應答的狀態時，就會返回該狀態碼。其中，服務端因維護須要而中止服務屬於有意的狀況。而當服務器自身負載太高，處於沒法響應的狀態時，則屬於無心的狀況。另外，負載均衡器或者web服務器的前置機等這些地方的服務器也有可能返回503.nginx

常見緣由

nginx進行限流，超過限速則返回503
後端服務進行常規維護，好比pause tomcat

nginx限流返回503實例

config

http{
    ## test 503
    limit_conn_zone $binary_remote_addr zone=addr:10m;
    server {
        listen 8686;
        location /timeout {
            limit_conn addr 1;
            proxy_connect_timeout    30;
            proxy_read_timeout       100;
            proxy_send_timeout       2;
            proxy_pass http://192.168.99.100:8080/timeout/ ;
        }
    }
}

error.log

2017/12/24 20:58:29 [error] 5#5: *1473 limiting connections by zone "addr", client: 192.168.99.1, server: , request: "GET /timeout/busy HTTP/1.1", host: "192.168.99.100:8686"

access.log

192.168.99.1 - - [24/Dec/2017:20:58:39 +0800] "GET /timeout/busy HTTP/1.1" 503 219 "-" "-" "-" "-"

client

wrk -t12 -c200 -d100s -T60s  --latency http://192.168.99.100:8686/timeout/busy
➜  ~ curl -i http://192.168.99.100:8686/timeout/busy
HTTP/1.1 503 Service Temporarily Unavailable
Server: openresty/1.9.15.1
Date: Sun, 24 Dec 2017 12:58:26 GMT
Content-Type: text/html
Content-Length: 219
Connection: keep-alive
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>openresty/1.9.15.1</center>
</body>
</html>

tomcat返回503實例

Http11Processor

tomcat-embed-core-8.5.23-sources.jar!/org/apache/coyote/http11/Http11Processor.javaweb

@Override
    public SocketState service(SocketWrapperBase<?> socketWrapper)
        throws IOException {
        RequestInfo rp = request.getRequestProcessor();
        rp.setStage(org.apache.coyote.Constants.STAGE_PARSE);

        // Setting up the I/O
        setSocketWrapper(socketWrapper);
        inputBuffer.init(socketWrapper);
        outputBuffer.init(socketWrapper);

        // Flags
        keepAlive = true;
        openSocket = false;
        readComplete = true;
        boolean keptAlive = false;
        SendfileState sendfileState = SendfileState.DONE;

        while (!getErrorState().isError() && keepAlive && !isAsync() && upgradeToken == null &&
                sendfileState == SendfileState.DONE && !endpoint.isPaused()) {
                //......

                if (endpoint.isPaused()) {
                    // 503 - Service unavailable
                    response.setStatus(503);
                    setErrorState(ErrorState.CLOSE_CLEAN, null);
                } else {
                    keptAlive = true;
                    // Set this every time in case limit has been changed via JMX
                    request.getMimeHeaders().setLimit(endpoint.getMaxHeaderCount());
                    if (!inputBuffer.parseHeaders()) {
                        // We've read part of the request, don't recycle it
                        // instead associate it with the socket
                        openSocket = true;
                        readComplete = false;
                        break;
                    }
                    if (!disableUploadTimeout) {
                        socketWrapper.setReadTimeout(connectionUploadTimeout);
                    }
                }

        }    
    }

只要endpoint的狀態是paused，則返回503

AbstractEndpoint

tomcat-embed-core-8.5.23-sources.jar!/org/apache/tomcat/util/net/AbstractEndpoint.javaapache

/**
     * Pause the endpoint, which will stop it accepting new connections.
     */
    public void pause() {
        if (running && !paused) {
            paused = true;
            unlockAccept();
            getHandler().pause();
        }
    }

    /**
     * Resume the endpoint, which will make it start accepting new connections
     * again.
     */
    public void resume() {
        if (running) {
            paused = false;
        }
    }

這裏是endpoint的pause以及resume方法

請求

當請求進入Http11Processor的service方法到執行endpoint.isPaused()方法期間，tomcat被pause了，這個時候，就會返回503，以下：

➜  ~ curl -i http://localhost:8080/demo/test
HTTP/1.1 503
Transfer-Encoding: chunked
Date: Sun, 24 Dec 2017 14:10:16 GMT
Connection: close

小結

一般是後端服務掛了或在重啓

一般是請求的接口執行耗時，亦或是後端服務負載高，執行耗時

一般是nginx限流或後端服務pause進行維護

doc

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。

聊聊nginx與tomcat的5xx

序

502

定義

常見緣由

實例

504

定義

常見緣由

實例

503(相對少見)

定義

常見緣由

nginx限流返回503實例

tomcat返回503實例

小結

doc

503(`相對少見`)