nginx的upstream異常

異常

upstream server temporarily disabled while connecting to upstream

no live upstreams while connecting to upstream

max_fails與fail_timeout

max_fails默認值爲1,fail_timeout默認值爲10秒。

nginx能夠經過設置max_fails（最大嘗試失敗次數）和fail_timeout（失效時間，在到達最大嘗試失敗次數後，在fail_timeout的時間範圍內節點被置爲失效，除非全部節點都失效，不然該時間內，節點不進行恢復）對節點失敗的嘗試次數和失效時間進行設置，當超過最大嘗試次數或失效時間未超過配置失效時間，則nginx會對節點狀會置爲失效狀態，nginx不對該後端進行鏈接，直到超過失效時間或者全部節點都失效後，該節點從新置爲有效，從新探測.

upstream backend {
    server backend1.example.com weight=5;
    server 127.0.0.1:8080       max_fails=3 fail_timeout=30s;
    server unix:/tmp/backend3;

    server backup1.example.com  backup;
}

fail的標準

好比

connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "POST /demo HTTP/1.1", subrequest: "/capture/getstatus", upstream: "http://192.168.99.100:8080/api/demo/

好比

upstream timed out (110: Connection timed out) while reading response header from upstream

Nginx 默認判斷失敗節點狀態以connect refuse和time out狀態爲準，不以HTTP錯誤狀態進行判斷失敗，由於HTTP只要能返回狀態說明該節點還能夠正常鏈接，因此nginx判斷其仍是存活狀態；除非添加了proxy_next_upstream指令設置對40四、50二、50三、50四、500和time out等錯誤進行轉到備機處理，在next_upstream過程當中，會對fails進行累加，若是備用機處理仍是錯誤則直接返回錯誤信息（但404不進行記錄到錯誤數，若是不配置錯誤狀態也不對其進行錯誤狀態記錄），綜述，nginx記錄錯誤數量只記錄timeout 、connect refuse、50二、500、50三、504這6種狀態，timeout和connect refuse是永遠被記錄錯誤狀態，而50二、500、50三、504只有在配置proxy_next_upstream後nginx纔會記錄這4種HTTP錯誤到fails中，當fails大於等於max_fails時，則該節點失效.

探測機制

若是探測全部節點均失效，備機也爲失效時，那麼nginx會對全部節點恢復爲有效，從新嘗試探測有效節點，若是探測到有效節點則返回正確節點內容，若是仍是所有錯誤，那麼繼續探測下去，當沒有正確信息時，節點失效時默認返回狀態爲502，可是下次訪問節點時會繼續探測正確節點，直到找到正確的爲止。

實驗log

upstream test_server{
        server 192.168.99.100:80801;
        server 192.168.99.100:80802;
        server 192.168.99.100:80803;
    }
##for capture
location /api/test/demo{
            proxy_pass http://test_server/api/demo;
}    
location /api/demo{
            default_type application/json;
            content_by_lua_file conf/lua/demo.lua;
}

lua

local cjson = require "cjson.safe"
testres = ngx.location.capture("/api/test/demo",{
    method= ngx.HTTP_POST,
    body = "arg1=xxxx&arg2=xxxxx"
})
ngx.log(ngx.ERR,"status"..testres.status)
local testbody = cjson.decode(testres.body)
ngx.log(ngx.ERR,testbody==nil)

請求192.168.99.100:8080/api/demo,裏頭的lua會發起一個capture，請求/api/test/demo

請求一次

2017/02/09 14:48:57 [error] 5#5: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.99.1, server: , request: "POST /api/demo HTTP/1.1", subrequest: "/api/test/demo", upstream: "http://192.168.99.100:80801/api/demo", host: "192.168.99.100:8080"
2017/02/09 14:48:57 [warn] 5#5: *1 upstream server temporarily disabled while connecting to upstream, client: 192.168.99.1, server: , request: "POST /api/demo HTTP/1.1", subrequest: "/api/test/demo", upstream: "http://192.168.99.100:80801/api/demo", host: "192.168.99.100:8080"
2017/02/09 14:48:57 [error] 5#5: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.99.1, server: , request: "POST /api/demo HTTP/1.1", subrequest: "/api/test/demo", upstream: "http://192.168.99.100:80802/api/demo", host: "192.168.99.100:8080"
2017/02/09 14:48:57 [warn] 5#5: *1 upstream server temporarily disabled while connecting to upstream, client: 192.168.99.1, server: , request: "POST /api/demo HTTP/1.1", subrequest: "/api/test/demo", upstream: "http://192.168.99.100:80802/api/demo", host: "192.168.99.100:8080"
2017/02/09 14:48:57 [error] 5#5: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.99.1, server: , request: "POST /api/demo HTTP/1.1", subrequest: "/api/test/demo", upstream: "http://192.168.99.100:80803/api/demo", host: "192.168.99.100:8080"
2017/02/09 14:48:57 [warn] 5#5: *1 upstream server temporarily disabled while connecting to upstream, client: 192.168.99.1, server: , request: "POST /api/demo HTTP/1.1", subrequest: "/api/test/demo", upstream: "http://192.168.99.100:80803/api/demo", host: "192.168.99.100:8080"
2017/02/09 14:48:57 [error] 5#5: *1 [lua] demo.lua:44: status502 while sending to client, client: 192.168.99.1, server: , request: "POST /api/demo HTTP/1.1", host: "192.168.99.100:8080"

對upstream逐個請求，都失敗，則capture的subrequest返回502，對client返回的status code取決於lua腳本

再請求一次

2017/02/09 15:09:34 [error] 6#6: *11 no live upstreams while connecting to upstream, client: 192.168.99.1, server: , request: "POST /api/demo HTTP/1.1", subrequest: "/api/test/demo", upstream: "http://test_server/api/demo", host: "192.168.99.100:8080"

該upstream下面的server都掛的狀況下出現no live upstreams while connecting to upstream

doc

• ngx_http_upstream_module

• nginx中健康檢查(health_check)機制深刻分析

• nginx upstream 容錯機制 原創-胡志廣

• 線上nginx的一次「no live upstreams while connecting to upstream 」分析