nginx dynamic resolve upstream servers

問題

upstream不可resolve會阻塞start/reload

使用nginx時, upstream中如有server不可resolve. start/reload都會失敗.
當把nginx做爲網關時, 可能有多個服務, 任意服務掛掉. 致使整個網關不可啓動. 這明顯是不合理的.

only resolve on start/reload

默認狀況下, nginx會在start/reload時解析upstream裏的server, 並緩存ips. 由於docker容器重啓時, ip會變化, 這會致使緩存的ip失效.

解決方案

Use nginx variables

https://serverfault.com/questions/700894/make-nginx-ignore-site-config-when-its-upstream-cannot-be-reached
https://stackoverflow.com/questions/32845674/setup-nginx-not-to-crash-if-host-in-upstream-is-not-found
https://sandro-keil.de/blog/let-nginx-start-if-upstream-host-is-unavailable-or-down/
如:

resolver 127.0.0.11 valid=30s;
    location / {
        set $target xxx_server_name:3031;
        uwsgi_pass $target;
    }

優勢

能夠解決上述兩個問題.

缺點

1. 不支持多個server配置.
2. 沒法使用nginx upstream帶來的好處, 如各類負載均衡策略等等.

tengine ngx_http_upstream_dynamic

https://github.com/alibaba/tengine/blob/master/modules/ngx_http_upstream_dynamic_module/ngx_http_upstream_dynamic_module.c
https://tengine.taobao.org/document_cn/http_upstream_dynamic_cn.html
如:

upstream backend {
    dynamic_resolve fallback=stale fail_timeout=30s;

    server a.com;
    server b.com;
}

server {
    ...

    location / {
        proxy_pass http://backend;
    }
}

優勢

1. 大廠, 用的人多.
2. 能夠解決only resolve on start/reload問題.
3. 能夠和其它nginx-module配合.

缺點

1. start/reload時有不可resolve的server會失敗.

jdomain

https://github.com/wdaike/ngx...

upstream backend {                                                        
    jdomain www.baidu.com port=80;                                                    
    jdomain www.baidu.com port=81; # 只有一個有效
}

優勢

能夠解決上述兩個問題.

缺點

1. 不支持server配置, 不支持多個server, 即jdomain的配置, 沒法和其它nginx-module配合.
2. 不支持未resolved狀況下的啓動 (有一個pr能夠解決這個問題). https://github.com/wdaike/ngx_upstream_jdomain/pull/12

nginx-upstream-dynamic-servers

https://github.com/GUI/nginx-...

優勢

1. 能夠解決上述兩個問題.
2. 能夠和其它nginx-module配合.

缺點

1. 替換了nginx server directive, 須要隨着nginx版本變動維護. 做者測試了的nginx version: 1.6, 1.7, 1.8, 1.9
2. 用的人不夠多.

維護和測試

咱們使用的是tengine-2.2.2, 看代碼能夠知道. 替換tengine server directive會致使id/host 沒有設置到ngx_http_upstream_server_t結構體中. 由於id/host在咱們的使用場景中都沒有用到. 因此沒有問題. 若是要用到這2個配置, 能夠考慮fork一份代碼出來改.

nginx-upstream-dynamic-servers

ngx_http_upstream_dynamic_servers.c:260

#if nginx_version >= 1007002
    us->name = u.url;
#endif
    us->addrs = u.addrs;
    us->naddrs = u.naddrs;
    us->weight = weight;
    us->max_fails = max_fails;
    us->fail_timeout = fail_timeout;

    return NGX_CONF_OK;

tengine-2.2.2

ngx_http_upstream_server_t:

typedef struct {
    ngx_str_t                        name;
    ngx_addr_t                      *addrs;
    ngx_uint_t                       naddrs;
    ngx_uint_t                       weight;
    ngx_uint_t                       max_fails;
    time_t                           fail_timeout;
    ngx_str_t                        id;
    ngx_str_t                        host;

    unsigned                         down:1;
    unsigned                         backup:1;
} ngx_http_upstream_server_t;

ngx_http_upstream.c:5610

us->name = u.url;
    us->addrs = u.addrs;
    us->naddrs = u.naddrs;
    us->host = u.host;
    us->weight = weight;
    us->max_fails = max_fails;
    us->fail_timeout = fail_timeout;
    us->id = id;

    return NGX_CONF_OK;

測試

由於nginx-upstream-dynamic-servers須要隨着nginx/tengine版本升級而維護server directive, 因此這裏提一下測試.
https://github.com/alibaba/tengine/wiki/How-to-test

diff --git a/tests/nginx-tests/nginx-tests/upstream.t b/tests/nginx-tests/nginx-tests/upstream.t
index debbf5d..5dd670b 100644
--- a/tests/nginx-tests/nginx-tests/upstream.t
+++ b/tests/nginx-tests/nginx-tests/upstream.t
@@ -37,8 +37,8 @@ http {
     %%TEST_GLOBALS_HTTP%%
 
     upstream u {
-        server 127.0.0.1:8081 max_fails=3 fail_timeout=10s;
-        server 127.0.0.1:8082 max_fails=3 fail_timeout=10s;
+        server 127.0.0.1:8081 max_fails=3 fail_timeout=10s resolve;
+        server 127.0.0.1:8082 max_fails=3 fail_timeout=10s resolve;
     }

~/git/tengine/tests(52fff0e*) » TEST_NGINX_BINARY=/usr/sbin/nginx prove -r nginx-tests
...
nginx-tests/nginx-tests/upstream.t ........................... ok   
nginx-tests/nginx-tests/upstream_hash.t ...................... ok     
nginx-tests/nginx-tests/upstream_hash_memcached.t ............ skipped: Cache::Memcached not installed
nginx-tests/nginx-tests/upstream_ip_hash.t ................... skipped: no realip available
nginx-tests/nginx-tests/upstream_least_conn.t ................ ok   
nginx-tests/nginx-tests/upstream_zone_ssl.t .................. skipped: no http_ssl available
nginx-tests/nginx-tests/userid.t ............................. 1/34 skip() needs to know $how_many tests are in the block at nginx-tests/nginx-tests/userid.t line 185
...

總結

Use nginx variables/jdomain 沒法和其它nginx-upstream-module很好地配合使用, 不推薦在要求較高的產品中使用.nginx-upstream-dynamic-servers/tengine ngx_http_upstream_dynamic 能夠按實際狀況使用.