聊聊jdk httpclient的retry參數

序

本文主要研究一下jdk httpclient的retry參數

DEFAULT_MAX_ATTEMPTS

java.net.http/jdk/internal/net/http/MultiExchange.java

class MultiExchange<T> {

    static final Logger debug =
            Utils.getDebugLogger("MultiExchange"::toString, Utils.DEBUG);

    private final HttpRequest userRequest; // the user request
    private final HttpRequestImpl request; // a copy of the user request
    final AccessControlContext acc;
    final HttpClientImpl client;
    final HttpResponse.BodyHandler<T> responseHandler;
    final HttpClientImpl.DelegatingExecutor executor;
    final AtomicInteger attempts = new AtomicInteger();
    HttpRequestImpl currentreq; // used for retries & redirect
    HttpRequestImpl previousreq; // used for retries & redirect
    Exchange<T> exchange; // the current exchange
    Exchange<T> previous;
    volatile Throwable retryCause;
    volatile boolean expiredOnce;
    volatile HttpResponse<T> response = null;

    // Maximum number of times a request will be retried/redirected
    // for any reason

    static final int DEFAULT_MAX_ATTEMPTS = 5;
    static final int max_attempts = Utils.getIntegerNetProperty(
            "jdk.httpclient.redirects.retrylimit", DEFAULT_MAX_ATTEMPTS
    );

    //......

}

• 這裏有一個AtomicInteger類型的attempts變量，用來記錄請求次數
• 另外還有一個max_attempts，讀取的是jdk.httpclient.redirects.retrylimit值，讀取不到默認取DEFAULT_MAX_ATTEMPTS，爲5

MultiExchange.responseAsyncImpl

java.net.http/jdk/internal/net/http/MultiExchange.java

private CompletableFuture<Response> responseAsyncImpl() {
        CompletableFuture<Response> cf;
        if (attempts.incrementAndGet() > max_attempts) {
            cf = failedFuture(new IOException("Too many retries", retryCause));
        } else {
            if (currentreq.timeout().isPresent()) {
                responseTimerEvent = ResponseTimerEvent.of(this);
                client.registerTimer(responseTimerEvent);
            }
            try {
                // 1. apply request filters
                // if currentreq == previousreq the filters have already
                // been applied once. Applying them a second time might
                // cause some headers values to be added twice: for
                // instance, the same cookie might be added again.
                if (currentreq != previousreq) {
                    requestFilters(currentreq);
                }
            } catch (IOException e) {
                return failedFuture(e);
            }
            Exchange<T> exch = getExchange();
            // 2. get response
            cf = exch.responseAsync()
                     .thenCompose((Response response) -> {
                        HttpRequestImpl newrequest;
                        try {
                            // 3. apply response filters
                            newrequest = responseFilters(response);
                        } catch (IOException e) {
                            return failedFuture(e);
                        }
                        // 4. check filter result and repeat or continue
                        if (newrequest == null) {
                            if (attempts.get() > 1) {
                                Log.logError("Succeeded on attempt: " + attempts);
                            }
                            return completedFuture(response);
                        } else {
                            this.response =
                                new HttpResponseImpl<>(currentreq, response, this.response, null, exch);
                            Exchange<T> oldExch = exch;
                            return exch.ignoreBody().handle((r,t) -> {
                                previousreq = currentreq;
                                currentreq = newrequest;
                                expiredOnce = false;
                                setExchange(new Exchange<>(currentreq, this, acc));
                                return responseAsyncImpl();
                            }).thenCompose(Function.identity());
                        } })
                     .handle((response, ex) -> {
                        // 5. handle errors and cancel any timer set
                        cancelTimer();
                        if (ex == null) {
                            assert response != null;
                            return completedFuture(response);
                        }
                        // all exceptions thrown are handled here
                        CompletableFuture<Response> errorCF = getExceptionalCF(ex);
                        if (errorCF == null) {
                            return responseAsyncImpl();
                        } else {
                            return errorCF;
                        } })
                     .thenCompose(Function.identity());
        }
        return cf;
    }

• 進入該方法的時候，調用attempts.incrementAndGet()，遞增請求次數，而後判斷有無超出限制，有則返回帶有new IOException("Too many retries", retryCause)異常的failedFuture，即經過CompletableFuture.completeExceptionally返回
• 若是沒有超出限制，可是執行請求失敗，則調用getExceptionalCF來判斷是否應該重試，若是返回null，則重試，經過再次調用responseAsyncImpl，經過這種遞歸調用完成重試邏輯

MultiExchange.getExceptionalCF

java.net.http/jdk/internal/net/http/MultiExchange.java

/**
     * Takes a Throwable and returns a suitable CompletableFuture that is
     * completed exceptionally, or null.
     */
    private CompletableFuture<Response> getExceptionalCF(Throwable t) {
        if ((t instanceof CompletionException) || (t instanceof ExecutionException)) {
            if (t.getCause() != null) {
                t = t.getCause();
            }
        }
        if (cancelled && t instanceof IOException) {
            if (!(t instanceof HttpTimeoutException)) {
                t = toTimeoutException((IOException)t);
            }
        } else if (retryOnFailure(t)) {
            Throwable cause = retryCause(t);

            if (!(t instanceof ConnectException)) {
                if (!canRetryRequest(currentreq)) {
                    return failedFuture(cause); // fails with original cause
                }
            }

            // allow the retry mechanism to do its work
            retryCause = cause;
            if (!expiredOnce) {
                if (debug.on())
                    debug.log(t.getClass().getSimpleName() + " (async): retrying...", t);
                expiredOnce = true;
                // The connection was abruptly closed.
                // We return null to retry the same request a second time.
                // The request filters have already been applied to the
                // currentreq, so we set previousreq = currentreq to
                // prevent them from being applied again.
                previousreq = currentreq;
                return null;
            } else {
                if (debug.on()) {
                    debug.log(t.getClass().getSimpleName()
                            + " (async): already retried once.", t);
                }
                t = cause;
            }
        }
        return failedFuture(t);
    }

    private boolean retryOnFailure(Throwable t) {
        return t instanceof ConnectionExpiredException
                || (RETRY_CONNECT && (t instanceof ConnectException));
    }

    /** Returns true if the given request can be automatically retried. */
    private static boolean canRetryRequest(HttpRequest request) {
        if (RETRY_ALWAYS)
            return true;
        if (isIdempotentRequest(request))
            return true;
        return false;
    }

    /** Returns true is given request has an idempotent method. */
    private static boolean isIdempotentRequest(HttpRequest request) {
        String method = request.method();
        switch (method) {
            case "GET" :
            case "HEAD" :
                return true;
            default :
                return false;
        }
    }

    private Throwable retryCause(Throwable t) {
        Throwable cause = t instanceof ConnectionExpiredException ? t.getCause() : t;
        return cause == null ? t : cause;
    }

    /** True if ALL ( even non-idempotent ) requests can be automatic retried. */
    private static final boolean RETRY_ALWAYS = retryPostValue();
    /** True if ConnectException should cause a retry. Enabled by default */
    private static final boolean RETRY_CONNECT = retryConnect();

    private static boolean retryPostValue() {
        String s = Utils.getNetProperty("jdk.httpclient.enableAllMethodRetry");
        if (s == null)
            return false;
        return s.isEmpty() ? true : Boolean.parseBoolean(s);
    }

    private static boolean retryConnect() {
        String s = Utils.getNetProperty("jdk.httpclient.disableRetryConnect");
        if (s == null)
            return false;
        return s.isEmpty() ? true : Boolean.parseBoolean(s);
    }

若是cancelled爲true且是IOException則直接返回，不然先判斷retryOnFailure再判斷canRetryRequest( 若是不是ConnectException才走canRetryRequest這個判斷)

• retryOnFailure方法判斷若是是ConnectionExpiredException或者是ConnectException且開啓retryConnect，則返回true
• RETRY_CONNECT讀取的是jdk.httpclient.disableRetryConnect參數，若是值爲null，則方法返回false，即不進行retryConnect
• canRetryRequest首先判斷RETRY_ALWAYS，在判斷isIdempotentRequest(GET、HEAD方法才重試)，都不是則返回false
• RETRY_ALWAYS讀取的是jdk.httpclient.enableAllMethodRetry，若是值爲null，則方法返回false，即不進行retryPostValue
• 若是該重試的話，則返回null，responseAsyncImpl裏頭在getExceptionalCF返回null的時候，從新調用了一次responseAsyncImpl，經過遞歸調用來完成重試邏輯

NetProperties

java.base/sun/net/NetProperties.java

public class NetProperties {
    private static Properties props = new Properties();
    static {
        AccessController.doPrivileged(
            new PrivilegedAction<Void>() {
                public Void run() {
                    loadDefaultProperties();
                    return null;
                }});
    }

    private NetProperties() { };


    /*
     * Loads the default networking system properties
     * the file is in jre/lib/net.properties
     */
    private static void loadDefaultProperties() {
        String fname = StaticProperty.javaHome();
        if (fname == null) {
            throw new Error("Can't find java.home ??");
        }
        try {
            File f = new File(fname, "conf");
            f = new File(f, "net.properties");
            fname = f.getCanonicalPath();
            InputStream in = new FileInputStream(fname);
            BufferedInputStream bin = new BufferedInputStream(in);
            props.load(bin);
            bin.close();
        } catch (Exception e) {
            // Do nothing. We couldn't find or access the file
            // so we won't have default properties...
        }
    }

    /**
     * Get a networking system property. If no system property was defined
     * returns the default value, if it exists, otherwise returns
     * <code>null</code>.
     * @param      key  the property name.
     * @throws  SecurityException  if a security manager exists and its
     *          <code>checkPropertiesAccess</code> method doesn't allow access
     *          to the system properties.
     * @return the <code>String</code> value for the property,
     *         or <code>null</code>
     */
    public static String get(String key) {
        String def = props.getProperty(key);
        try {
            return System.getProperty(key, def);
        } catch (IllegalArgumentException e) {
        } catch (NullPointerException e) {
        }
        return null;
    }

    /**
     * Get an Integer networking system property. If no system property was
     * defined returns the default value, if it exists, otherwise returns
     * <code>null</code>.
     * @param   key     the property name.
     * @param   defval  the default value to use if the property is not found
     * @throws  SecurityException  if a security manager exists and its
     *          <code>checkPropertiesAccess</code> method doesn't allow access
     *          to the system properties.
     * @return the <code>Integer</code> value for the property,
     *         or <code>null</code>
     */
    public static Integer getInteger(String key, int defval) {
        String val = null;

        try {
            val = System.getProperty(key, props.getProperty(key));
        } catch (IllegalArgumentException e) {
        } catch (NullPointerException e) {
        }

        if (val != null) {
            try {
                return Integer.decode(val);
            } catch (NumberFormatException ex) {
            }
        }
        return defval;
    }

    /**
     * Get a Boolean networking system property. If no system property was
     * defined returns the default value, if it exists, otherwise returns
     * <code>null</code>.
     * @param   key     the property name.
     * @throws  SecurityException  if a security manager exists and its
     *          <code>checkPropertiesAccess</code> method doesn't allow access
     *          to the system properties.
     * @return the <code>Boolean</code> value for the property,
     *         or <code>null</code>
     */
    public static Boolean getBoolean(String key) {
        String val = null;

        try {
            val = System.getProperty(key, props.getProperty(key));
        } catch (IllegalArgumentException e) {
        } catch (NullPointerException e) {
        }

        if (val != null) {
            try {
                return Boolean.valueOf(val);
            } catch (NumberFormatException ex) {
            }
        }
        return null;
    }

}

• 這裏經過loadDefaultProperties先加載默認配置，讀取的是JAVA_HOME/conf/net.properties文件
• 而後getString、getInteger、getBoolean方法採用的是System.getProperty來讀取，而net.properties值僅僅做爲System.getProperty的defaultValue
• 所以要設置httpclient相關參數，只須要經過System.setProperty或者-D來設置便可
• net.properties

/Library/java/JavaVirtualMachines/jdk-11.jdk/Contents/home/conf/net.properties

java.net.useSystemProxies=false
http.nonProxyHosts=localhost|127.*|[::1]
ftp.nonProxyHosts=localhost|127.*|[::1]
jdk.http.auth.tunneling.disabledSchemes=Basic

net.properties文件默認設置瞭如上四個參數

相關異常

HttpTimeoutException

java.net.http/java/net/http/HttpTimeoutException.java

/**
 * Thrown when a response is not received within a specified time period.
 *
 * @since 11
 */
public class HttpTimeoutException extends IOException {

    private static final long serialVersionUID = 981344271622632951L;

    /**
     * Constructs an {@code HttpTimeoutException} with the given detail message.
     *
     * @param message
     *        The detail message; can be {@code null}
     */
    public HttpTimeoutException(String message) {
        super(message);
    }
}

• 屬於java.net.http包，繼承至IOException
• 若是設置了request的timeout，則註冊ResponseTimerEvent，在超時時拋出HttpTimeoutException: request timed out，同時設置MultiExchange的cancelled爲true
• 這類因爲客戶端設置超時引發的HttpTimeoutException，不會進行重試，即便開啓相關重試參數
• 若是這個時間設置得過短，則在connect的時候就超時了，這個時候會拋出HttpConnectTimeoutException，而非HttpTimeoutException: request timed out

HttpConnectTimeoutException

java.net.http/java/net/http/HttpConnectTimeoutException.java

/**
 * Thrown when a connection, over which an {@code HttpRequest} is intended to be
 * sent, is not successfully established within a specified time period.
 *
 * @since 11
 */
public class HttpConnectTimeoutException extends HttpTimeoutException {

    private static final long serialVersionUID = 321L + 11L;

    /**
     * Constructs an {@code HttpConnectTimeoutException} with the given detail
     * message.
     *
     * @param message
     *        The detail message; can be {@code null}
     */
    public HttpConnectTimeoutException(String message) {
        super(message);
    }
}

• 屬於java.net.http包，繼承至HttpTimeoutException
• 若是設置了client的connectTimeout，則會註冊ConnectTimerEvent，在超時時拋出ConnectException("HTTP connect timed out")，同時設置MultiExchange的cancelled爲true，這個在MultiExchange.getExceptionalCF方法裏頭會被包裝爲HttpConnectTimeoutException

ConnectionExpiredException

java.net.http/jdk/internal/net/http/common/ConnectionExpiredException.java

/**
 * Signals that an end of file or end of stream has been reached
 * unexpectedly before any protocol specific data has been received.
 */
public final class ConnectionExpiredException extends IOException {
    private static final long serialVersionUID = 0;

    /**
     * Constructs a {@code ConnectionExpiredException} with a detail message of
     * "subscription is finished" and the given cause.
     *
     * @param   cause the throwable cause
     */
    public ConnectionExpiredException(Throwable cause) {
        super("subscription is finished", cause);
    }
}

• 通常是在read error的時候觸發，好比EOFException，IOException("connection reset by peer)，或者SSLHandshakeException

小結

jdk httpclient的retry參數涉及到的參數以下：

• jdk.httpclient.redirects.retrylimit(默認爲5，用來控制重試次數，不過實際上還有expiredOnce參數，看代碼貌似頂多重試一次)
• jdk.httpclient.disableRetryConnect(默認爲null，即RETRY_CONNECT爲false，不在ConnectException的時候retry)
• jdk.httpclient.enableAllMethodRetry(默認爲null，即RETRY_ALWAYS爲false，即須要判斷請求方法是否冪等來決定是否重試)

是否重試的判斷邏輯以下：

• 若是重試次數超過限制，則返回失敗，不然往下
• 若是cancelled爲true(這裏若是request設置了timeout，觸發時cancelled設置爲true)且是IOException(例如設置了鏈接超時拋出的HttpConnectTimeoutException)，則不走重試邏輯；不然往下

• 若是retryOnFailure(ConnectionExpiredException，或者ConnectException且開啓retryConnect)，則往下

  • 若是是異常不是ConnectException，則還額外判斷canRetryRequest(判斷該請求類型是否容許重試)，知足則往下
  • 若是expiredOnce爲false，則返回null，即知足重試條件，走遞歸重試

doc

• HttpClient javadoc