java之httpClient 3.x、AsyncHttpClient1.9.x使用總結

時間 2019-12-10

標籤 java httpclient 3.x asynchttpclient1.9.x asynchttpclient 使用總結欄目 Java 简体版

原文原文鏈接

　　首先請大牛們見諒菜鳥重複造輪子的學習方式，本文適合新手看~html

　　下面使用的同步http是HttpClient 3.X的版本，不過早已不在維護，若是剛開始使用http，建議你們都換成4.X版本，別看下面的有關同步http的部分了，4.x效率有質地提升，總結3.X只是由於無奈舊項目還在使用。後面再更新一篇有關4.x的，最新的HttpClient 4.X官方地址：http://hc.apache.org/httpcomponents-client-4.5.x/index.htmlgit

　　但鑑於可能有些舊的系統仍是採用3.X版本的HttpClient，因此本文仍是先記錄下使用方法。github

　　相反下面的異步http是Async Http Client 的1.9.8版本，這個版本仍是挺好的。API請見：http://asynchttpclient.github.io/async-http-client/apidocs/com/ning/http/client/AsyncHttpClient.htmlapache

　　http使用場景不少，據以往經驗，對於客戶端來講，咱們使用http通常會發出如下幾種常見的場景：設計模式

以get方式請求服務器
1. 不帶任何參數
2. 帶上key-value對
以post方式請求服務器
1. 不帶任何參數
2. 帶上key-value對
3. 帶上字節數組
4. 帶上文件
5. 帶上文件+key-value對

　　以上的場景通常能夠知足通常的需求，而後，咱們能夠在這基礎上擴展一點點：假如遇到一個相似於報表的子系統，主系統要在關鍵的邏輯鏈路中「打點」，經過http調用報表子系統記錄一些相關的信息時，那麼若是咱們使用同步http來請求報表子系統的話，一旦報表子系統掛了，那麼確定會影響到主系統的運行。api

　　爲了避免影響到主系統的運行，咱們能夠採用「異步」的方式經過http（AsyncHttpClient ）請求報表子系統，那麼即便子系統掛了，對主系統的關鍵鏈路的執行也不會產生多大的影響。因此，封裝一個http組件，天然而然少不了封裝異步http請求。而異步http所可以作的事情，也應該覆蓋上面提到的幾種場景。數組

　　再者，考慮到效率問題，除非有足夠的理由，不然每次調用http接口，都建立立一個新的鏈接，是至關沒效率的，因此MultiThreadedHttpConnectionManager 誕生了，HttpClient在內部維護一個鏈接池，經過MultiThreadedHttpConnectionManager 咱們能夠設置「默認鏈接數」、「最大鏈接數」、「鏈接超時」、「讀取數據超時」等等配置，從而來提升效率。服務器

　　廢話完了，怎麼實現以上需求呢。app

　　包的引用：異步

　　同步的http我使用的是org.apache.commons.httpclient的HttpClient的3.1版本。

　　maven配置爲：

<!-- httpClient -->
<dependency>
<groupId>commons-httpclient</groupId>
<artifactId>commons-httpclient</artifactId>
<version>3.1</version>
</dependency>

　　異步的http我使用的是com.ning.http.client的AsyncHttpClien的1.9.8版本。注意，其須要依賴幾個日誌相關的組件、分別爲log4j、slf4j、slf4j-log4j

　　maven配置爲：

<!-- slf4j-log4j -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.7</version>
</dependency>
<!-- slf4j -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.5</version>
</dependency>
<!-- log4j -->
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.16</version>
</dependency>
<!-- 異步IO -->
<dependency>
<groupId>com.ning</groupId>
<artifactId>async-http-client</artifactId>
<version>1.9.8</version>
</dependency>

　　爲了實現鏈接池，咱們經過一個工廠類來生成httpClient，爲了上一層方便調用，咱們定義了一個接口，規範了同步、異步http應該實現的方法。包結構以下：

1、同步的HttpClient 3.X

　　從工廠入手，工廠負責初始化httpClient的配置，包括「默認鏈接數」、「最大鏈接數」、「鏈接超時」、「讀取數據超時」等等，不一樣的服務咱們應該建立不一樣的manager，由於不可能咱們調服務A和調服務B使用同一套配置是吧，好比超時時間，應該考慮會有所差別。初始化完配置後，把 manager傳到實現類，在實現類中new HttpClient。

工廠代碼以下：

    // 專門針對xx服務器的鏈接管理對象
    // 由於不一樣服務可能超時等參數不用，因此針對不一樣服務，把鏈接管理對象區分開來，這只是其中一個
    private static MultiThreadedHttpConnectionManager xxconnectionManager = new MultiThreadedHttpConnectionManager();
    
    static {
        // 專門針對xx服務器的鏈接參數
        xxconnectionManager = new MultiThreadedHttpConnectionManager();
        HttpConnectionManagerParams paramsSearch = new HttpConnectionManagerParams();
        paramsSearch.setDefaultMaxConnectionsPerHost(1000);    // 默認鏈接數
        paramsSearch.setMaxTotalConnections(1000);            // 最大鏈接數
        paramsSearch.setConnectionTimeout(30000);             // 鏈接超時
        paramsSearch.setSoTimeout(20000);                     // 讀數據超時
        xxconnectionManager.setParams(paramsSearch);
    }

    /*
     * 返回針對XX服務的httpClient包裝類
     */
    public static SyncHttpClientWapperImpl getXXSearchHttpClient() {
        return new SyncHttpClientWapperImpl(xxconnectionManager);
    }

　　注意一點，這些鏈接數，超時等的配置，要作要調查工做以後再定奪，是根據訪問服務的不一樣，咱們本身的機器能有多少剩餘的可用空間的不一樣而不一樣的，而不是隨隨便便就設置一個參數。

實現類的構造方法以下：

private HttpClient client;// httpClient
    
    private final static String CHARACTER     = "UTF-8";

    // 構造器，由工廠調用
    public SyncHttpClientWapperImpl(MultiThreadedHttpConnectionManager connectionManager) {
        client = new HttpClient(connectionManager);
        // 字符集
        client.getParams().setParameter(HttpMethodParams.HTTP_CONTENT_CHARSET, CHARACTER);
    }

這裏有一個挺困惑的點：HttpClient有必要弄成靜態的嗎？即直接在工廠裏面爲每種服務生成一個靜態的HttpClient，而後傳到實現類？經測試，改爲靜態的效率並無提升，在文件傳輸的測試中，甚至降低了，這個有點困惑，你們能夠試一試一塊兒討論一下。

而後，在實現類中實現各類方法。

第一種，經過URL，以get方式請求服務器，返回字節數組。

public byte[] getWithQueryURL(String queryURL) throws HttpClientException {
        if(queryURL == null) {
            throw new HttpClientException("queryURL is null.");
        }
        byte[] newbuf = executeByGet(queryURL);
        if ((newbuf == null) || (newbuf.length == 0)) {
            throw new HttpClientException("Server response is null: " + queryURL);
        }
        return newbuf;
    }

private byte[] executeByGet(String url) throws HttpClientException {
        HttpMethod method = new GetMethod(url);
        
        // RequestHeader
          method.setRequestHeader("Content-type" , "text/html; charset=UTF-8");
        
        // 提交請求
        try {
            client.executeMethod(method);
        } catch (Exception e) {
            method.releaseConnection();
            throw new HttpClientException(url, e);
        }
        
        // 返回字節流
        byte[] responseBody = null;
        try {
            responseBody = getBytesFromInpuStream(method.getResponseBodyAsStream());
        } catch (IOException e) {
            throw new HttpClientException(e);
        } finally {
            method.releaseConnection();
        }
        return responseBody;
    }

接着，寫一個通用的流解析方法，負責把返回的流解析成字節數組。

private byte[] getBytesFromInpuStream(InputStream instream) throws IOException {
        ByteArrayOutputStream outstream = new ByteArrayOutputStream();
        try {
            int length;
            byte[] tmp = new byte[8096];
            while ((length = instream.read(tmp)) != -1) {
                outstream.write(tmp, 0, length);
            }
            return outstream.toByteArray();
        } finally {
            instream.close();
            outstream.close();
        }
    }

這樣就完成了最簡單的get請求的調用了。

第二種：經過URL和paramsMap參數，以post方式請求服務器，返回字節數組。

public byte[] postWithParamsMap( String queryURL, Map<String,String> paramsMap) throws HttpClientException{
        if(queryURL == null) {
            throw new HttpClientException("queryURL is null.");
        }
        byte[] newbuf = executeByPostWithParamsMap(queryURL,paramsMap);
        if ((newbuf == null) || (newbuf.length == 0)) {
            throw new HttpClientException("Server response is null: " + queryURL);
        }
        return newbuf;
    }

private byte[] executeByPostWithParamsMap(String URL, Map<String,String> paramsMap)  throws HttpClientException {
        PostMethod method = new PostMethod(URL);
        // 構造參數
        if(paramsMap != null) {
            Set<Entry<String, String>> entrySet = paramsMap.entrySet();
            Iterator<Entry<String, String>> iterator = entrySet.iterator();
            NameValuePair[] nvps = new NameValuePair[paramsMap.size()];
            int i = 0 ;
            while(iterator.hasNext()) {
                Entry<String, String> entry = iterator.next();
                if(entry.getKey() != null) {
                    NameValuePair nvp = new NameValuePair(entry.getKey(),entry.getValue());
                    nvps[i++] = nvp;
                }
            }
            method.setRequestBody(nvps);
        }
        
        // RequestHeader，key-value對的話，httpClient自動帶上application/x-www-form-urlencoded
        method.setRequestHeader("Content-type" , "application/x-www-form-urlencoded; charset=UTF-8");
        
        // 提交請求
        try {
            client.executeMethod(method);
        } catch (Exception e) {
            method.releaseConnection();
            throw new HttpClientException(URL, e);
        }
        // 返回字節流
        byte[] responseBody = null;
        try {
            responseBody = getBytesFromInpuStream(method.getResponseBodyAsStream());
        } catch (IOException e) {
            throw new HttpClientException(e);
        } finally {
            method.releaseConnection();
        }
        return responseBody;
    }

第三種：經過URL和bytes參數，以post方式請求服務器，返回字節數組。

public byte[] postWithBytes(String queryURL , byte[] bytes) throws HttpClientException{
        if(queryURL == null) {
            throw new HttpClientException("queryURL is null.");
        }
        byte[] newbuf = executeByPostWithBytes(queryURL,bytes);
        if ((newbuf == null) || (newbuf.length == 0)) {
            throw new HttpClientException("Server response is null: " + queryURL);
        }
        return newbuf;
    }

private byte[] executeByPostWithBytes(String queryURL, byte[] bytes) throws HttpClientException {
        PostMethod method = new PostMethod(queryURL);
        RequestEntity requestEntity = new ByteArrayRequestEntity(bytes);
        method.setRequestEntity(requestEntity);
        
        // RequestHeader
        method.setRequestHeader("Content-type" , "text/plain; charset=UTF-8");
        
        // 提交請求
        try {
            client.executeMethod(method);
        } catch (Exception e) {
            method.releaseConnection();
            throw new HttpClientException(queryURL, e);
        }
        // 返回字節流
        byte[] responseBody = null;
        try {
            responseBody = getBytesFromInpuStream(method.getResponseBodyAsStream());
        } catch (IOException e) {
            throw new HttpClientException(e);
        } finally {
            method.releaseConnection();
        }
        return responseBody;
    }

第四種：經過URL、fileList、paramMap參數，以post方式請求服務器，返回字節數組。

public byte[] postWithFileListAndParamMap(String queryURL,List<File> fileList,Map<String,String> paramMap) throws HttpClientException, HttpException, IOException {
        if(queryURL == null) {
            throw new HttpClientException("queryURL is null.");
        }
        if(fileList == null) {
            throw new HttpClientException("file is null.");
        }
        if(paramMap == null){
            throw new HttpClientException("paramMap is null.");
        }
        return executeByPostWithFileListAndParamMap(queryURL, fileList, paramMap);
    }

private byte[] executeByPostWithFileListAndParamMap (String queryURL,List<File> fileList,Map<String,String> paramMap) throws HttpException, IOException, HttpClientException {
        if(queryURL != null && fileList != null && fileList.size() > 0) {
            // post方法
            PostMethod method = new PostMethod(queryURL);
            
            // Part[]
            Part[] parts = null;
            if(paramMap != null) {
                parts = new Part[fileList.size()+paramMap.size()];
            }
            else {
                parts = new Part[fileList.size()];
            }
            int i = 0 ;
            
            // FilePart
            for(File file : fileList){
                Part filePart = new FilePart(file.getName(),file);
                parts[i++] = filePart;
            }
            
            // StringPart
            if(paramMap != null ) {
                Set<Entry<String, String>> entrySet = paramMap.entrySet();
                Iterator<Entry<String, String>> it = entrySet.iterator();
                while(it.hasNext()) {
                    Entry<String, String> entry = it.next();
                    Part stringPart = new StringPart(entry.getKey(),entry.getValue(),CHARACTER);
                    parts[i++] = stringPart;
                }
            }
            
            // Entity
            RequestEntity requestEntity = new MultipartRequestEntity(parts, method.getParams());
            method.setRequestEntity(requestEntity);
            
            // RequestHeader，文件的話，HttpClient自動加上multipart/form-data
//            method.setRequestHeader("Content-type" , "multipart/form-data; charset=UTF-8");
            
            // excute
            try {
                client.executeMethod(method);
            } catch (Exception e) {
                method.releaseConnection();
                throw new HttpClientException(queryURL, e);
            }
            
            // return 
            byte[] responseBody = null;
            try {
                responseBody = getBytesFromInpuStream(method.getResponseBodyAsStream());
            } catch (IOException e) {
                throw new HttpClientException(e);
            } finally {
                method.releaseConnection();
            }
            return responseBody;
        }
        return null;
    }

注意Part stringPart = new StringPart(entry.getKey(),entry.getValue(),CHARACTER);這句代碼，必定要加上編碼格式，否則服務端會亂碼。

2、異步的AsyncHttpClient

一樣的，按照這種思路，異步的AsyncHttpClient也有相似的實現，不過寫法不一樣而已，在工廠中，AsyncHttpClient使用的是AsyncHttpClientConfig.Builder做爲管理配置的類，也有相似鏈接超時，最大鏈接數等配置。

工廠類：

// 專門針對xx服務器的鏈接管理對象
    // 由於不一樣服務可能超時等參數不用，因此針對不一樣服務，把鏈接管理對象區分開來，這只是其中一個
    private static AsyncHttpClientConfig.Builder xxbuilder = new AsyncHttpClientConfig.Builder();

    static {
        xxbuilder.setConnectTimeout(3000);    // 鏈接超時
        xxbuilder.setReadTimeout(2000);        // 讀取數據超時
        xxbuilder.setMaxConnections(1000);    // 最大鏈接數
    }

    /*
     * 返回針對XX服務的httpClient包裝類
     */
    public static AsyncHttpClientWapperImpl getXXSearchHttpClient() {
        return new AsyncHttpClientWapperImpl(xxbuilder);
    }

其使用了builder 的設計模式，活生生的一個例子，值得學習。

實現類的構造方法：

    private AsyncHttpClient client;
    
    public AsyncHttpClientWapperImpl(Builder xxbuilder) {
        client = new AsyncHttpClient(xxbuilder.build());
    }

這樣，AsyncHttpClient對象就建立完畢了。接下來是各類場景的實現，感受異步的AsyncHttpClient封裝得比HttpClient 3.X更加容易使用，設計得更好。

第一種：經過URL，以get方式請求服務器，返回字節數組。

    public byte[] getWithQueryURL(String queryURL)
            throws HttpClientException, HttpClientException {
        if(queryURL == null) {
            throw new HttpClientException("queryURL爲空");
        }
        byte[] newbuf = executeByGet(queryURL);
        if ((newbuf == null) || (newbuf.length == 0)) {
            throw new HttpClientException("Server response is null: " + queryURL);
        }
        return newbuf;
    }

    private byte[] executeByGet(String queryURL) throws HttpClientException {
         byte[] responseBody = null;
        try {
            
            Future<Response> f = client.prepareGet(queryURL).execute();  
            responseBody = getBytesFromInpuStream(f.get().getResponseBodyAsStream());
        } catch (Exception e) {
            throw new HttpClientException(e);
        }
        return responseBody;
    }

一樣的，咱們寫了一個getBytesFromInputStream()方法解析服務端返回的流，咱們發現，兩個實現類裏面都有一些共同的方法，這裏能夠考慮寫一個父類，把這些方法提取出來。

第二種：經過URL和paramsMap參數，以post方式請求服務器，返回字節數組。

public byte[] postWithParamsMap(String queryURL,
            Map<String, String> paramsMap) throws HttpClientException {
        if(queryURL == null) {
            throw new HttpClientException("queryURL爲空");
        }
        byte[] newbuf = executeByPostByParamMap(queryURL,paramsMap);
        if ((newbuf == null) || (newbuf.length == 0)) {
            throw new HttpClientException("Server response is null: " + queryURL);
        }
        return newbuf;
    }

private byte[] executeByPostByParamMap(String queryURL,Map<String,String> paramsMap) throws HttpClientException {
        byte[] responseBody = null;
        try {
            RequestBuilder requestBuilder = new RequestBuilder();
            
            // 添加 key-value參數
            if(paramsMap != null && paramsMap.size() > 0) {
                Set<Entry<String, String>> entrySet = paramsMap.entrySet();
                Iterator<Entry<String, String>> iterator = entrySet.iterator();
                while(iterator.hasNext()) {
                    Entry<String, String> entry = iterator.next();
                    if(entry.getKey() != null) {
                        requestBuilder.addFormParam(entry.getKey(), entry.getValue());
                    }
                }
            }
            
            // 添加RequestHeader，key
            requestBuilder.addHeader("Content-type", "application/x-www-form-urlencoded; charset=UTF-8");
            requestBuilder.setMethod("POST");
            // 添加URL
            requestBuilder.setUrl(queryURL);
            
            // request
            Request request = requestBuilder.build();
            
            // 提交
            ListenableFuture<Response> f = client.executeRequest(request);
            responseBody = getBytesFromInpuStream(f.get().getResponseBodyAsStream());
        } catch (Exception e) {
            throw new HttpClientException(e);
        }
        return responseBody;
    }

第三種：經過URL和bytes參數，以post方式請求服務器，返回字節數組。

public byte[] postWithBytes(String queryURL, byte[] bytes)
            throws HttpClientException {
        if(queryURL == null) {
            throw new HttpClientException("queryURL is null.");
        }
        byte[] newbuf = executeByPostWithBytes(queryURL,bytes);
        if ((newbuf == null) || (newbuf.length == 0)) {
            throw new HttpClientException("Server response is null: " + queryURL);
        }
        return newbuf;
    }

private byte[] executeByPostWithBytes(String queryURL, byte[] bytes) throws HttpClientException {
        byte[] responseBody = null;
        try {
            RequestBuilder requestBuilder = new RequestBuilder();
            
            // 添加 bytes參數
            requestBuilder.setBody(bytes);
            
            // 添加RequestHeader，key
            requestBuilder.addHeader("Content-type", "text/plain; charset=UTF-8");
            requestBuilder.setMethod("POST");
            // 添加URL
            requestBuilder.setUrl(queryURL);
            
            // request
            Request request = requestBuilder.build();
            
            // 提交
            ListenableFuture<Response> f = client.executeRequest(request);
            responseBody = getBytesFromInpuStream(f.get().getResponseBodyAsStream());
        } catch (Exception e) {
            throw new HttpClientException(e);
        }
        return responseBody;
    }

第四種：經過URL、fileList、paramMap參數，以post方式請求服務器，返回字節數組。

public byte[] postWithFileListAndParamMap(String queryURL,
            List<File> fileList, Map<String, String> paramMap)
            throws HttpClientException, HttpException, IOException {
        if(queryURL == null) {
            throw new HttpClientException("queryURL is null.");
        }
        if(fileList == null || fileList.size() == 0) {
            throw new HttpClientException("fileList is null.");
        }
        if(paramMap == null || paramMap.size() == 0) {
            throw new HttpClientException("paramMap is null.");
        }
        return executeByPostWithFileListAndParamMap(queryURL, fileList, paramMap);
    }

private byte[] executeByPostWithFileListAndParamMap (String queryURL,List<File> fileList,Map<String,String> paramsMap) throws HttpException, IOException, HttpClientException {
        if(queryURL != null && fileList != null && fileList.size() > 0) {
            byte[] responseBody = null;
            try {
                RequestBuilder requestBuilder = new RequestBuilder();
                
                // FilePart
                for(File file : fileList){
                    Part filePart = new FilePart(file.getName(),file);
                    requestBuilder.addBodyPart(filePart);
                }
                
                // StringPart
                if(paramsMap != null ) {
                    Set<Entry<String, String>> entrySet = paramsMap.entrySet();
                    Iterator<Entry<String, String>> it = entrySet.iterator();
                    while(it.hasNext()) {
                        Entry<String, String> entry = it.next();
                        Part stringPart = new StringPart(entry.getKey(),entry.getValue());
                        requestBuilder.addBodyPart(stringPart);
                    }
                }
                
                // 添加RequestHeader，key
                requestBuilder.addHeader("Content-type", "multipart/form-data; charset=UTF-8");
                requestBuilder.setMethod("POST");
                // 添加URL
                requestBuilder.setUrl(queryURL);
                
                // request
                Request request = requestBuilder.build();
                
                // 提交
                ListenableFuture<Response> f = client.executeRequest(request);
                responseBody = getBytesFromInpuStream(f.get().getResponseBodyAsStream());
            } catch (Exception e) {
                throw new HttpClientException(e);
            }
            return responseBody;
        }
        return null;
    }