spring cloud使用@FeignClient在微服務之間調用,間歇性出現404

環境: jdk 1.8 + spring cloud + Dalston.M1 (使用的openFeign爲9.3.1)

問題描述:

微服務之間使用FeignClient調用,

針對在異步@Async註解標識的方法中請求其它微服務,間歇性出現404錯誤碼.

在logstore日誌裏面,發現間歇性的,而且是非必現的場景,就讓人很頭疼.

在同步的方法中,生成環境的日誌未出現過404日誌. 

at java.lang.Thread.run(Thread.java:745)
Caused by: feign.FeignException: status 404 reading XXXXClient#getXXXXX(String); content:
{"timestamp":1507687795225,"status":404,"error":"Not Found","message":"No message available","path":"/api/user/xxxx/yyyyyyy"}
	at feign.FeignException.errorStatus(FeignException.java:62)
	at feign.codec.ErrorDecoder$Default.decode(ErrorDecoder.java:91)
	at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:138)
	at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:76)
	at feign.hystrix.HystrixInvocationHandler$1.run(HystrixInvocationHandler.java:108)
	at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:302)
	at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:298)
	at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46)

跟蹤源代碼,一進去,就是一大片一大片的代碼, 

Object executeAndDecode(RequestTemplate template) throws Throwable {
    Request request = targetRequest(template);

    if (logLevel != Logger.Level.NONE) {
      logger.logRequest(metadata.configKey(), logLevel, request);
    }

    Response response;
    long start = System.nanoTime();
    try {
      response = client.execute(request, options);
      // ensure the request is set. TODO: remove in Feign 10
      response.toBuilder().request(request).build();
    } catch (IOException e) {
      if (logLevel != Logger.Level.NONE) {
        logger.logIOException(metadata.configKey(), logLevel, e, elapsedTime(start));
      }
      throw errorExecuting(request, e);
    }
    long elapsedTime = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start);

    boolean shouldClose = true;
    try {
      if (logLevel != Logger.Level.NONE) {
        response =
            logger.logAndRebufferResponse(metadata.configKey(), logLevel, response, elapsedTime);
        // ensure the request is set. TODO: remove in Feign 10
        response.toBuilder().request(request).build();
      }
      if (Response.class == metadata.returnType()) {
        if (response.body() == null) {
          return response;
        }
        if (response.body().length() == null ||
                response.body().length() > MAX_RESPONSE_BUFFER_SIZE) {
          shouldClose = false;
          return response;
        }
        // Ensure the response body is disconnected
        byte[] bodyData = Util.toByteArray(response.body().asInputStream());
        return response.toBuilder().body(bodyData).build();
      }
      if (response.status() >= 200 && response.status() < 300) {
        if (void.class == metadata.returnType()) {
          return null;
        } else {
          return decode(response);
        }
      } else if (decode404 && response.status() == 404) {
        return decoder.decode(response, metadata.returnType());
      } else {
        throw errorDecoder.decode(metadata.configKey(), response);
      }
    } catch (IOException e) {
      if (logLevel != Logger.Level.NONE) {
        logger.logIOException(metadata.configKey(), logLevel, e, elapsedTime);
      }
      throw errorReading(request, response, e);
    } finally {
      if (shouldClose) {
        ensureClosed(response.body());
      }
    }
  }

 

就是這鬼地方,會執行到:

throw errorDecoder.decode(metadata.configKey(), response);

添加監視日誌:

用來打印全部模塊出現非200~300的錯誤碼

package com.demo.core.server.filter;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.servlet.*;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

/**
 * 過濾特殊狀態碼,打印日誌
 *
 */
public class BadRequestFilter implements Filter {
    protected final Logger logger = LoggerFactory.getLogger(this.getClass());

    private String appName ;
    public BadRequestFilter(String currentAppName){
        appName = currentAppName;
    }

    protected static List<Pattern> patterns = new ArrayList<Pattern>();

    @Override
    public void init(FilterConfig filterConfig) throws ServletException {

    }

    @Override
    public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain chain) throws IOException, ServletException {
        HttpServletRequest httpRequest = (HttpServletRequest) servletRequest;
        HttpServletResponse httpResponse = (HttpServletResponse) servletResponse;
        String url = httpRequest.getRequestURL().toString();
        chain.doFilter(httpRequest, httpResponse);

        if(httpResponse != null && !(httpResponse.getStatus() >= 200 && httpResponse.getStatus() < 300)) {
            // appName|method|url|status
            if(!StringUtils.isEmpty(httpRequest.getQueryString())){
                logger.error("{}|{}|{}|{}", appName, httpRequest.getMethod(),
                        String.format("%s?%s",url, httpRequest.getQueryString()),
                        httpResponse.getStatus());
            } else {
                logger.error("{}|{}|{}|{}", appName, httpRequest.getMethod(), url, httpResponse.getStatus());
            }
        }
    }

    @Override
    public void destroy() {

    }
}

config

package com.demo.srv.user.config;

import com.demo.core.server.filter.BadRequestFilter;
import org.springframework.boot.web.servlet.FilterRegistrationBean;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import javax.servlet.Filter;


@Configuration
public class BadFilterConfig {
    @Bean
    public FilterRegistrationBean someFilterRegistration() {
        FilterRegistrationBean registration = new FilterRegistrationBean();
        registration.setFilter(badRequestFilter());
        registration.addUrlPatterns("/*");
        registration.setName("badRequestFilter");
        registration.setOrder(Integer.MAX_VALUE - 10);
        return registration;
    }

    /**
     * 建立一個bean
     * @return
     */
    @Bean(name = "badRequestFilter")
    public Filter badRequestFilter() {
        return new BadRequestFilter("user");
    }

}

注意, 設置當前模塊爲user模塊,這樣在日誌裏面能夠打印出是哪一個模塊出現了404, 微服務之間調用沒有通過gateway沒地方看請求日誌.

Filter的序號放到比較大.這樣能夠確保都最後執行該Filter.

 

而後使用ab壓力測試,由於是post請求,因此須要在執行命令的目錄下,建立一個pos.json文件.裏面存放改接口須要的數據.

ab -c 100 -n 100000  -p pos.json -T "application/json" http://127.0.0.1:8092/api/user/testapi

-c併發數, -n 全部併發請求總共要調用多少次

 

請求User微服務,

User微服務會異步調用 A-SERVICE和B-SERVICE微服務. 

經過以上日誌分析,  最終調用A,B微服務的地址會錯亂

在請求A微服務的時候,出現必定機率請求到B微服務中,因此出現404

解決方法:

升級spring cloud : Dalston.RC1 不敢升級大版本,等時間充足測試充分再升級大版本

分析依賴,發現openFeign升級到9.4.0

https://github.com/OpenFeign/feign/releases/tag/9.4.0

壓力測試後未發現404的狀況.