Spring Cloud Config Server遷移節點或容器化帶來的問題
若是您跟我同樣，目前正在使用Spring Cloud Config作爲配置中心的話，本篇未來要描述的問題，強烈推薦瞭解和關注！由於這個問題目前存在於全部的版本中，尚未徹底修復。java
原文連接：http://blog.didispace.com/Spring-Cloud-Config-Server-ip-change-problem/git
問題現象
爲了說明下面的內容，咱們能夠先嚐試重現一下問題：在一個測試環境中，將Spring Cloud Config的配置中心遷移到另一個節點上，即配置中心的IP地址發生了變化。在完成遷移以後，咱們會發現該環境下各個微服務應用的健康狀態會變得時好時壞，而且在日誌中會出現相似下面的報錯：github
2018-05-13 17:01:28,569 WARN [http-nio-9920-exec-1] org.springframework.cloud.config.client.ConfigServerHealthIndicator - Health check failed
java.lang.IllegalStateException: Could not locate PropertySource and the fail fast property is set, failing
at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locate(ConfigServicePropertySourceLocator.java:132)
at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator$$FastClassBySpringCGLIB$$fa44b2a.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.retry.interceptor.RetryOperationsInterceptor$1.doWithRetry(RetryOperationsInterceptor.java:91)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:164)
at org.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:118)
at org.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:153)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator$$EnhancerBySpringCGLIB$$3a43a1f4.locate(<generated>)
at org.springframework.cloud.config.client.ConfigServerHealthIndicator.getPropertySource(ConfigServerHealthIndicator.java:54)
at org.springframework.cloud.config.client.ConfigServerHealthIndicator.doHealthCheck(ConfigServerHealthIndicator.java:35)
at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:43)
at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68)
at org.springframework.boot.actuate.endpoint.HealthEndpoint.invoke(HealthEndpoint.java:85)
at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.getCurrentHealth(HealthMvcEndpoint.java:177)
at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.getHealth(HealthMvcEndpoint.java:166)
at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke(HealthMvcEndpoint.java:143)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.yonghui.feign.filter.RequestOriginFilter.doFilter(RequestOriginFilter.java:41)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at com.yonghui.rpc.feature.web.boot.RpcHolder4BootFilter.doFilterInternal(RpcHolder4BootFilter.java:29)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at com.yonghui.rpc.feature.web.boot.FeatureSupport4BootFilter.doFilterInternal(FeatureSupport4BootFilter.java:24)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://192.168.5.103:9010/config-server/test": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:674)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:621)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:539)
at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.getRemoteEnvironment(ConfigServicePropertySourceLocator.java:172)
at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locate(ConfigServicePropertySourceLocator.java:93)
... 95 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:78)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:660)
... 99 more
複製代碼
能夠看到相似上面的健康檢查失敗錯誤，可是並非一直這樣，這個環境下的微服務會出現時好時壞的狀況，那麼爲何會出現這種現象呢？web
緣由分析
從錯誤日誌中咱們能夠發現一個很是關鍵的信息：I/O error on GET request for "http://192.168.5.103:9010/config-server/test"。spring
報錯說明了微服務檢查配置中心獲取配置的鏈接是否暢通的時候出現了鏈接不上的狀況，可是這個連接信息其實並非當前配置中心的地址，而是咱們遷移以前的配置中心的地址。apache
從健康檢查的實現源碼ConfigServerHealthIndicator中爲入口去分析和調試，咱們能夠解答上面現象的兩個疑問：tomcat
@Override
protected void doHealthCheck(Builder builder) throws Exception {
PropertySource<?> propertySource = getPropertySource();
builder.up();
if (propertySource instanceof CompositePropertySource) {
List<String> sources = new ArrayList<>();
for (PropertySource<?> ps : ((CompositePropertySource) propertySource).getPropertySources()) {
sources.add(ps.getName());
}
builder.withDetail("propertySources", sources);
} else if (propertySource!=null) {
builder.withDetail("propertySources", propertySource.toString());
} else {
builder.unknown().withDetail("error", "no property sources located");
}
}
private PropertySource<?> getPropertySource() {
long accessTime = System.currentTimeMillis();
if (isCacheStale(accessTime)) {
this.lastAccess = accessTime;
this.cached = locator.locate(this.environment);
}
return this.cached;
}
複製代碼
爲何會健康檢查訪問的仍是老的配置中心地址？
真正致使健康檢查失敗的語句是getPropertySource中的 this.cached = locator.locate(this.environment);而這裏的具體實如今org.springframework.cloud.config.client.ConfigServicePropertySourceLocator類中，具體實現以下：websocket
@Override
@Retryable(interceptor = "configServerRetryInterceptor")
public org.springframework.core.env.PropertySource<?> locate(
org.springframework.core.env.Environment environment) {
ConfigClientProperties properties = this.defaultProperties.override(environment);
CompositePropertySource composite = new CompositePropertySource("configService");
RestTemplate restTemplate = this.restTemplate == null ? getSecureRestTemplate(properties)
: this.restTemplate;
Exception error = null;
String errorBody = null;
logger.info("Fetching config from server at: " + properties.getRawUri());
...
}
複製代碼
能夠看到，真正去訪問的地址是直接從properties.getRawUri()獲取的，它已是一個固化的值，而不是經過服務發現機制來動態獲取的。這就致使了當咱們把配置中心作了遷移，或者直接部署在容器中出現重啓的時候，IP發生變化，而全部的微服務還覺得訪問的是原來的配置中心地址，就會出現健康檢查失敗的問題，致使服務不可用的現象。mvc
爲何健康檢查時好時壞？
上面的問題會致使健康檢查失敗，可是這個服務並非一直都很差，而是間斷性的出現不健康。這主要仍是健康檢查時間中的機制致使，這裏能夠具體看ConfigServerHealthIndicator的getPropertySource函數，該方法執行的時候中間並非每一次檢查都會去訪問配置中心（執行locator.locate(this.environment)方法），所以客戶端的健康檢查並不會每次都健康檢查失敗，從而出現了微服務健康檢查時好時壞的狀況。app
如何解決
該問題目前也在官方的issue中被提出，還處於open狀態，具體可見：https://github.com/spring-cloud/spring-cloud-config/issues/514
因爲該問題目前並無獲得解決，雖然提交了一個PR，可是還有待完善以及提供一些測試，本來想徹底處理好以後再寫一篇文章，可是發現最近很多問過相似問題，因此索性先寫一篇文章，提醒一下用戶以及給一些相關的建議。
當前版本上不太容易經過擴展的方式去解決的這個問題，因此你們能夠變通的去避免這個問題：
- 部署在虛擬機上而不是容器上，避免IP的變更
- 能夠考慮關閉微服務隊config客戶端的健康檢查，增長參數
management.health.config.enabled=false；可是這個操做有一個弊端，雖然遷移不會引起服務時好時壞的問題了，可是若是有動態配置刷新需求，若是遷移了配置中心，刷新配置操做也是會失敗的。 - 補充一個網友給的思路，經過內部域名來綁定配置中心，而不是ip地址
相關文章
- 1. Spring Cloud Config Server 節點遷移引發的問題，請格外注意這一點！
- 2. Spring Cloud Config
- 3. spring cloud:spring cloud config
- 4. spring cloud: 使用consul來替換config server
- 5. spring-cloud-config安全問題
- 6. Spring cloud - Spring Cloud Config
- 7. Spring Cloud Config——Github
- 8. spring cloud config
- 9. spring cloud Eureka server 問題 Spring Cloud java.lang.TypeNotPresentException
- 10. 【八】Spring Cloud Config
- 更多相關文章...
- • XML DOM - 訪問節點 - XML DOM 教程
- • Spring IoC容器：BeanFactory和ApplicationContext - Spring教程
- • Docker容器實戰(一) - 封神Server端技術
- • Docker容器實戰(七) - 容器眼光下的文件系統
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
歡迎關注本站公眾號,獲取更多信息
相關文章
- 1. Spring Cloud Config Server 節點遷移引發的問題，請格外注意這一點！
- 2. Spring Cloud Config
- 3. spring cloud:spring cloud config
- 4. spring cloud: 使用consul來替換config server
- 5. spring-cloud-config安全問題
- 6. Spring cloud - Spring Cloud Config
- 7. Spring Cloud Config——Github
- 8. spring cloud config
- 9. spring cloud Eureka server 問題 Spring Cloud java.lang.TypeNotPresentException
- 10. 【八】Spring Cloud Config