原文地址:https://windmt.com/2018/04/17/spring-cloud-6-turbine/java
上一篇咱們介紹了使用 Hystrix Dashboard 來展現 Hystrix 用於熔斷的各項度量指標。經過 Hystrix Dashboard,咱們能夠方便的查看服務實例的綜合狀況,好比:服務調用次數、服務調用延遲等。可是僅經過 Hystrix Dashboard 咱們只能實現對服務當個實例的數據展示,在生產環境咱們的服務是確定須要作高可用的,那麼對於多實例的狀況,咱們就須要將這些度量指標數據進行聚合。下面,在本篇中,咱們就來介紹一下另一個工具:Turbine。react
在開始使用 Turbine 以前,咱們先回顧一下上一篇中實現的架構,以下圖所示:git
其中,咱們構建的內容包括:es6
eureka-consumer-hystrix
服務的 Hystrix 數據下面,咱們將在上述架構基礎上,引入 Turbine 來對服務的 Hystrix 數據進行聚合展現。 這裏咱們將分別介紹兩種聚合方式。github
建立一個標準的 Spring Boot 工程,命名爲:turbine。spring
在 pom.xml 中添加如下依賴express
1 |
<dependency> |
在啓動類上使用@EnableTurbine
註解開啓 Turbinejson
1 |
|
在 application.yml 加入 Eureka 和 Turbine 的相關配置架構
1 |
spring: |
參數說明app
turbine.app-config
參數指定了須要收集監控信息的服務名;turbine.cluster-name-expression
參數指定了集羣名稱爲 default
,當咱們服務數量很是多的時候,能夠啓動多個 Turbine 服務來構建不一樣的聚合集羣,而該參數能夠用來區分這些不一樣的聚合集羣,同時該參數值能夠在 Hystrix 儀表盤中用來定位不一樣的聚合集羣,只須要在 Hystrix Stream 的 URL 中經過 cluster 參數來指定;turbine.combine-host-port
參數設置爲true
,可讓同一主機上的服務經過主機名與端口號的組合來進行區分,默認狀況下會以 host 來區分不一樣的服務,這會使得在本地調試的時候,本機上的不一樣服務聚合成一個服務來統計。注意:new String("default")
這個必定要用 String 來包一下,不然啓動的時候會拋出異常:
1 |
org.springframework.expression.spel.SpelEvaluationException: EL1008E: Property or field 'default' cannot be found on object of type 'com.netflix.appinfo.InstanceInfo' - maybe not public or not valid? |
在完成了上面的內容構建以後,咱們來體驗一下 Turbine 對集羣的監控能力。分別啓動
訪問 Hystrix Dashboard 並開啓對 http://localhost:8080/turbine.stream 的監控,這時候,咱們將看到針對服務 eureka-consumer-hystrix
的聚合監控數據。
Spring Cloud 在封裝 Turbine 的時候,還實現了基於消息代理的收集實現。因此,咱們能夠將全部須要收集的監控信息都輸出到消息代理中,而後 Turbine 服務再從消息代理中異步的獲取這些監控信息,最後將這些監控信息聚合並輸出到 Hystrix Dashboard 中。經過引入消息代理,咱們的 Turbine 和 Hystrix Dashoard 實現的監控架構能夠改爲以下圖所示的結構:
從圖中咱們能夠看到,這裏多了一個重要元素:RabbitMQ。對於 RabbitMQ 的安裝與基本時候咱們能夠查看以前的 MQ 系列,這裏不作過多的說明。下面,咱們能夠來構建一個新的應用來實現基於消息代理的 Turbine 聚合服務。
關於經過 MQ 的聚合,在 Finchley.RC1 版本下有好多坑,好在最後能正常運行了。
UPDATED:2018-06-01
Finchley.RC2 已經出了,下邊提到的 bug 都已經被修復了,直接添加 @EnableTurbineStream
就能夠正常使用了。最新代碼實例請看:https://github.com/zhaoyibo/spring-cloud-study/tree/master/turbine-stream
須要注意一點的是,Turbine Stream 默認的端口已經從 8989 改成 8080 了。
UPDATED:2018-05-18
如下關於 Turbine Stream 的內容僅適用於 Finchley.RC1 版本。今天嘗試一下最新的 Finchley.BUILD-SNAPSHOT 發現 netty 的默認端口已經從 8989 改到 8080,而且須要依賴spring-cloud-starter-netflix-hystrix
,由於目前 BUILD-SNAPSHOT 依舊有 bug 不肯定他們會怎麼改,我就暫時先不更新了。等到 RC2 release 的時候再來更新一發。
建立一個標準的 Spring Boot 工程,命名爲:turbine-stream-rabbitmq
1 |
<dependency> |
1 |
spring: |
1 |
|
以以前的 eureka-consumer-hystrix 項目爲基礎,在 pom.xml 里加入如下依賴
1 |
<dependency> |
再在啓動類上加上@EnableHystrix
註解
1 |
|
分別啓動 eureka-consumer-hystrix、turbine-stream-rabbitmq 這兩個項目,而後在 RabbitMQ 的管理後臺能夠看到,自動建立了一個 Exchange 和 Queue
看到這仍是挺高興的,可是……
當訪問了一下 consumer 中的接口後,就開始了艱辛的爬坑路程……
這個 Turbine Stream 以前應該是叫 Turbine AMQP,當時有個 spring-cloud-starter-turbine-amqp 依賴能夠用,裏邊包裝了相關的依賴,可是如今它被 deprecated 了,讓用 spring-cloud-starter-netflix-turbine-stream 來代替,這就得靠本身來組合了。而坑主要就出在這裏,至於哪些是必須的,哪些是添加了後就出問題的,還有依賴衝突的問題,都得靠本身來搞了。
相關 issue:https://github.com/spring-cloud/spring-cloud-netflix/issues/2858
Turbine Stream 從 RabbitMQ 取數據的時候拋出如下異常:
1 |
com.fasterxml.jackson.core.JsonParseException: Unexpected character (',' (code 44)): Expected space separating root-level values |
實際的 json 串相似如下格式:
1 |
{"origin":{"host":"172.16.106.93","port":9013,"serviceId":"eureka-consumer-hystrix","id":"application-1"},"event":"message","data":{"type":"HystrixCommand","name":"eureka-consumer-hystrix.HelloRemote#hello(String)","group":"eureka-producer","currentTime":1523938311789,"isCircuitBreakerOpen":false,"errorPercentage":0,"errorCount":0,"requestCount":0,"rollingCountCollapsedRequests":0,"rollingCountExceptionsThrown":0,"rollingCountFailure":0,"rollingCountFallbackFailure":0,"rollingCountFallbackRejection":0,"rollingCountFallbackSuccess":0,"rollingCountResponsesFromCache":0,"rollingCountSemaphoreRejected":0,"rollingCountShortCircuited":0,"rollingCountSuccess":0,"rollingCountThreadPoolRejected":0,"rollingCountTimeout":0,"currentConcurrentExecutionCount":1,"latencyExecute_mean":0,"latencyExecute":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"latencyTotal_mean":0,"latencyTotal":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"propertyValue_circuitBreakerRequestVolumeThreshold":20,"propertyValue_circuitBreakerSleepWindowInMilliseconds":5000,"propertyValue_circuitBreakerErrorThresholdPercentage":50,"propertyValue_circuitBreakerForceOpen":false,"propertyValue_circuitBreakerForceClosed":false,"propertyValue_circuitBreakerEnabled":true,"propertyValue_executionIsolationStrategy":"THREAD","propertyValue_executionIsolationThreadTimeoutInMilliseconds":1000,"propertyValue_executionIsolationThreadInterruptOnTimeout":true,"propertyValue_executionIsolationThreadPoolKeyOverride":null,"propertyValue_executionIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"propertyValue_requestCacheEnabled":true,"propertyValue_requestLogEnabled":true,"reportingHosts":1}} |
上邊這串字符串轉爲 byte[] 後就是
1 |
123,34,111,114,105,103,105,110,34,58,123,34,104,111,115,116,34,58,34,49,55,50,46,49,54,46,49,48,54,46,57,51,34,44,34,112,111,114,116,34,58,57,48,49,51,44,34,115,101,114,118,105,99,101,73,100,34,58,34,101,117,114,101,107,97,45,99,111,110,115,117,109,101,114,45,104,121,115,116,114,105,120,34,44,34,105,100,34,58,34,97,112,112,108,105,99,97,116,105,111,110,45,49,34,125,44,34,101,118,101,110,116,34,58,34,109,101,115,115,97,103,101,34,44,34,100,97,116,97,34,58,123,34,116,121,112,101,34,58,34,72,121,115,116,114,105,120,67,111,109,109,97,110,100,34,44,34,110,97,109,101,34,58,34,101,117,114,101,107,97,45,99,111,110,115,117,109,101,114,45,104,121,115,116,114,105,120,46,72,101,108,108,111,82,101,109,111,116,101,35,104,101,108,108,111,40,83,116,114,105,110,103,41,34,44,34,103,114,111,117,112,34,58,34,101,117,114,101,107,97,45,112,114,111,100,117,99,101,114,34,44,34,99,117,114,114,101,110,116,84,105,109,101,34,58,49,53,50,51,57,51,56,56,57,49,55,53,48,44,34,105,115,67,105,114,99,117,105,116,66,114,101,97,107,101,114,79,112,101,110,34,58,102,97,108,115,101,44,34,101,114,114,111,114,80,101,114,99,101,110,116,97,103,101,34,58,48,44,34,101,114,114,111,114,67,111,117,110,116,34,58,48,44,34,114,101,113,117,101,115,116,67,111,117,110,116,34,58,48,44,34,114,111,108,108,105,110,103,67,111,117,110,116,67,111,108,108,97,112,115,101,100,82,101,113,117,101,115,116,115,34,58,48,44,34,114,111,108,108,105,110,103,67,111,117,110,116,69,120,99,101,112,116,105,111,110,115,84,104,114,111,119,110,34,58,48,44,34,114,111,108,108,105,110,103,67,111,117,110,116,70,97,105,108,117,114,101,34,58,48,44,34,114,111,108,108,105,110,103,67,111,117,110,116,70,97,108,108,98,97,99,107,70,97,105,108,117,114,101,34,58,48,44,34,114,111,108,108,105,110,103,67,111,117,110,116,70,97,108,108,98,97,99,107,82,101,106,101,99,116,105,111,110,34,58,48,44,34,114,111,108,108,105,110,103,67,111,117,110,116,70,97,108,108,98,97,99,107,83,117,99,99,101,115,115,34,58,48,44,34,114,111,108,108,105,110,103,67,111,117,110,116,82,101,115,112,111,110,115,101,115,70,114,111,109,67,97,99,104,101,34,58,48,44,34,114,111,108,108,105,110,103,67,111,117,110,116,83,101,109,97,112,104,111,114,101,82,101,106,101,99,116,101,100,34,58,48,44,34,114,111,108,108,105,110,103,67,111,117,110,116,83,104,111,114,116,67,105,114,99,117,105,116,101,100,34,58,48,44,34,114,111,108,108,105,110,103,67,111,117,110,116,83,117,99,99,101,115,115,34,58,48,44,34,114,111,108,108,105,110,103,67,111,117,110,116,84,104,114,101,97,100,80,111,111,108,82,101,106,101,99,116,101,100,34,58,48,44,34,114,111,108,108,105,110,103,67,111,117,110,116,84,105,109,101,111,117,116,34,58,48,44,34,99,117,114,114,101,110,116,67,111,110,99,117,114,114,101,110,116,69,120,101,99,117,116,105,111,110,67,111,117,110,116,34,58,49,44,34,108,97,116,101,110,99,121,69,120,101,99,117,116,101,95,109,101,97,110,34,58,48,44,34,108,97,116,101,110,99,121,69,120,101,99,117,116,101,34,58,123,34,48,34,58,48,44,34,50,53,34,58,48,44,34,53,48,34,58,48,44,34,55,53,34,58,48,44,34,57,48,34,58,48,44,34,57,53,34,58,48,44,34,57,57,34,58,48,44,34,57,57,46,53,34,58,48,44,34,49,48,48,34,58,48,125,44,34,108,97,116,101,110,99,121,84,111,116,97,108,95,109,101,97,110,34,58,48,44,34,108,97,116,101,110,99,121,84,111,116,97,108,34,58,123,34,48,34,58,48,44,34,50,53,34,58,48,44,34,53,48,34,58,48,44,34,55,53,34,58,48,44,34,57,48,34,58,48,44,34,57,53,34,58,48,44,34,57,57,34,58,48,44,34,57,57,46,53,34,58,48,44,34,49,48,48,34,58,48,125,44,34,112,114,111,112,101,114,116,121,86,97,108,117,101,95,99,105,114,99,117,105,116,66,114,101,97,107,101,114,82,101,113,117,101,115,116,86,111,108,117,109,101,84,104,114,101,115,104,111,108,100,34,58,50,48,44,34,112,114,111,112,101,114,116,121,86,97,108,117,101,95,99,105,114,99,117,105,116,66,114,101,97,107,101,114,83,108,101,101,112,87,105,110,100,111,119,73,110,77,105,108,108,105,115,101,99,111,110,100,115,34,58,53,48,48,48,44,34,112,114,111,112,101,114,116,121,86,97,108,117,101,95,99,105,114,99,117,105,116,66,114,101,97,107,101,114,69,114,114,111,114,84,104,114,101,115,104,111,108,100,80,101,114,99,101,110,116,97,103,101,34,58,53,48,44,34,112,114,111,112,101,114,116,121,86,97,108,117,101,95,99,105,114,99,117,105,116,66,114,101,97,107,101,114,70,111,114,99,101,79,112,101,110,34,58,102,97,108,115,101,44,34,112,114,111,112,101,114,116,121,86,97,108,117,101,95,99,105,114,99,117,105,116,66,114,101,97,107,101,114,70,111,114,99,101,67,108,111,115,101,100,34,58,102,97,108,115,101,44,34,112,114,111,112,101,114,116,121,86,97,108,117,101,95,99,105,114,99,117,105,116,66,114,101,97,107,101,114,69,110,97,98,108,101,100,34,58,116,114,117,101,44,34,112,114,111,112,101,114,116,121,86,97,108,117,101,95,101,120,101,99,117,116,105,111,110,73,115,111,108,97,116,105,111,110,83,116,114,97,116,101,103,121,34,58,34,84,72,82,69,65,68,34,44,34,112,114,111,112,101,114,116,121,86,97,108,117,101,95,101,120,101,99,117,116,105,111,110,73,115,111,108,97,116,105,111,110,84,104,114,101,97,100,84,105,109,101,111,117,116,73,110,77,105,108,108,105,115,101,99,111,110,100,115,34,58,49,48,48,48,44,34,112,114,111,112,101,114,116,121,86,97,108,117,101,95,101,120,101,99,117,116,105,111,110,73,115,111,108,97,116,105,111,110,84,104,114,101,97,100,73,110,116,101,114,114,117,112,116,79,110,84,105,109,101,111,117,116,34,58,116,114,117,101,44,34,112,114,111,112,101,114,116,121,86,97,108,117,101,95,101,120,101,99,117,116,105,111,110,73,115,111,108,97,116,105,111,110,84,104,114,101,97,100,80,111,111,108,75,101,121,79,118,101,114,114,105,100,101,34,58,110,117,108,108,44,34,112,114,111,112,101,114,116,121,86,97,108,117,101,95,101,120,101,99,117,116,105,111,110,73,115,111,108,97,116,105,111,110,83,101,109,97,112,104,111,114,101,77,97,120,67,111,110,99,117,114,114,101,110,116,82,101,113,117,101,115,116,115,34,58,49,48,44,34,112,114,111,112,101,114,116,121,86,97,108,117,101,95,102,97,108,108,98,97,99,107,73,115,111,108,97,116,105,111,110,83,101,109,97,112,104,111,114,101,77,97,120,67,111,110,99,117,114,114,101,110,116,82,101,113,117,101,115,116,115,34,58,49,48,44,34,112,114,111,112,101,114,116,121,86,97,108,117,101,95,109,101,116,114,105,99,115,82,111,108,108,105,110,103,83,116,97,116,105,115,116,105,99,97,108,87,105,110,100,111,119,73,110,77,105,108,108,105,115,101,99,111,110,100,115,34,58,49,48,48,48,48,44,34,112,114,111,112,101,114,116,121,86,97,108,117,101,95,114,101,113,117,101,115,116,67,97,99,104,101,69,110,97,98,108,101,100,34,58,116,114,117,101,44,34,112,114,111,112,101,114,116,121,86,97,108,117,101,95,114,101,113,117,101,115,116,76,111,103,69,110,97,98,108,101,100,34,58,116,114,117,101,44,34,114,101,112,111,114,116,105,110,103,72,111,115,116,115,34,58,49,125,125 |
從上邊異常能夠看出,這其實就是將這個 byte[] 轉爲 String 的時候出錯了。
在源碼裏找了很久,最後發現原來是啓動的時候要初始化一個 ConfigurableCompositeMessageConverter
,可是這個類默認的只提供如下 4 個MessageConverter
:
這 4 個 Converter 處理 byte[] -> String 的時候都會出問題(上邊的異常就是MappingJackson2MessageConverter
這個拋出的)。說到這你可能會問了,byte[] 轉 String 有這麼難嗎?不就一個 new String(bytes) 就解決了。我也這麼想啊,就差本身動手寫了。
這時候發現了CompositeMessageConverterFactory
,從名字上能夠看出就是 MessageConverter
的工廠類,彷佛是一根救命稻草。它裏邊默認提供了 7 個 Converter,第一個就是ApplicationJsonMessageMarshallingConverter
,看了裏邊的實現,這不正是我須要的嘛!
1 |
if (message.getPayload() instanceof byte[] && targetClass.isAssignableFrom(String.class)) { |
因此就本身手動注入這個 bean
1 |
|
其實這個 Bean 在
ContentTypeConfiguration
中已經聲明瞭但沒效果,我只是原封不動的 copy 出來。
1 |
|
這個的異常信息以下:
1 |
Exception in thread "RxComputationScheduler-1" java.lang.IllegalStateException: Fatal Exception thrown on Scheduler.Worker thread. |
同時在 Hystrix Dashboard 中 monitor 相應的地址的會提示 「Unable to connect to Command Metric Stream.」 並在 Console 裏報錯:
1 |
EventSource's response has a MIME type ("text/plain") that is not "text/event-stream". Aborting the connection. |
這個問題真不知道怎麼搞了,先棄坑了,有空了再研究吧。
2018-05-06 更新:
這個問題能夠詳見 Github 上的 這個 issus 感謝 @MadeInChina
爲了本文的完整性,我這裏也說明下這個問題的解決思路和解決辦法。
這個異常信息關鍵的一行是是第 22 行
1 |
at org.springframework.cloud.netflix.turbine.stream.TurbineStreamConfiguration.lambda$null$6(TurbineStreamConfiguration.java:106) |
咱們就打開 TurbineStreamConfiguration 看一下這 106 行到底寫了點啥
若是這時你點進去看 ServerSentEvent 這個類,問題其實就明瞭了(我當時就是困在了response.writeAndFlush
這個方法上,根本沒注意到 ServerSentEvent)
這個類直接報錯,由於沒有完整實現 ByteBufHolder 裏邊的方法,從這你已經能看出來應該是相關依賴的問題了。咱們這裏注意一下這個類所在的 JAR 包 io.reactivex:rxnetty:0.4.9
,而後再看一下它實現的這個接口
這個接口的變動記錄能夠看這裏。最後一次的修改是在 2016.5.17 添加了幾個方法,也就是說 ByteBufHolder 在io.netty:netty-buffer:4.1.0.Final
(2016.5.25)就已經被修改了,更不用說io.netty:netty-buffer:4.1.23.Final
(2018.4.4)了。而io.reactivex:rxnetty:0.4.9
是一個 2015.5.6 發佈的 jar 包。
那咱們就用io.reactivex:rxnetty:0.4.20
最新的版原本試試,pom.xml 裏添加如下內容
1 |
<dependency> |
再次啓動並測試
1 |
$ curl http://localhost:8989 |
發現依舊報錯,控制檯裏錯誤以下
1 |
2018-05-06 23:29:37.056 WARN 52477 --- [o-eventloop-3-2] i.n.c.AbstractChannelHandlerContext : An exception '{}' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception: |
就僅看異常棧的第一行,你會發現 UnicastContentSubject 這個類在當前環境下有三個
將三個都打開來看一下,會發現io.reactivex:rxnetty
那兩個看起來問題不大,而com.netflix.rxnetty:rx-netty
這個直接又是報錯,依舊是未徹底實現接口。
根據 DEBUG 的信息來看,而實際使用的就是com.netflix.rxnetty:rx-netty:0.3.18
裏邊的這個,那就分析一下依賴吧,把這個殘疾的老古董(2014.11.6)給排除掉
1 |
$ mvn dependency:tree -Dverbose -Dincludes=*:*netty* |
最終 pom.xml 裏的依賴座標以下(這也是能正常啓動 Turbine Stream 的最小配置了,Spring Cloud 的版本爲 Finchley.RC1):
1 |
<dependency> |
再次測試就能正常收到 SSE 了
1 |
$ curl http://localhost:8989 |
控制檯也不報錯了
1 |
2018-05-06 23:57:25.833 INFO 53309 --- [o-eventloop-3-1] o.s.c.n.t.s.TurbineStreamConfiguration : SSE Request Received |
而後在 Hystrix Dashboard 的地址欄裏輸入 http://localhost:8989 就能看到了(/turbine.stream
可加可不加,若是要修改端口號,在 Spring Boot 的配置文件中修改 turbine.stream.port
)