ElasticSearch查詢超時緣由一種及解決

問題

最近一次上線後 線上忽然多了好多ES查詢超時異常

2020-02-01 11:40:43.851 ERROR 2852 --- [o-8022-exec-438] c.c.p.common.dao.SearchDao               : Search person by id failed

java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-895 [ACTIVE]
        at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:789) ~[elasticsearch-rest-client-7.1.1.jar!/:7.1.1]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:225) ~[elasticsearch-rest-client-7.1.1.jar!/:7.1.1]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:212) ~[elasticsearch-rest-client-7.1.1.jar!/:7.1.1]
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1433) ~[elasticsearch-rest-high-level-client-7.1.1.jar!/:7.1.1]
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1403) ~[elasticsearch-rest-high-level-client-7.1.1.jar!/:7.1.1]
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1373) ~[elasticsearch-rest-high-level-client-7.1.1.jar!/:7.1.1]
        at org.elasticsearch.client.RestHighLevelClient.get(RestHighLevelClient.java:699) ~[elasticsearch-rest-high-level-client-7.1.1.jar!/:7.1.1]
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-895 [ACTIVE]
        at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.11.jar!/:4.4.11]

以爲很奇怪 由於就是簡單的query by id 怎麼也會超時呢？

curl 'http://localhost:9201/person/_doc/301024026?pretty'

緣由

當超時異常發生的時候 執行hot_threads查詢 得知原來底層執行了refresh操做

curl 'http://localhost:9201/_nodes/hot_threads?pretty'

101.4% (507.1ms out of 500ms) cpu usage by thread 'elasticsearch[node-1][get][T#1]'
   8/10 snapshots sharing following 32 elements
     app//org.apache.lucene.index.SingletonSortedNumericDocValues.nextDoc(SingletonSortedNumericDocValues.java:53)
     app//org.apache.lucene.codecs.lucene80.Lucene80DocValuesConsumer.writeValues(Lucene80DocValuesConsumer.java:161)
     app//org.apache.lucene.codecs.lucene80.Lucene80DocValuesConsumer.addNumericField(Lucene80DocValuesConsumer.java:111)
     app//org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addNumericField(PerFieldDocValuesFormat.java:109)
     app//org.apache.lucene.index.ReadersAndUpdates.handleDVUpdates(ReadersAndUpdates.java:368)
     app//org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:570)
     app//org.apache.lucene.index.ReaderPool.writeAllDocValuesUpdates(ReaderPool.java:228)
     app//org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3308)
     app//org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:520)
     app//org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:294)
     app//org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269)
     app//org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259)
     app//org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112)
     app//org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140)
     app//org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156)
     app//org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
     app//org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
     app//org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
     app//org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1548)
     app//org.elasticsearch.index.engine.InternalEngine.get(InternalEngine.java:652)
     app//org.elasticsearch.index.shard.IndexShard.get(IndexShard.java:916)
     app//org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:169)
     app//org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:93)
     app//org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:84)
     app//org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:106)
     app//org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:45)

查看官網文檔 得知若是該文檔有更新 且還沒有執行refresh 此時查詢該文檔會顯式執行refresh操做

By default, the get API is realtime, and is not affected by the refresh rate of the index (when data will become visible for search). If a document has been updated but is not yet refreshed, the get API will issue a refresh call in-place to make the document visible. This will also make other documents changed since the last refresh visible. In order to disable realtime GET, one can set the realtimeparameter to false.
https://www.elastic.co/guide/...

這就能解釋的通了 由於進入人物詳情頁面後 會對人物索引的viewCount加一 即文檔有更新 默認30秒後纔會執行refresh操做("refresh_interval" : "30s") 故後續查詢該文檔的時候 會顯式執行refresh操做 致使get by id 超時 尤爲是有大量文檔更新的狀況下

解決

禁止relatime便可

GetRequest getRequest = new GetRequest(personIndex, id.toString());  
getRequest.realtime(false);

參考文檔

https://qbox.io/blog/refresh-...