Elastic search performance testing finding
Setup
1 Node cluster on my local laptop: 8core, Xms=8G, Xmx=8G html
Indexing performance (Single index):
10 million payments, each one about 5KB, with batch size = 10000. Each batch takes roughly 2.5 s → 4 s, total time to index 10 million payment is around 50 minapp
Indexing performance (Multiple indices):
20 separate indices store totally 10 million payments. Indexing execution is slightly faster than single index case. Each batch takes roughly 1.7 s → 3.8 s, total time to index 10 million payment is around 38 minelasticsearch
Parameters required for bulk load operation
Elasticsearch config: http.max_content_length: 500mbide
Client time out adjustment: ui
RestClient.builder(HttpHost("localhost", 9200))
.setRequestConfigCallback {
it.apply {
this.setConnectTimeout(5000)
this.setSocketTimeout(60000)
}
}.setMaxRetryTimeoutMillis(60000))
Initially batch size is set to 100000, elastic search server becomes unstable with high GC frequency, occupying a large percent of CPU time. So larger batch size does not always imply higher performancethis
Query aggregation performance: 3d
Test query: real aggregation query used by rule enginerest
{
"aggregations": {
"date_range": {
"range": {
"field": "createdAt",
"ranges": [
{
"key": "LAST_7_DAYS",
"from": 1544400968485,
"to": 1545005768486
}
],
"keyed": false
},
"aggregations": {
"filter_aggregator": {
"filters": {
"filters": {
"602c7d66-e990-4dfb-b6e2-72b62ff159d5": {
"terms": {
"beneficiaryId.keyword": [
"602c7d66-e990-4dfb-b6e2-72b62ff159d5"
],
"boost": 1
}
},
"67cab0c8-2510-443d-8f00-bce19c04815e": {
"terms": {
"bankAccountUserId.keyword": [
"67cab0c8-2510-443d-8f00-bce19c04815e"
],
"boost": 1
}
},
"8da52e51-eabf-4f6c-b9f0-e222933c1cb7": {
"terms": {
"payerId.keyword": [
"8da52e51-eabf-4f6c-b9f0-e222933c1cb7"
],
"boost": 1
}
},
"8da52e51-eabf-4f6c-b9f0-e222933c1cb7_602c7d66-e990-4dfb-b6e2-72b62ff159d5": {
"bool": {
"filter": [
{
"terms": {
"payerId.keyword": [
"8da52e51-eabf-4f6c-b9f0-e222933c1cb7"
],
"boost": 1
}
},
{
"terms": {
"beneficiaryId.keyword": [
"602c7d66-e990-4dfb-b6e2-72b62ff159d5"
],
"boost": 1
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
},
"9a1b4bad-ccf5-4c67-8718-02696cb351e4": {
"terms": {
"clientId.keyword": [
"9a1b4bad-ccf5-4c67-8718-02696cb351e4"
],
"boost": 1
}
},
"9a1b4bad-ccf5-4c67-8718-02696cb351e4_602c7d66-e990-4dfb-b6e2-72b62ff159d5": {
"bool": {
"filter": [
{
"terms": {
"clientId.keyword": [
"9a1b4bad-ccf5-4c67-8718-02696cb351e4"
],
"boost": 1
}
},
{
"terms": {
"beneficiaryId.keyword": [
"602c7d66-e990-4dfb-b6e2-72b62ff159d5"
],
"boost": 1
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
},
"9a1b4bad-ccf5-4c67-8718-02696cb351e4_8da52e51-eabf-4f6c-b9f0-e222933c1cb7": {
"bool": {
"filter": [
{
"terms": {
"clientId.keyword": [
"9a1b4bad-ccf5-4c67-8718-02696cb351e4"
],
"boost": 1
}
},
{
"terms": {
"payerId.keyword": [
"8da52e51-eabf-4f6c-b9f0-e222933c1cb7"
],
"boost": 1
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
},
"9a1b4bad-ccf5-4c67-8718-02696cb351e4_8da52e51-eabf-4f6c-b9f0-e222933c1cb7_602c7d66-e990-4dfb-b6e2-72b62ff159d5": {
"bool": {
"filter": [
{
"terms": {
"clientId.keyword": [
"9a1b4bad-ccf5-4c67-8718-02696cb351e4"
],
"boost": 1
}
},
{
"terms": {
"payerId.keyword": [
"8da52e51-eabf-4f6c-b9f0-e222933c1cb7"
],
"boost": 1
}
},
{
"terms": {
"beneficiaryId.keyword": [
"602c7d66-e990-4dfb-b6e2-72b62ff159d5"
],
"boost": 1
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
}
},
"other_bucket": false,
"other_bucket_key": "_other_"
},
"aggregations": {
"beneficiary_amount": {
"stats": {
"field": "beneficiaryAmountUsd"
}
},
"payer_amount": {
"stats": {
"field": "payerAmountUsd"
}
},
"distinct_count_beneficiary": {
"cardinality": {
"field": "beneficiaryId.keyword"
}
},
"distinct_count_payer": {
"cardinality": {
"field": "payerId.keyword"
}
},
"distinct_count_client": {
"cardinality": {
"field": "clientId.keyword"
}
},
"distinct_count_bank_acc": {
"cardinality": {
"field": "bankAccountUserId.keyword"
}
},
"distinct_count_bene_country": {
"cardinality": {
"field": "beneficiaryCountry.keyword"
}
},
"distinct_count_payer_country": {
"cardinality": {
"field": "payerCountry.keyword"
}
},
"distinct_count_bene_currency": {
"cardinality": {
"field": "beneficiaryCurrency.keyword"
}
},
"distinct_count_payer_currency": {
"cardinality": {
"field": "payerCurrency.keyword"
}
},
"structured_payment_amount_personal": {
"range": {
"field": "payerAmountUsd",
"ranges": [
{
"from": 9000,
"to": 9999.999
}
],
"keyed": false
}
},
"structured_payment_amount_company": {
"range": {
"field": "payerAmountUsd",
"ranges": [
{
"from": 112500000,
"to": 124999999.999
}
],
"keyed": false
}
}
}
}
}
}
}
}
Test result : (Single Index)code
| Scenario | Number of run | Execution times | Min | Max | Average |
|---|---|---|---|---|---|
| Single thread Search result hit Result size unset | 10 | 6s, 5.9s, 5.9s, 6s, 5.9s, 6.2s, 6.2s, 6.2s, 6.6s, 8.2s | 5.9s | 8.2s | 6.93s |
| Single thread Search result hit Result size = 0 | 10 | 7200ms, 12ms, 19ms, 20ms, 23ms, 22ms, 28ms, 21ms, 19ms, 29ms | 12ms | 7200ms | 19.3ms (First execution takes 7.2s, deviates too much from the rest, so exclude it) |
| Single thread Search result none hit Result size unset | 10 | 510ms, 529ms, 549ms, 512ms, 489ms, 520ms, 506ms, 500ms, 493ms, 499ms | 489ms | 549ms | 510.7ms |
| Single thread Search result none hit Result size = 0 | 10 | 389ms, 1ms, 2ms, 3ms, 3ms, 1ms, 1ms, 1ms, 2ms, 3ms | 389ms | 1ms | 1.89ms (First execution takes 389ms, deviates too much from the rest, so exclude it) |
| 20 threads Search result hit Result size unset | 10 | Exception, listener timeout after waiting for [60000] ms | |||
| 20 threads Search result hit Result size = 0 | 10 | 18ms, 29ms, 61ms, 53ms, 93ms, 20ms, 39ms, 82ms, 17ms, 24ms | 17ms | 82ms | 43.6ms |
| 20 threads Search result none hit Result size unset | 10 | 4.4s, 5.8s, 4.9s, 5.6s 4.6s, 5.4s, 5.3s 4.7s, 5.8s, 4.7s | 5.8s | 4.4s | 5.1s |
| 20 threads Search result none hit Result size = 0 | 10 | 1.4s, 25ms, 6ms, 22ms, 276ms, 250ms, 19ms, 30ms, 11ms, 18ms | 1.4s | 11ms | 73ms (First execution takes 1.4s, deviates too much from the rest, so exclude it) |
Test result: (Multiple Index)orm
| Scenario | Number of run | Execution times | Min | Max | Average |
|---|---|---|---|---|---|
| Single thread Search result hit Result size unset | 10 | 12.6s, 12.7s 12.6s 12.7s, 13.1s, 13.1s, 13.2s, 13,1s, 13.1s, 13.1s | 12.6s | 13.2s | 12.9s |
| Single thread Search result hit Result size = 0 | 10 | 212ms, 147ms, 170ms, 272ms, 223ms, 222ms, 207ms, 255ms, 219ms, 219ms | 147ms | 272ms | 214.6ms |
| Single thread Search result none hit Result size unset | 10 | 1.2s, 1.1s, 1.1s, 1.1s, 1.1s, 1.2s, 1.1s, 1.1s, 1.2s, 1.1s | 1.1s | 1.2s | 1.13s |
| Single thread Search result none hit Result size = 0 | 10 | 909ms, 23ms, 23ms, 35ms, 14ms, 26ms, 19ms, 12ms, 15ms, 22m | 12ms | 909ms | 18.9ms (First execution takes 909ms, deviates too much from the rest, so exclude it) |
| 20 threads Search result hit Result size unset | 10 | Exception, listener timeout after waiting for [60000] ms | |||
| 20 threads Search result hit Result size = 0 | 10 | 18ms, 29ms, 61ms, 53ms, 93ms, 20ms, 39ms, 82ms, 17ms, 24ms | 17ms | 93ms | 43.6ms |
| 20 threads Search result none hit Result size unset | 10 | 7.8s, 7.6s, 7.1s, 6.9s, 7.4s, 8s, 7.3, 7.2s, 7.3s, 7.4s | 6.9s | 8s | 7.4s |
| 20 threads Search result none hit Result size = 0 | 10 | 202ms, 91ms, 100ms, 146ms, 182ms, 131ms, 158ms, 48ms, 171ms, 152ms | 48ms | 202ms | 138.1ms |
Conclusion:
Aggregation performance hinges on the number of documents that matches the aggregation
Result size parameter has significant impact on aggregation performance. not only because it skipped returning hit documents, but also because it enables caching for aggregation result, otherwise, you have to force result caching by explicitly setting request_cache=true
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/shard-request-cache.html
Executing query concurrently can also have negative impact on performance
Increasing number of indices have positive impact on index speed but have large negative impact on aggregation if the aggregation is performed across indices
- 1. Performance Testing
- 2. Understanding performance, load and stress testing
- 3. Notes: harddisk performance testing
- 4. logstash performance testing
- 5. Elastic Search
- 6. elastic search
- 7. 【Elastic Search】入門
- 8. Getting Started with Elastic Search in .NET
- 9. Elastic Search 初探
- 10. elastic search介紹
- 更多相關文章...
- • Docker search 命令 - Docker命令大全
- • PHP tmpfile() 函數 - PHP參考手冊
- • Composer 安裝與使用
- • 算法總結-二分查找法
-
每一个你不满意的现在，都有一个你没有努力的曾经。