【轉載】Elasticsearch 5.x 字段摺疊的使用，廣度搜索
https://elasticsearch.cn/article/132html
備註，分組字段只能是 keyword或num類型，不能是text類型
在 Elasticsearch 5.x 有一個字段摺疊（Field Collapsing， #22337）的功能很是有意思，在這裏分享一下，
字段摺疊是一個頗有歷史的需求了，能夠看這個 issue，編號 #256，最初是2010年7月提的issue，也是討論最多的帖子之一（240+評論），熬了6年才支持的特性，你說牛不牛，哈哈。
目測該特性將於5.3發佈，嚐鮮地址： Elasticsearch-5.3.0-SNAPSHOT，文檔地址： search-request-collapse。
So，什麼是字段摺疊，能夠理解就是按特定字段進行合併去重，好比咱們有一個菜譜搜索，我但願按菜譜的「菜系」字段進行摺疊，即返回結果每一個菜系都返回一個結果，也就是按菜系去重，我搜索關鍵字「魚」，要去返回的結果裏面各類菜系都有，有湘菜，有粵菜，有中餐，有西餐，別全是湘菜，就是這個意思，經過按特定字段摺疊以後，來豐富搜索結果的多樣性。
說到這裏，有人確定會想到，使用 term agg+ top hits agg 來實現啊，這種組合兩種聚和的方式能夠實現上面的功能，不過也有一些侷限性，好比，不能分頁， #4915；結果不夠精確（top term+top hits，es 的聚合實現選擇了犧牲精度來提升速度）；數據量大的狀況下，聚合比較慢，影響搜索體驗。
而新的的字段摺疊的方式是怎麼實現的的呢，有這些要點：
- 摺疊+取 inner_hits 分兩階段執行（組合聚合的方式只有一個階段），因此 top hits 永遠是精確的。
- 字段摺疊只在 top hits 層執行，不須要每次都在完整的結果集上對爲每一個摺疊主鍵計算實際的 doc values 值，只對 top hits 這小部分數據操做就能夠，和 term agg 相比要節省不少內存。
- 由於只在 top hits 上進行摺疊，因此相比組合聚合的方式，速度要快不少。
- 摺疊 top docs 不須要使用全局序列（global ordinals）來轉換 string，相比 agg 這也節省了不少內存。
- 分頁成爲可能，和常規搜索同樣，具備相同的侷限，先獲取 from+size 的內容，再合併。
- search_after 和 scroll 暫未實現，不過具有可行性。
- 摺疊隻影響搜索結果，不影響聚合，搜索結果的 total 是全部的命中紀錄數，去重的結果數未知（沒法計算）。
下面來看看具體的例子，就知道怎麼回事了，使用起來很簡單。
- 先準備索引和數據，這裏以菜譜爲例，name：菜譜名，type 爲菜系，rating 爲用戶的累積平均評分
DELETE recipes
PUT recipes
POST recipes/type/_mapping
{
"properties": {
"name":{
"type": "text"
},
"rating":{
"type": "float"
},"type":{
"type": "keyword"
}
}
}
POST recipes/type/
{
"name":"清蒸魚頭","rating":1,"type":"湘菜"
}
POST recipes/type/
{
"name":"剁椒魚頭","rating":2,"type":"湘菜"
}
POST recipes/type/
{
"name":"紅燒鯽魚","rating":3,"type":"湘菜"
}
POST recipes/type/
{
"name":"鯽魚湯（辣）","rating":3,"type":"湘菜"
}
POST recipes/type/
{
"name":"鯽魚湯（微辣）","rating":4,"type":"湘菜"
}
POST recipes/type/
{
"name":"鯽魚湯（變態辣）","rating":5,"type":"湘菜"
}
POST recipes/type/
{
"name":"廣式鯽魚湯","rating":5,"type":"粵菜"
}
POST recipes/type/
{
"name":"魚香肉絲","rating":2,"type":"川菜"
}
POST recipes/type/
{
"name":"奶油鮑魚湯","rating":2,"type":"西菜"
}
- 如今咱們看看普通的查詢效果是怎麼樣的，搜索關鍵字帶「魚」的菜，返回3條數據
POST recipes/type/_search
{
"query": {"match": {
"name": "魚"
}},"size": 3
} 全是湘菜，個人天，最近上火不想吃辣，這個第一頁的結果對我來講就是垃圾，以下：
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 9,
"max_score": 0.26742277,
"hits": [
{
"_index": "recipes",
"_type": "type",
"_id": "AVoESHYF_OA-dG63Txsd",
"_score": 0.26742277,
"_source": {
"name": "鯽魚湯（變態辣）",
"rating": 5,
"type": "湘菜"
}
},
{
"_index": "recipes",
"_type": "type",
"_id": "AVoESHXO_OA-dG63Txsa",
"_score": 0.19100356,
"_source": {
"name": "紅燒鯽魚",
"rating": 3,
"type": "湘菜"
}
},
{
"_index": "recipes",
"_type": "type",
"_id": "AVoESHWy_OA-dG63TxsZ",
"_score": 0.19100356,
"_source": {
"name": "剁椒魚頭",
"rating": 2,
"type": "湘菜"
}
}
]
}
}咱們再看看，此次我想加個評分排序，你們都喜歡的是那些，看看有沒有喜歡吃的，執行查詢：
POST recipes/type/_search
{
"query": {"match": {
"name": "魚"
}},"sort": [
{
"rating": {
"order": "desc"
}
}
],"size": 3
} 結果稍微好點了，不過3個裏面2個是湘菜，仍是有點不合適，結果以下：
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 9,
"max_score": null,
"hits": [
{
"_index": "recipes",
"_type": "type",
"_id": "AVoESHYF_OA-dG63Txsd",
"_score": null,
"_source": {
"name": "鯽魚湯（變態辣）",
"rating": 5,
"type": "湘菜"
},
"sort": [
5
]
},
{
"_index": "recipes",
"_type": "type",
"_id": "AVoESHYW_OA-dG63Txse",
"_score": null,
"_source": {
"name": "廣式鯽魚湯",
"rating": 5,
"type": "粵菜"
},
"sort": [
5
]
},
{
"_index": "recipes",
"_type": "type",
"_id": "AVoESHX7_OA-dG63Txsc",
"_score": null,
"_source": {
"name": "鯽魚湯（微辣）",
"rating": 4,
"type": "湘菜"
},
"sort": [
4
]
}
]
}
}
如今我知道了，我要看看其餘菜系，這家不是還有西餐、廣東菜等各類菜系的麼，來來，幫我每一個菜系來一個菜看看，換 terms agg 先獲得惟一的 term 的 bucket，再組合 top_hits agg，返回按評分排序的第一個 top hits，有點複雜，不要緊，看下面的查詢就知道了：
GET recipes/type/_search
{
"query": {
"match": {
"name": "魚"
}
},
"sort": [
{
"rating": {
"order": "desc"
}
}
],"aggs": {
"type": {
"terms": {
"field": "type",
"size": 10
},"aggs": {
"rated": {
"top_hits": {
"sort": [{
"rating": {"order": "desc"}
}],
"size": 1
}
}
}
}
},
"size": 0,
"from": 0
} 看下面的結果，雖然 json 結構有點複雜，不過總算是咱們想要的結果了，湘菜、粵菜、川菜、西菜都出來了，每樣一個，不重樣：
{
"took": 4,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 9,
"max_score": 0,
"hits": []
},
"aggregations": {
"type": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "湘菜",
"doc_count": 6,
"rated": {
"hits": {
"total": 6,
"max_score": null,
"hits": [
{
"_index": "recipes",
"_type": "type",
"_id": "AVoESHYF_OA-dG63Txsd",
"_score": null,
"_source": {
"name": "鯽魚湯（變態辣）",
"rating": 5,
"type": "湘菜"
},
"sort": [
5
]
}
]
}
}
},
{
"key": "川菜",
"doc_count": 1,
"rated": {
"hits": {
"total": 1,
"max_score": null,
"hits": [
{
"_index": "recipes",
"_type": "type",
"_id": "AVoESHYr_OA-dG63Txsf",
"_score": null,
"_source": {
"name": "魚香肉絲",
"rating": 2,
"type": "川菜"
},
"sort": [
2
]
}
]
}
}
},
{
"key": "粵菜",
"doc_count": 1,
"rated": {
"hits": {
"total": 1,
"max_score": null,
"hits": [
{
"_index": "recipes",
"_type": "type",
"_id": "AVoESHYW_OA-dG63Txse",
"_score": null,
"_source": {
"name": "廣式鯽魚湯",
"rating": 5,
"type": "粵菜"
},
"sort": [
5
]
}
]
}
}
},
{
"key": "西菜",
"doc_count": 1,
"rated": {
"hits": {
"total": 1,
"max_score": null,
"hits": [
{
"_index": "recipes",
"_type": "type",
"_id": "AVoESHY3_OA-dG63Txsg",
"_score": null,
"_source": {
"name": "奶油鮑魚湯",
"rating": 2,
"type": "西菜"
},
"sort": [
2
]
}
]
}
}
}
]
}
}
}
上面的實現方法，前面已經說了，能夠作，有侷限性，那看看新的字段摺疊法如何作到呢，查詢以下，加一個 collapse 參數，指定對那個字段去重就好了，這裏固然對菜系「type」字段進行去重了：
GET recipes/type/_search
{
"query": {
"match": {
"name": "魚"
}
},
"collapse": {
"field": "type"
},
"size": 3,
"from": 0
}結果很理想嘛，命中結果仍是熟悉的那個味道（和查詢結果長的同樣嘛），以下：
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 9,
"max_score": null,
"hits": [
{
"_index": "recipes",
"_type": "type",
"_id": "AVoDNlRJ_OA-dG63TxpW",
"_score": 0.018980097,
"_source": {
"name": "鯽魚湯（微辣）",
"rating": 4,
"type": "湘菜"
},
"fields": {
"type": [
"湘菜"
]
}
},
{
"_index": "recipes",
"_type": "type",
"_id": "AVoDNlRk_OA-dG63TxpZ",
"_score": 0.013813315,
"_source": {
"name": "魚香肉絲",
"rating": 2,
"type": "川菜"
},
"fields": {
"type": [
"川菜"
]
}
},
{
"_index": "recipes",
"_type": "type",
"_id": "AVoDNlRb_OA-dG63TxpY",
"_score": 0.0125863515,
"_source": {
"name": "廣式鯽魚湯",
"rating": 5,
"type": "粵菜"
},
"fields": {
"type": [
"粵菜"
]
}
}
]
}
}我再試試翻頁，把 from 改一下，如今返回了3條數據，from 改爲3，新的查詢以下：
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 9,
"max_score": null,
"hits": [
{
"_index": "recipes",
"_type": "type",
"_id": "AVoDNlRw_OA-dG63Txpa",
"_score": 0.012546891,
"_source": {
"name": "奶油鮑魚湯",
"rating": 2,
"type": "西菜"
},
"fields": {
"type": [
"西菜"
]
}
}
]
}
}上面的結果只有一條了，去重以後原本就只有4條數據，上面的工做正常，每一個菜系只有一個菜啊，那我不樂意了，幫我每一個菜系裏面多返回幾條，我好選菜啊，加上參數 inner_hits 來控制返回的條數，這裏返回2條，按 rating 也排個序，新的查詢構造以下：
GET recipes/type/_search
{
"query": {
"match": {
"name": "魚"
}
},
"collapse": {
"field": "type",
"inner_hits": {
"name": "top_rated",
"size": 2,
"sort": [
{
"rating": "desc"
}
]
}
},
"sort": [
{
"rating": {
"order": "desc"
}
}
],
"size": 2,
"from": 0
}查詢結果以下，完美：
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 9,
"max_score": null,
"hits": [
{
"_index": "recipes",
"_type": "type",
"_id": "AVoESHYF_OA-dG63Txsd",
"_score": null,
"_source": {
"name": "鯽魚湯（變態辣）",
"rating": 5,
"type": "湘菜"
},
"fields": {
"type": [
"湘菜"
]
},
"sort": [
5
],
"inner_hits": {
"top_rated": {
"hits": {
"total": 6,
"max_score": null,
"hits": [
{
"_index": "recipes",
"_type": "type",
"_id": "AVoESHYF_OA-dG63Txsd",
"_score": null,
"_source": {
"name": "鯽魚湯（變態辣）",
"rating": 5,
"type": "湘菜"
},
"sort": [
5
]
},
{
"_index": "recipes",
"_type": "type",
"_id": "AVoESHX7_OA-dG63Txsc",
"_score": null,
"_source": {
"name": "鯽魚湯（微辣）",
"rating": 4,
"type": "湘菜"
},
"sort": [
4
]
}
]
}
}
}
},
{
"_index": "recipes",
"_type": "type",
"_id": "AVoESHYW_OA-dG63Txse",
"_score": null,
"_source": {
"name": "廣式鯽魚湯",
"rating": 5,
"type": "粵菜"
},
"fields": {
"type": [
"粵菜"
]
},
"sort": [
5
],
"inner_hits": {
"top_rated": {
"hits": {
"total": 1,
"max_score": null,
"hits": [
{
"_index": "recipes",
"_type": "type",
"_id": "AVoESHYW_OA-dG63Txse",
"_score": null,
"_source": {
"name": "廣式鯽魚湯",
"rating": 5,
"type": "粵菜"
},
"sort": [
5
]
}
]
}
}
}
}
]
}
}好了，字段摺疊介紹就到這裏。
[尊重社區原創，轉載請保留或註明出處]
本文地址：http://elasticsearch.cn/article/132
本文地址：http://elasticsearch.cn/article/132
相關文章
- 1. Elasticsearch 5.x 字段摺疊的使用
- 2. Elasticsearch 多字段搜索
- 3. elasticsearch 高亮搜索字段
- 4. 寬度/廣度優先搜索（BFS）的代碼實現（轉載）
- 5. 深度搜索與廣度搜索
- 6. Elasticsearch系列---多字段搜索
- 7. [Elasticsearch] 多字段搜索 (五) - 以字段爲中心的查詢
- 8. Using Field in Searching（使用字段搜索）
- 9. 【轉】MB51搜索字段的設置
- 10. ElasticSearch 2 (15) - 深刻搜索系列之多字段搜索
- 更多相關文章...
- • jQuery Mobile 可摺疊塊 - jQuery Mobile 教程
- • SQL AUTO INCREMENT 字段 - SQL 教程
- • 算法總結-廣度優先算法
- • Composer 安裝與使用
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
最新文章
- 1. 升級Gradle後報錯Gradle‘s dependency cache may be corrupt (this sometimes occurs
- 2. Smarter, Not Harder
- 3. mac-2019-react-native 本地環境搭建(xcode-11.1和android studio3.5.2中Genymotion2.12.1 和VirtualBox-5.2.34 )
- 4. 查看文件中關鍵字前後幾行的內容
- 5. XXE萌新進階全攻略
- 6. Installation failed due to: ‘Connection refused: connect‘安卓studio端口占用
- 7. zabbix5.0通過agent監控winserve12
- 8. IT行業UI前景、潛力如何？
- 9. Mac Swig 3.0.12 安裝
- 10. Windows上FreeRDP-WebConnect是一個開源HTML5代理，它提供對使用RDP的任何Windows服務器和工作站的Web訪問
歡迎關注本站公眾號,獲取更多信息
