elasticsearch學習筆記（三十）——Elasticsearch 相關度評分 TF&IDF算法
Elasticsearch的相關度評分（relevance score）算法採用的是term frequency/inverse document frequency算法，簡稱爲TF/IDF算法。node
算法介紹
relevance score算法，簡單來講就是，就是計算出一個索引中的文本，與搜索文本，它們之間的關聯匹配程度。
TF/IDF算法，分爲兩個部分，IF 和IDF
Term Frequency(TF): 搜索文本中的各個詞條在field文本中出現了多少次，出現的次數越多，就越相關
例如：
搜索請求：hello world
doc1: hello you, and world is very good
doc2: hello, how are you
那麼此時根據TF算法，doc1的相關度要比doc2的要高算法
Inverse Document Frequency(IDF): 搜索文本中的各個詞條在整個索引的全部文檔中出現的次數，出現的次數越多，就越不相關。
搜索請求： hello world
doc1: hello, today is very good.
doc2: hi world, how are you.
好比在index中有1萬條document, hello這個單詞在全部的document中，一共出現了1000次，world這個單詞在全部的document中一共出現100次。那麼根據IDF算法此時doc2的相關度要比doc1要高。code
對於ES還有一個Field-length norm
field-length norm就是field長度越長，相關度就越弱
搜索請求：hello world
doc1: {"title": "hello article", "content": "1萬個單詞"}
doc2: {"title": "my article", "content": "1萬個單詞， hi world"}
此時hello world在整個index中出現的次數是同樣多的。可是根據Field-length norm此時doc1比doc2相關度要高。由於title字段更短。orm
_score是如何被計算出來的
GET /test_index/_search?explain=true
{
"query": {
"match": {
"test_field": "hello"
}
}
}
{
"took" : 9,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 0.20521778,
"hits" : [
{
"_shard" : "[test_index][0]",
"_node" : "P-b-TEvyQOylMyEcMEhApQ",
"_index" : "test_index",
"_type" : "_doc",
"_id" : "2",
"_score" : 0.20521778,
"_source" : {
"test_field" : "hello, how are you"
},
"_explanation" : {
"value" : 0.20521778,
"description" : "weight(test_field:hello in 0) [PerFieldSimilarity], result of:",
"details" : [
{
"value" : 0.20521778,
"description" : "score(freq=1.0), product of:",
"details" : [
{
"value" : 2.2,
"description" : "boost",
"details" : [ ]
},
{
"value" : 0.18232156,
"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
"details" : [
{
"value" : 2,
"description" : "n, number of documents containing term",
"details" : [ ]
},
{
"value" : 2,
"description" : "N, total number of documents with field",
"details" : [ ]
}
]
},
{
"value" : 0.5116279,
"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
"details" : [
{
"value" : 1.0,
"description" : "freq, occurrences of term within document",
"details" : [ ]
},
{
"value" : 1.2,
"description" : "k1, term saturation parameter",
"details" : [ ]
},
{
"value" : 0.75,
"description" : "b, length normalization parameter",
"details" : [ ]
},
{
"value" : 4.0,
"description" : "dl, length of field",
"details" : [ ]
},
{
"value" : 5.5,
"description" : "avgdl, average length of field",
"details" : [ ]
}
]
}
]
}
]
}
},
{
"_shard" : "[test_index][0]",
"_node" : "P-b-TEvyQOylMyEcMEhApQ",
"_index" : "test_index",
"_type" : "_doc",
"_id" : "1",
"_score" : 0.16402164,
"_source" : {
"test_field" : "hello you, and world is very good"
},
"_explanation" : {
"value" : 0.16402164,
"description" : "weight(test_field:hello in 0) [PerFieldSimilarity], result of:",
"details" : [
{
"value" : 0.16402164,
"description" : "score(freq=1.0), product of:",
"details" : [
{
"value" : 2.2,
"description" : "boost",
"details" : [ ]
},
{
"value" : 0.18232156,
"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
"details" : [
{
"value" : 2,
"description" : "n, number of documents containing term",
"details" : [ ]
},
{
"value" : 2,
"description" : "N, total number of documents with field",
"details" : [ ]
}
]
},
{
"value" : 0.40892193,
"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
"details" : [
{
"value" : 1.0,
"description" : "freq, occurrences of term within document",
"details" : [ ]
},
{
"value" : 1.2,
"description" : "k1, term saturation parameter",
"details" : [ ]
},
{
"value" : 0.75,
"description" : "b, length normalization parameter",
"details" : [ ]
},
{
"value" : 7.0,
"description" : "dl, length of field",
"details" : [ ]
},
{
"value" : 5.5,
"description" : "avgdl, average length of field",
"details" : [ ]
}
]
}
]
}
]
}
}
]
}
}
匹配的文檔有兩個，下面直接用一個文檔來分析出ES各個算法的公式。
從上面能夠看出第一個文檔的相關度分數是0.20521778索引
{
"_shard" : "[test_index][0]",
"_node" : "P-b-TEvyQOylMyEcMEhApQ",
"_index" : "test_index",
"_type" : "_doc",
"_id" : "2",
"_score" : 0.20521778,
"_source" : {
"test_field" : "hello, how are you"
},
"_explanation" : {
"value" : 0.20521778,
"description" : "weight(test_field:hello in 0) [PerFieldSimilarity], result of:",
"details" : [
{
"value" : 0.20521778,
"description" : "score(freq=1.0), product of:",
"details" : [
{
"value" : 2.2,
"description" : "boost",
"details" : [ ]
},
{
"value" : 0.18232156,
"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
"details" : [
{
"value" : 2,
"description" : "n, number of documents containing term",
"details" : [ ]
},
{
"value" : 2,
"description" : "N, total number of documents with field",
"details" : [ ]
}
]
},
{
"value" : 0.5116279,
"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
"details" : [
{
"value" : 1.0,
"description" : "freq, occurrences of term within document",
"details" : [ ]
},
{
"value" : 1.2,
"description" : "k1, term saturation parameter",
"details" : [ ]
},
{
"value" : 0.75,
"description" : "b, length normalization parameter",
"details" : [ ]
},
{
"value" : 4.0,
"description" : "dl, length of field",
"details" : [ ]
},
{
"value" : 5.5,
"description" : "avgdl, average length of field",
"details" : [ ]
}
]
}
]
}
]
}
}
經過觀察咱們能夠知道 _score = boost * idf * tf 此時boost = 2.2, idf = 0.18232156, tf = 0.5116279 idf = log(1 + (N - n + 0.5) / (n + 0.5)) 此時n = 2 (n, number of documents containing term), N = 2(N, total number of documents with field) tf = freq / (freq + k1 * (1 - b + b * dl / avgdl)) 此時freq = 1(freq, occurrences of term within document), k1 = 1.2(k1, term saturation parameter), b = 0.75(b, length normalization parameter), d1 = 4 (dl, length of field), avgdl = 5.5(avgdl, average length of field)
相關文章
- 1. ElasticSearch學習筆記-相關度得分記錄
- 2. elasticsearch學習筆記（三十三）——Elasticsearch Bouncing Results問題
- 3. ElasticSearch學習筆記
- 4. elasticsearch學習筆記（三十六）——Elasticsearch 內核原理
- 5. Elasticsearch學習筆記三
- 6. elasticsearch學習筆記（三十二）——Elasticsearch 解密query、fetch phrase原理
- 7. elasticsearch學習筆記（三十五）——Elasticsearch 索引管理
- 8. Elasticsearch學習筆記
- 9. Elasticsearch 學習筆記
- 10. 樂優商城學習筆記十一-Elasticsearch學習（三）
- 更多相關文章...
- • XML 相關技術 - XML 教程
- • 我們已經學習了 SQL，下一步學習什麼呢？ - SQL 教程
- • Tomcat學習筆記（史上最全tomcat學習筆記）
- • NewSQL-TiDB相關
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
最新文章
- 1. 在windows下的虛擬機中,安裝華爲電腦的deepin操作系統
- 2. 強烈推薦款下載不限速解析神器
- 3. 【區塊鏈技術】孫宇晨：區塊鏈技術帶來金融服務的信任變革
- 4. 搜索引起的鏈接分析-計算網頁的重要性
- 5. TiDB x 微衆銀行 | 耗時降低 58%，分佈式架構助力實現普惠金融
- 6. 《數字孿生體技術白皮書》重磅發佈（附完整版下載）
- 7. 雙十一「避坑」指南：區塊鏈電子合同爲電商交易保駕護航！
- 8. 區塊鏈產業，怎樣「鏈」住未來？
- 9. OpenglRipper使用教程
- 10. springcloud請求一次好用一次不好用zuul Name or service not known
歡迎關注本站公眾號,獲取更多信息
相關文章
- 1. ElasticSearch學習筆記-相關度得分記錄
- 2. elasticsearch學習筆記（三十三）——Elasticsearch Bouncing Results問題
- 3. ElasticSearch學習筆記
- 4. elasticsearch學習筆記（三十六）——Elasticsearch 內核原理
- 5. Elasticsearch學習筆記三
- 6. elasticsearch學習筆記（三十二）——Elasticsearch 解密query、fetch phrase原理
- 7. elasticsearch學習筆記（三十五）——Elasticsearch 索引管理
- 8. Elasticsearch學習筆記
- 9. Elasticsearch 學習筆記
- 10. 樂優商城學習筆記十一-Elasticsearch學習（三）