elasticsearch學習筆記(三十)——Elasticsearch 相關度評分 TF&IDF算法

Elasticsearch的相關度評分(relevance score)算法採用的是term frequency/inverse document frequency算法,簡稱爲TF/IDF算法。node

算法介紹

relevance score算法,簡單來講就是,就是計算出一個索引中的文本,與搜索文本,它們之間的關聯匹配程度。

TF/IDF算法,分爲兩個部分,IF 和IDF

Term Frequency(TF): 搜索文本中的各個詞條在field文本中出現了多少次,出現的次數越多,就越相關

例如:
搜索請求:hello world
doc1: hello you, and world is very good
doc2: hello, how are you
那麼此時根據TF算法,doc1的相關度要比doc2的要高算法

Inverse Document Frequency(IDF): 搜索文本中的各個詞條在整個索引的全部文檔中出現的次數,出現的次數越多,就越不相關。

搜索請求: hello world
doc1: hello, today is very good.
doc2: hi world, how are you.
好比在index中有1萬條document, hello這個單詞在全部的document中,一共出現了1000次,world這個單詞在全部的document中一共出現100次。那麼根據IDF算法此時doc2的相關度要比doc1要高。code

對於ES還有一個Field-length norm

field-length norm就是field長度越長,相關度就越弱
搜索請求:hello world
doc1: {"title": "hello article", "content": "1萬個單詞"}
doc2: {"title": "my article", "content": "1萬個單詞, hi world"}
此時hello world在整個index中出現的次數是同樣多的。可是根據Field-length norm此時doc1比doc2相關度要高。由於title字段更短。orm

_score是如何被計算出來的

GET /test_index/_search?explain=true
{
  "query": {
    "match": {
      "test_field": "hello"
    }
  }
}

{
  "took" : 9,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 0.20521778,
    "hits" : [
      {
        "_shard" : "[test_index][0]",
        "_node" : "P-b-TEvyQOylMyEcMEhApQ",
        "_index" : "test_index",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : 0.20521778,
        "_source" : {
          "test_field" : "hello, how are you"
        },
        "_explanation" : {
          "value" : 0.20521778,
          "description" : "weight(test_field:hello in 0) [PerFieldSimilarity], result of:",
          "details" : [
            {
              "value" : 0.20521778,
              "description" : "score(freq=1.0), product of:",
              "details" : [
                {
                  "value" : 2.2,
                  "description" : "boost",
                  "details" : [ ]
                },
                {
                  "value" : 0.18232156,
                  "description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
                  "details" : [
                    {
                      "value" : 2,
                      "description" : "n, number of documents containing term",
                      "details" : [ ]
                    },
                    {
                      "value" : 2,
                      "description" : "N, total number of documents with field",
                      "details" : [ ]
                    }
                  ]
                },
                {
                  "value" : 0.5116279,
                  "description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
                  "details" : [
                    {
                      "value" : 1.0,
                      "description" : "freq, occurrences of term within document",
                      "details" : [ ]
                    },
                    {
                      "value" : 1.2,
                      "description" : "k1, term saturation parameter",
                      "details" : [ ]
                    },
                    {
                      "value" : 0.75,
                      "description" : "b, length normalization parameter",
                      "details" : [ ]
                    },
                    {
                      "value" : 4.0,
                      "description" : "dl, length of field",
                      "details" : [ ]
                    },
                    {
                      "value" : 5.5,
                      "description" : "avgdl, average length of field",
                      "details" : [ ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      },
      {
        "_shard" : "[test_index][0]",
        "_node" : "P-b-TEvyQOylMyEcMEhApQ",
        "_index" : "test_index",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 0.16402164,
        "_source" : {
          "test_field" : "hello you, and world is very good"
        },
        "_explanation" : {
          "value" : 0.16402164,
          "description" : "weight(test_field:hello in 0) [PerFieldSimilarity], result of:",
          "details" : [
            {
              "value" : 0.16402164,
              "description" : "score(freq=1.0), product of:",
              "details" : [
                {
                  "value" : 2.2,
                  "description" : "boost",
                  "details" : [ ]
                },
                {
                  "value" : 0.18232156,
                  "description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
                  "details" : [
                    {
                      "value" : 2,
                      "description" : "n, number of documents containing term",
                      "details" : [ ]
                    },
                    {
                      "value" : 2,
                      "description" : "N, total number of documents with field",
                      "details" : [ ]
                    }
                  ]
                },
                {
                  "value" : 0.40892193,
                  "description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
                  "details" : [
                    {
                      "value" : 1.0,
                      "description" : "freq, occurrences of term within document",
                      "details" : [ ]
                    },
                    {
                      "value" : 1.2,
                      "description" : "k1, term saturation parameter",
                      "details" : [ ]
                    },
                    {
                      "value" : 0.75,
                      "description" : "b, length normalization parameter",
                      "details" : [ ]
                    },
                    {
                      "value" : 7.0,
                      "description" : "dl, length of field",
                      "details" : [ ]
                    },
                    {
                      "value" : 5.5,
                      "description" : "avgdl, average length of field",
                      "details" : [ ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
    ]
  }
}

匹配的文檔有兩個,下面直接用一個文檔來分析出ES各個算法的公式。
從上面能夠看出第一個文檔的相關度分數是0.20521778索引

{
        "_shard" : "[test_index][0]",
        "_node" : "P-b-TEvyQOylMyEcMEhApQ",
        "_index" : "test_index",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : 0.20521778,
        "_source" : {
          "test_field" : "hello, how are you"
        },
        "_explanation" : {
          "value" : 0.20521778,
          "description" : "weight(test_field:hello in 0) [PerFieldSimilarity], result of:",
          "details" : [
            {
              "value" : 0.20521778,
              "description" : "score(freq=1.0), product of:",
              "details" : [
                {
                  "value" : 2.2,
                  "description" : "boost",
                  "details" : [ ]
                },
                {
                  "value" : 0.18232156,
                  "description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
                  "details" : [
                    {
                      "value" : 2,
                      "description" : "n, number of documents containing term",
                      "details" : [ ]
                    },
                    {
                      "value" : 2,
                      "description" : "N, total number of documents with field",
                      "details" : [ ]
                    }
                  ]
                },
                {
                  "value" : 0.5116279,
                  "description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
                  "details" : [
                    {
                      "value" : 1.0,
                      "description" : "freq, occurrences of term within document",
                      "details" : [ ]
                    },
                    {
                      "value" : 1.2,
                      "description" : "k1, term saturation parameter",
                      "details" : [ ]
                    },
                    {
                      "value" : 0.75,
                      "description" : "b, length normalization parameter",
                      "details" : [ ]
                    },
                    {
                      "value" : 4.0,
                      "description" : "dl, length of field",
                      "details" : [ ]
                    },
                    {
                      "value" : 5.5,
                      "description" : "avgdl, average length of field",
                      "details" : [ ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
經過觀察咱們能夠知道
_score = boost * idf * tf 
此時boost = 2.2, idf = 0.18232156, tf = 0.5116279

idf = log(1 + (N - n + 0.5) / (n + 0.5))
此時n = 2 (n, number of documents containing term), N = 2(N, total number of documents with field)

tf = freq / (freq + k1 * (1 - b + b * dl / avgdl))
此時freq = 1(freq, occurrences of term within document), k1 = 1.2(k1, term saturation parameter), b = 0.75(b, length normalization parameter), d1 = 4 (dl, length of field), avgdl = 5.5(avgdl, average length of field)
相關文章
相關標籤/搜索