Logstash處理json格式日誌文件

時間 2019-12-11

原文原文鏈接

假設日誌文件中的每一行記錄格式爲json的，如：java

{"Method":"JSAPI.JSTicket","Message":"JSTicket:kgt8ON7yVITDhtdwci0qeZg4L-Dj1O5WF42Nog47n_0aGF4WPJDIF2UA9MeS8GzLe6MPjyp2WlzvsL0nlvkohw","CreateTime":"2015/10/13 9:39:59","AppGUID":"cb54ba2d-1d38-45f2-9ed1-abff0bf7dd3d","_PartitionKey":"cb54ba2d-1d38-45f2-9ed1-abff0bf7dd3d","_RowKey":"1444700398710_ad4d33ce-a9d9-4d11-932e-e2ccebdb726c","_UnixTS":1444700398710}

默認配置下，logstash處理插入進elasticsearch後，查到的結果是這樣的：mysql

{
    "_index": "logstash-2015.10.16", "_type": "voip_feedback", "_id": "sheE9eXiQASMDVtRJ0EYcg", "_version": 1, "found": true, "_source": { "message": "{\"Method\":\"JSAPI.JSTicket\",\"Message\":\"JSTicket:kgt8ON7yVITDhtdwci0qeZg4L-Dj1O5WF42Nog47n_0aGF4WPJDIF2UA9MeS8GzLe6MPjyp2WlzvsL0nlvkohw\",\"CreateTime\":\"2015/10/13 9:39:59\",\"AppGUID\":\"cb54ba2d-1d38-45f2-9ed1-abff0bf7dd3d\",\"_PartitionKey\":\"cb54ba2d-1d38-45f2-9ed1-abff0bf7dd3d\",\"_RowKey\":\"1444700398710_ad4d33ce-a9d9-4d11-932e-e2ccebdb726c\",\"_UnixTS\":1444700398710}", "@version": "1", "@timestamp": "2015-10-16T00:39:51.252Z", "type": "voip_feedback", "host": "ipphone", "path": "/usr1/data/voip_feedback.txt" } }

即會將json記錄作爲一個字符串放到」message」下，可是我是想讓logstash自動解析json記錄，將各字段放入elasticsearch中。有三種配置方式能夠實現。sql

第一種，直接設置format => json

file {
        type => "voip_feedback" path => ["/usr1/data/voip_feedback.txt"] format => json sincedb_path => "/home/jfy/soft/logstash-1.4.2/voip_feedback.access" }

這種方式查詢出的結果是：json

{
    "_index": "logstash-2015.10.16", "_type": "voip_feedback", "_id": "NrNX8HrxSzCvLl4ilKeyCQ", "_version": 1, "found": true, "_source": { "Method": "JSAPI.JSTicket", "Message": "JSTicket:kgt8ON7yVITDhtdwci0qeZg4L-Dj1O5WF42Nog47n_0aGF4WPJDIF2UA9MeS8GzLe6MPjyp2WlzvsL0nlvkohw", "CreateTime": "2015/10/13 9:39:59", "AppGUID": "cb54ba2d-1d38-45f2-9ed1-abff0bf7dd3d", "_PartitionKey": "cb54ba2d-1d38-45f2-9ed1-abff0bf7dd3d", "_RowKey": "1444700398710_ad4d33ce-a9d9-4d11-932e-e2ccebdb726c", "_UnixTS": 1444700398710, "@version": "1", "@timestamp": "2015-10-16T00:16:11.455Z", "type": "voip_feedback", "host": "ipphone", "path": "/usr1/data/voip_feedback.txt" } }

能夠看到，json記錄已經被直接解析成各字段放入到了_source中，可是原始記錄內容沒有被保存數組

第二種，使用codec => json

file {
        type => "voip_feedback" path => ["/usr1/data/voip_feedback.txt"] sincedb_path => "/home/jfy/soft/logstash-1.4.2/voip_feedback.access" codec => json { charset => "UTF-8" } }

這種方式查詢出的結果與第一種同樣，字段被解析，原始記錄內容也沒有保存markdown

第三種，使用filter json

filter {
    if [type] == "voip_feedback" { json { source => "message" #target => "doc" #remove_field => ["message"] } } }

這種方式查詢出的結果是這樣的：app

{
    "_index": "logstash-2015.10.16", "_type": "voip_feedback", "_id": "CUtesLCETAqhX73NKXZfug", "_version": 1, "found": true, "_source": { "message": "{\"Method222\":\"JSAPI.JSTicket\",\"Message\":\"JSTicket:kgt8ON7yVITDhtdwci0qeZg4L-Dj1O5WF42Nog47n_0aGF4WPJDIF2UA9MeS8GzLe6MPjyp2WlzvsL0nlvkohw\",\"CreateTime\":\"2015/10/13 9:39:59\",\"AppGUID\":\"cb54ba2d-1d38-45f2-9ed1-abff0bf7dd3d\",\"_PartitionKey\":\"cb54ba2d-1d38-45f2-9ed1-abff0bf7dd3d\",\"_RowKey\":\"1444700398710_ad4d33ce-a9d9-4d11-932e-e2ccebdb726c\",\"_UnixTS\":1444700398710}", "@version": "1", "@timestamp": "2015-10-16T00:28:20.018Z", "type": "voip_feedback", "host": "ipphone", "path": "/usr1/data/voip_feedback.txt", "Method222": "JSAPI.JSTicket", "Message": "JSTicket:kgt8ON7yVITDhtdwci0qeZg4L-Dj1O5WF42Nog47n_0aGF4WPJDIF2UA9MeS8GzLe6MPjyp2WlzvsL0nlvkohw", "CreateTime": "2015/10/13 9:39:59", "AppGUID": "cb54ba2d-1d38-45f2-9ed1-abff0bf7dd3d", "_PartitionKey": "cb54ba2d-1d38-45f2-9ed1-abff0bf7dd3d", "_RowKey": "1444700398710_ad4d33ce-a9d9-4d11-932e-e2ccebdb726c", "_UnixTS": 1444700398710, "tags": [ "111", "222" ] } }

能夠看到，原始記錄被保存，同時字段也被解析保存。若是確認不須要保存原始記錄內容，能夠加設置：remove_field => [「message」]elasticsearch

比較以上三種方法，最方便直接的就是在file中設置format => jsonide

另外須要注意的是，logstash會在向es插入數據時默認會在_source下增長type,host,path三個字段，若是json內容中自己也含有type,host,path字段，那麼解析後將覆蓋掉logstash默認的這三個字段，尤爲是type字段，這個同時也是作爲index/type用的，覆蓋掉後，插入進es中的index/type就是json數據記錄中的內容，將再也不是logstash config中配置的type值。post

這時須要設置filter.json.target，設置該字段後json原始內容將不會放在_source下，而是放到設置的」doc」下：

{
    "_index": "logstash-2015.10.20", "_type": "3alogic_log", "_id": "xfj3ngd5S3iH2YABjyU6EA", "_version": 1, "found": true, "_source": { "@version": "1", "@timestamp": "2015-10-20T11:36:24.503Z", "type": "3alogic_log", "host": "server114", "path": "/usr1/app/log/mysql_3alogic_log.log", "doc": { "id": 633796, "identity": "13413602120", "type": "EAP_TYPE_PEAP", "apmac": "88-25-93-4E-1F-96", "usermac": "00-65-E0-31-62-5D", "time": "20151020-193624", "apmaccompany": "TP-LINK TECHNOLOGIES CO.,LTD", "usermaccompany": "" } } }

這樣就不會覆蓋掉_source下的type,host,path值
並且在kibana中顯示時字段名稱爲doc.type,doc.id…

json中嵌套json：

上傳的json：

{
    "indexName": "tv_app_default",
    "baseInfo": {
      "deviceId": "458ec202-e02e-4b82-a7ca-18e5cb4e3df1",
      "deviceModel": "PRO 7-H",
      "deviceSubModel": "",
      "devicePlatform": "Android",
      "appName": "IMetis",
      "appVersion": "1.0",
      "networkStatus": "wifi鏈接",
      "systemVersion": "24"
    },
    "event": {
      "id": "cd478c7b167a7a1030deaeb40036b0f9",
      "name": "TestFragment.java_com.italkbb.test.TestFragment_Bundle[{name=test1}]",
      "event_index": "tv_app_default",
      "event_level": "verbose",
      "timestamp": "2019-03-20T15:15:15.029+08:00",
      "duration": "13043968231158",
      "instant": "0",
      "line": -1
    }
  }

filter裏面這麼寫，會把event和baseInfo裏面的json字串解析出來。

filter {
   json {
      source => "message"
      remove_field => ["message"]
    }

    mutate {
      add_field => { "eventtmp" => "%{event}" }
    }
    json {
      source => "eventtmp"
      remove_field => ["eventtmp"]
    }

    mutate {
      add_field => { "baseInfotmp" => "%{baseInfo}" }
    }
    json {
      source => "baseInfotmp"
      remove_field => ["baseInfotmp" , "baseInfo"]   #只能加一個數組刪除 remove_field => ["eventtmp"] 這句裏面也加上就沒法插入es了。
#其實沒問題。是因爲kibana按timestamp排序了。時間轉換了不對。直接用search語句能夠搜索到。

  }

結果以下：

{
  "_index": "tv_app_default-2019.05.08",
  "_type": "doc",
  "_id": "k1aVlmoBZV0IMWE2odMF",
  "_version": 1,
  "_score": null,
  "_source": {
    "timestamp": "2019-05-08T14:10:47.340",
    "deviceId": "458ec202-e02e-4b82-a7ca-18e5cb4e3df1",
    "networkStatus": "wifi鏈接",
    "appName": "IMetis",
    "duration": "59960912157",
    "indexName": "tv_app_default",
    "id": "f353b68c07b661f2fdd42e2260e061d9",
    "deviceSubModel": "",
    "@timestamp": "2019-05-08T15:44:14.968Z",
    "systemVersion": "24",
    "deviceModel": "PRO 7-H",
    "event_level": "verbose",
    "appVersion": "1.0",
    "devicePlatform": "Android",
    "instant": "0",
    "@version": "1",
    "event_index": "tv_app_default",
    "event": {
      "instant": "0",
      "event_index": "tv_app_default",
      "name": "BaseActivity.java_com.italkbb.test.Main2Activity",
      "line": -1,
      "timestamp": "2019-05-08T14:10:47.340",
      "id": "f353b68c07b661f2fdd42e2260e061d9",
      "event_level": "verbose",
      "duration": "59960912157"
    },
    "line": -1,
    "type": "tv_app_log",
    "name": "BaseActivity.java_com.italkbb.test.Main2Activity"
  },
  "fields": {
    "event.timestamp": [
      "2019-05-08T14:10:47.340Z"
    ]
  },
  "sort": [
    1557324647340
  ]
}