ELK——Logstash 2.2 date 插件【翻譯+實踐】

時間  2019-11-06
標籤 elk logstash 2.2 date 插件 翻譯+實踐 欄目 日誌分析 简体版
原文   原文鏈接

官網地址css

本文內容

  • 語法
  • 測試數據
  • 可配置選項
  • 參考資料

date 插件是日期插件，這個插件，經常使用而重要。html

若是不用 date 插件，那麼 Logstash 將處理時間做爲時間戳。時間戳字段是 Logstash 本身添加的內置字段 @timestamp，在ES中關於時間的相關查詢，必須使用該字段，你固然也能夠修改該字段的值。java

遷移到：http://www.bdata-cap.com/newsinfo/1712677.html

語法

該插件必須是用 date 包裹，以下所示：jquery

 

date {
}

可用的配置選項以下表所示：git

設置 輸入類型 是否爲必填 默認值
add_field hash No {}
add_tag array No []
locale string No  
match array No []
periodic_flush boolean No false
remove_field array No []
remove_tag array No []
tag_on_failure array No ["_dateparsefailure"]
target string No "@timestamp"
timezone string No  

其中，add_field、remove_field、add_tag、remove_tag 是全部 Logstash 插件都有。它們在插件過濾成功後生效。這四個選項很少說。參見 ELK——Logstash 2.2 mutate 插件。github

測試數據

假設有 Tomcat access 日誌：express

192.168.6.25 - - [24/Apr/2016:01:25:53 +0800] GET "/goLogin" "" 8080 200 1692 23 "http://10.1.8.193:8080/goMain" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0"
192.168.6.25 - - [24/Apr/2016:01:25:53 +0800] GET "/js/common/jquery-1.10.2.min.js" "" 8080 304 - 67 "http://10.1.8.193:8080/goLogin" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0"
192.168.6.25 - - [24/Apr/2016:01:25:53 +0800] GET "/css/common/login.css" "" 8080 304 - 75 "http://10.1.8.193:8080/goLogin" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0"
192.168.6.25 - - [24/Apr/2016:01:25:53 +0800] GET "/js/system/login.js" "" 8080 304 - 53 "http://10.1.8.193:8080/goLogin" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0"

它是按以下 Tomcat 配置產生的：apache

<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log." suffix=".txt"
               pattern="%h %l %u %t %m &quot;%U&quot; &quot;%q&quot; %p %s %b %D &quot;%{Referer}i&quot; &quot;%{User-Agent}i&quot;" />

若用以下 Grok 表達式解析該日誌：api

%{IPORHOST:clientip} %{NOTSPACE:identd} %{NOTSPACE:auth} \[%{HTTPDATE:timestamp}\] %{WORD:http_method} %{NOTSPACE:request} %{NOTSPACE:request_query|-} %{NUMBER:port} %{NUMBER:statusCode} (%{NOTSPACE:bytes}|-) %{NUMBER:reqTime} %{QS:referer} %{QS:userAgent}

會獲得以下結果：數組

{
          "message" => "192.168.6.25 - - [24/Apr/2016:01:25:53 +0800] GET \"/goLogin\" \"\" 8080 200 1692 23 \"http://10.1.8.193:8080/goMain\" \"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0\"",
         "@version" => "1",
       "@timestamp" => "2016-05-17T08:26:07.794Z",
             "host" => "vcyber",
         "clientip" => "192.168.6.25",
           "identd" => "-",
             "auth" => "-",
        "timestamp" => "24/Apr/2016:01:25:53 +0800",
      "http_method" => "GET",
          "request" => "\"/goLogin\"",
    "request_query" => "\"\"",
             "port" => "8080",
       "statusCode" => "200",
            "bytes" => "1692",
          "reqTime" => "23",
          "referer" => "\"http://10.1.8.193:8080/goMain\"",
        "userAgent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0\""
}

注意，簡單起見，日誌拆分到各個字段後的數據類型全是字符串。

可配置選項

match

  • 值是數組 array
  • 默認值爲 []

The date formats allowed are anything allowed by Joda-Time (java time library). You can see the docs for this format here:

joda.time.format.DateTimeFormat

An array with field name first, and format patterns following, [ field, formats... ]

若是你的時間字段可能有多個格式，則可指定多個可能的日期格式：

match => [ "timestamp", "MMM dd YYY HH:mm:ss", "MMM  d YYY HH:mm:ss", "ISO8601" ]

Logstash 支持四種日期格式：

  • ISO8601 - should parse any valid ISO8601 timestamp, such as 2011-04-19T03:44:01.103Z
  • UNIX - will parse float or int value expressing unix time in seconds since epoch like 1326149001.132 as well as 1326149001
  • UNIX_MS - will parse int value expressing unix time in milliseconds since epoch like 1366125117000
  • TAI64N - will parse tai64n time values

例如，若是你有時間字段 timestamp，多是 Aug 13 2010 00:03:44，你應該使用以下配置：

filter {
  date {
    match => [ "logdate", "MMM dd YYYY HH:mm:ss" ]
  }
}

若是字段是嵌套結構，那麼你能夠使用嵌套語法（nested syntax） [foo][bar] 來匹配值。更多信息，參考 the section called 「Field Referencesedit」

periodic_flush

  • 值是 boolean
  • 默認值爲 false

Call the filter flush method at regular interval. Optional.

tag_on_failure

  • 值是 array
  • 默認值爲 ["_dateparsefailure"]

Append values to the tags field when there has been no successful match

target

  • 值是 string
  • 默認值爲 "@timestamp"

把 match 的時間字段保存到指定字段。若爲指定，默認更新到 @timestamp。

示例：

input {
        stdin {
        }
}
filter {
        grok {
                match=>["message","%{IPORHOST:clientip} %{NOTSPACE:identd} %{NOTSPACE:auth} \[%{HTTPDATE:timestamp}\] %{WORD:http_method} %{NOTSPACE:request} %{NOTSPACE:request_query|-} %{NUMBER:port} %{NUMBER:statusCode} (%{NOTSPACE:bytes}|-) %{NUMBER:reqTime} %{QS:referer} %{QS:userAgent}"]
        }
        date {
                match=>["timestamp","dd/MMM/yyyy:HH:mm:ss Z"]
                target=>"@timestamp"
        }
}
output{
        stdout{
                codec=>rubydebug
        }
}

獲得以下結果：

{
          "message" => "}192.168.6.25 - - [24/Apr/2016:01:25:53 +0800] GET \"/goLogin\" \"\" 8080 200 1692 23 \"http://10.1.8.193:8080/goMain\" \"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0\"",
         "@version" => "1",
       "@timestamp" => "2016-04-23T17:25:53.000Z",
             "host" => "vcyber",
         "clientip" => "192.168.6.25",
           "identd" => "-",
             "auth" => "-",
        "timestamp" => "24/Apr/2016:01:25:53 +0800",
      "http_method" => "GET",
          "request" => "\"/goLogin\"",
    "request_query" => "\"\"",
             "port" => "8080",
       "statusCode" => "200",
            "bytes" => "1692",
          "reqTime" => "23",
          "referer" => "\"http://10.1.8.193:8080/goMain\"",
        "userAgent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0\""
}

timezone

Specify a time zone canonical ID to be used for date parsing. The valid IDs are listed on the Joda.org available time zones page. This is useful in case the time zone cannot be extracted from the value, and is not the platform default. If this is not specified the platform default will be used. Canonical ID is good as it takes care of daylight saving time for you For example, America/Los_Angeles or Europe/Paris are valid IDs. This field can be dynamic and include parts of the event using the %{field} syntax

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程