logstash 各類時間轉換

<pre name="code" class="html">日期格式轉換:

/***** nginx 訪問日誌
[elk@zjtest7-frontend config]$ cat stdin02.conf 
input {
    stdin {
    }
}
filter {
    grok {
        match => ["message", "%{IPORHOST:clientip} \[%{HTTPDATE:time}\]"]
    }
    #date {
    #    match => ["time", "dd/MMM/yyyy:HH:mm:ss Z"]
    #}
}
output {
 stdout {
  codec=>rubydebug{}
   }
 }

[elk@zjtest7-frontend config]$ ../bin/logstash -f stdin02.conf 
Settings: Default pipeline workers: 1
Pipeline main started
 10.171.246.184 [22/Sep/2016:00:13:59 +0800] "GET /resources/css/base.css?06212016 HTTP/1.1" - 200 12638 "https://www.zjcap.cn/" 
{
       "message" => " 10.171.246.184 [22/Sep/2016:00:13:59 +0800] \"GET /resources/css/base.css?06212016 HTTP/1.1\" - 200 12638 \"https://www.zjcap.cn/\" ",
      "@version" => "1",
    "@timestamp" => "2016-09-22T00:54:17.154Z",
          "host" => "0.0.0.0",
      "clientip" => "10.171.246.184",
          "time" => "22/Sep/2016:00:13:59 +0800"
}


打開時間轉換:
[elk@zjtest7-frontend config]$ ../bin/logstash -f stdin02.conf 
Settings: Default pipeline workers: 1
Pipeline main started
 10.171.246.184 [22/Sep/2016:00:13:59 +0800] "GET /resources/css/base.css?06212016 HTTP/1.1" - 200 12638 "https://www.zjcap.cn/" 
{
       "message" => " 10.171.246.184 [22/Sep/2016:00:13:59 +0800] \"GET /resources/css/base.css?06212016 HTTP/1.1\" - 200 12638 \"https://www.zjcap.cn/\" ",
      "@version" => "1",
    "@timestamp" => "2016-09-21T16:13:59.000Z",
          "host" => "0.0.0.0",
      "clientip" => "10.171.246.184",
          "time" => "22/Sep/2016:00:13:59 +0800"
}



/***** nginx 錯誤日誌
[elk@zjtest7-frontend config]$ cat stdin02.conf 
input {
    stdin {
    }
}
filter {
    grok {
        match => ["message", "(?<time>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME})"]
    }
    #date {
    #    match => ["time", "yyyy/MM/dd HH:mm:ss"]
    #}
}
output {
 stdout {
  codec=>rubydebug{}
   }
 }
 
關閉date插件:
[elk@zjtest7-frontend config]$ ../bin/logstash -f stdin02.conf 
Settings: Default pipeline workers: 1
Pipeline main started
 2016/09/22 08:36:55 [error] 14486#0: *55574 open() "/var/www/zjzc-web-frontEnd/apple-app-site-association"
{
       "message" => " 2016/09/22 08:36:55 [error] 14486#0: *55574 open() \"/var/www/zjzc-web-frontEnd/apple-app-site-association\"",
      "@version" => "1",
    "@timestamp" => "2016-09-22T01:47:28.405Z",
          "host" => "0.0.0.0",
          "time" => "2016/09/22 08:36:55"
}



開啓date插件:


[elk@zjtest7-frontend config]$ cat stdin02.conf 
input {
    stdin {
    }
}
filter {
    grok {
        match => ["message", "(?<time>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME})"]
    }
    date {
        match => ["time", "yyyy/MM/dd HH:mm:ss"]
    }
}
output {
 stdout {
  codec=>rubydebug{}
   }
 }
 
[elk@zjtest7-frontend config]$ ../bin/logstash -f stdin02.conf 
Settings: Default pipeline workers: 1
Pipeline main started
 2016/09/22 08:36:55 [error] 14486#0: *55574 open() "/var/www/zjzc-web-frontEnd/apple-app-site-association"
{
       "message" => " 2016/09/22 08:36:55 [error] 14486#0: *55574 open() \"/var/www/zjzc-web-frontEnd/apple-app-site-association\"",
      "@version" => "1",
    "@timestamp" => "2016-09-22T00:36:55.000Z",
          "host" => "0.0.0.0",
          "time" => "2016/09/22 08:36:55"
}


/******tomcat access 日誌
[elk@zjtest7-frontend config]$ cat stdin02.conf 
input {
    stdin {
    }
}
filter {
    grok {
        match => ["message", "\s*%{IPORHOST:clientip}\s+\-\s+\-\s+\[%{HTTPDATE:time}\]"]
    }
    date {
         match => ["time", "dd/MMM/yyyy:HH:mm:ss Z"]
    }
}
output {
 stdout {
  codec=>rubydebug{}
   }
 }


[elk@zjtest7-frontend config]$ ../bin/logstash -f stdin02.conf 
Settings: Default pipeline workers: 1
Pipeline main started
10.171.246.184 - - [22/Sep/2016:07:59:04 +0800] "POST /api/notice/page HTTP/1.1" 200 1194 0.055 121.40.169.62
{
       "message" => "10.171.246.184 - - [22/Sep/2016:07:59:04 +0800] \"POST /api/notice/page HTTP/1.1\" 200 1194 0.055 121.40.169.62",
      "@version" => "1",
    "@timestamp" => "2016-09-21T23:59:04.000Z",
          "host" => "0.0.0.0",
      "clientip" => "10.171.246.184",
          "time" => "22/Sep/2016:07:59:04 +0800"
}

/**********tomcat catalina.out 日誌

elk@zjtest7-frontend config]$ cat stdin02.conf   
input {  
    stdin {  
    }  
}  
  
filter {  
   grok {    
        match => ["message", "(?m)\s*%{TIMESTAMP_ISO8601:time}\s+(?<Level>(\S+)).*"]    
    }   
    date {  
        match => ["time", "yyyy-MM-dd HH:mm:ss,SSS"]  
    }  
}  
output {  
 stdout {  
  codec=>rubydebug{}  
   }  
 }  
   
[elk@zjtest7-frontend config]$ ../bin/logstash -f stdin02.conf   
Settings: Default pipeline workers: 1  
Pipeline main started  
2016-09-21 19:10:01,538 INFO com.zjzc.common.utils.HttpUtil  
{  
       "message" => "2016-09-21 19:10:01,538 INFO com.zjzc.common.utils.HttpUtil",  
      "@version" => "1",  
    "@timestamp" => "2016-09-21T11:10:01.538Z",  
          "host" => "0.0.0.0",  
          "time" => "2016-09-21 19:10:01,538",  
         "Level" => "INFO"  
}  

/************mysql slow log