Logstash用法實例

首先提供一些參考的網站：

https://www.gitbook.com/book/chenryn/kibana-guide-cn

http://udn.yyuap.com/doc/logstash-best-practice-cn/index.html

https://www.gitbook.com/book/chenryn/kibana-guide-cn
http://www.learnes.net/index.html

下面由簡單到複雜，介紹一些Logstash的配置方法

安裝完Logstash後，默認安裝在/opt/logstash中，在bin目錄中有一些命令可使用。

最簡單的配置語法：

input {
    stdin {}
    syslog {}
}

Logstash的Helloworld：

[root@BLELOIST003 logstash]# bin/logstash -e 'input{stdin{}}output{stdout{}}'
!!! Please upgrade your java version, the current version '1.7.0_45-mockbuild_2013_10_22_03_37-b00' may cause problems. We recommend a minimum version of 1.7.0_51
Settings: Default pipeline workers: 8
Logstash startup completed
hello world
2016-03-17T03:30:02.147Z BLELOIST003.lenovo.com hello world
[root@BLELOIST003 logstash]#
[root@BLELOIST003 logstash]# bin/logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'
!!! Please upgrade your java version, the current version '1.7.0_45-mockbuild_2013_10_22_03_37-b00' may cause problems. We recommend a minimum version of 1.7.0_51
Settings: Default pipeline workers: 8
Logstash startup completed
hello world
{
       "message" => "hello world",
      "@version" => "1",
    "@timestamp" => "2016-03-17T03:31:30.308Z",
          "host" => "BLELOIST003.lenovo.com"
}

最基本的配置組成是input output，可選的是filter。input中的stdin和output中的stdout表明了命令行窗口的輸入輸出。

監控apache日誌，這裏沒有作過濾。

[root@BLELOIST003 logstash]# bin/logstash -e '
 input {
   file { 
     type => "apache-access" 
     path => "/apphome/ptc/Windchill_10.2/HTTPServer/logs/access.log*" 
     start_position => end 
     sincedb_path => "/dev/null"
   }
 }
 filter{
   
 }
 output {
   stdout{ codec=>rubydebug }
 }'
!!! Please upgrade your java version, the current version '1.7.0_45-mockbuild_2013_10_22_03_37-b00' may cause problems. We recommend a minimum version of 1.7.0_51
Settings: Default pipeline workers: 8
Logstash startup completed
{
       "message" => "10.100.90.241 - - [17/Mar/2016:11:35:00 +0800] \"GET / HTTP/1.1\" 304 - 411",
      "@version" => "1",
    "@timestamp" => "2016-03-17T03:35:01.162Z",
          "path" => "/apphome/ptc/Windchill_10.2/HTTPServer/logs/access.log_2016-03-17_00_00_00",
          "host" => "BLELOIST003.lenovo.com",
          "type" => "apache-access"
}
{
       "message" => "10.99.80.8 - dummy002 [17/Mar/2016:11:35:01 +0800] \"POST /LOIS/servlet/SEOTranslationService HTTP/1.0\" 200 90360 503291",
      "@version" => "1",
    "@timestamp" => "2016-03-17T03:35:03.178Z",
          "path" => "/apphome/ptc/Windchill_10.2/HTTPServer/logs/access.log_2016-03-17_00_00_00",
          "host" => "BLELOIST003.lenovo.com",
          "type" => "apache-access"
}

start_position => end 表明了只監控新增長的日誌內容。

加上正則表達式的過濾：

[root@BLELOIST003 logstash]# bin/logstash -e '
 input {
   file { 
     type => "apache-access" 
     path => "/apphome/ptc/Windchill_10.2/HTTPServer/logs/access.log*" 
     start_position => end 
     sincedb_path => "/dev/null"
   }
 }
 filter{
   if [path] =~ "access" {
     mutate { replace => { "type" => "ApacheLogs" } }
     grok {
       match => { "message" => "%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:userName} \[%{HTTPDATE:logTime}\] \"%{WORD:verb} %{NOTSPACE:request} (?:%{NOTSPACE:httpversion}|)\" (?:%{NUMBER:state}|-) (?:%{NUMBER:bytes}|-) %{NUMBER:duration}"}
     }
   }
   date {
     match => [ "logTime" , "dd/MMM/yyyy:HH:mm:ss Z" ]
   }
 }
 output {
   stdout{ codec=>rubydebug }
 }'

Settings: Default pipeline workers: 8
Logstash startup completed
{
        "message" => "10.100.90.241 - - [17/Mar/2016:12:05:27 +0800] \"GET /Windchill/ HTTP/1.1\" 404 208 424",
       "@version" => "1",
     "@timestamp" => "2016-03-17T04:05:27.000Z",
           "path" => "/apphome/ptc/Windchill_10.2/HTTPServer/logs/access.log_2016-03-17_00_00_00",
           "host" => "BLELOIST003.lenovo.com",
           "type" => "ApacheLogs",
       "clientip" => "10.100.90.241",
          "ident" => "-",
       "userName" => "-",
        "logTime" => "17/Mar/2016:12:05:27 +0800",
           "verb" => "GET",
        "request" => "/Windchill/",
    "httpversion" => "HTTP/1.1",
          "state" => "404",
          "bytes" => "208",
       "duration" => "424"
}
{
        "message" => "10.100.90.241 - - [17/Mar/2016:12:05:27 +0800] \"GET /favicon.ico HTTP/1.1\" 200 207 476",
       "@version" => "1",
     "@timestamp" => "2016-03-17T04:05:27.000Z",
           "path" => "/apphome/ptc/Windchill_10.2/HTTPServer/logs/access.log_2016-03-17_00_00_00",
           "host" => "BLELOIST003.lenovo.com",
           "type" => "ApacheLogs",
       "clientip" => "10.100.90.241",
          "ident" => "-",
       "userName" => "-",
        "logTime" => "17/Mar/2016:12:05:27 +0800",
           "verb" => "GET",
        "request" => "/favicon.ico",
    "httpversion" => "HTTP/1.1",
          "state" => "200",
          "bytes" => "207",
       "duration" => "476"
}

這裏的if [path] =~ "access"表明了判斷語句，能夠參考這裏：

http://kibana.logstash.es/content/logstash/get_start/full_config.html

這裏的過濾中grok中匹配：

match => { "message" => "%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:userName} \[%{HTTPDATE:logTime}\] \"%{WORD:verb} %{NOTSPACE:request} (?:%{NOTSPACE:httpversion}|)\" (?:%{NUMBER:state}|-) (?:%{NUMBER:bytes}|-) %{NUMBER:duration}"}

其中%{IPORHOST:clientip}表示具體的正則匹配，IPORHOST表明了logstash的pattern，具體能夠參考以下連接，而clientip則表明了在logstash中信息傳遞的變量。

https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns