ELK--grok正則分析日誌與多日誌收集

1、收集nginx日誌

logstash配置文件

訪問nginx，產生新的日誌

查找索引

有一條logstash-2019-08-28的索引

2、grok日誌切割

日誌格式

192.168.0.2 - - [28/Aug/2019:22:35:03 +0800] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko Core/1.70.3722.400 QQBrowser/10.5.3776.400"

grok正則匹配

(?<clientip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) - - (?<requesttime>\[[0-9]{1,2}\/[A-z]+\/[0-9]{4}\:[0-9]{2}\:[0-9]{2}\:[0-9]{2} \+[0-9]*\]) "(?<requesttype>[A-Z]+) (?<requesturl>[^ ]+) (?<requestv>HTTP/\d\.\d)" (?<requestnode>[0-9]+) (?<requestsize>[0-9]+) "(?<content>[^ ]|(http|https)://[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\/)" "(?<ua>(a-Z|0-9| |.)+)"

修改配置文件，增長grok正則匹配

input {

  file {

    path => "/usr/local/nginx/logs/access.log"

  }

}

filter {

    grok {

        match => {

            "message" => '(?<clientip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) - - (?<requesttime>\[[0-9]{1,2}\/[A-z]+\/[0-9]{4}\:[0-9]{2}\:[0-9]{2}\:[0-9]{2} \+[0-9]*\]) "(?<requesttype>[A-Z]+) (?<requesturl>[^ ]+) (?<requestv>HTTP/\d\.\d)" (?<requestnode>[0-9]+) (?<requestsize>[0-9]+) "(?<content>[^ ]|(http|https)://[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\/)" "(?<ua>(a-Z|0-9| |.)+)"'

        }

    }

}

output {

  elasticsearch {

    hosts => ["http://192.168.0.9:9200"]

  }

}

重載配置文件

ps -aux | grep logstash

kill -1 pid

網頁訪問

3、刪除不須要的字段

上面咱們能夠看到，不少字段實際上是不徹底須要的

修改配置文件並重載配置文件

         remove_field => ["message","log","beat","offset","prospector","host","@version"]

web界面顯示

4、logstash分析完整日誌

修改配置文件

    start_position => "beginning"

    sincedb_path => "/dev/null"

刪除索引後從新加載配置文件，這樣收集的日誌將從頭開始分析

5、多日誌收集，並對索引進行定義

修改配置文件

input {

  file {

    path => "/usr/local/nginx/logs/access.log"

    type => "nginx"

    start_position => "beginning"

    sincedb_path => "/dev/null"

  }

  file {

    path => "/var/log/secure"

    type => "secure"

    start_position => "beginning"

    sincedb_path => "/dev/null"

  }

}

filter {

    grok {

        match => {

            "message" => '(?<clientip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) - - (?<requesttime>\[[0-9]{1,2}\/[A-z]+\/[0-9]{4}\:[0-9]{2}\:[0-9]{2}\:[0-9]{2} \+[0-9]*\]) "(?<requesttype>[A-Z]+) (?<requesturl>[^ ]+) (?<requestv>HTTP/\d\.\d)" (?<requestnode>[0-9]+) (?<requestsize>[0-9]+) "(?<content>[^ ]|(http|https)://[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\/)" "(?<ua>(a-Z|0-9| |.)+)"'

        }

         remove_field => ["message","log","beat","offset","prospector","host","@version"]

    }

}

output {

 if [type] == "nginx" {

  elasticsearch {

    hosts => ["http://192.168.0.9:9200"]

    index => "nginx-%{+YYYY.MM.dd}"

        }

     }

  else if [type] == "secure" {

    elasticsearch {

    hosts => ["http://192.168.0.9:9200"]

    index => "secure-%{+YYYY.MM.dd}"

        }

     }

  }

刪除索引，重載配置文件

建立索引