【原創】大叔經驗分享（28）ELK分析nginx日誌

提早安裝好elk（elasticsearch、logstach、kibana）

一 啓動logstash

$LOGSTASH_HOME默認位於/usr/share/logstash或/opt/logstash

1 nginx日誌使用默認格式

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '

                      '$status $body_bytes_sent "$http_referer" '

                      '"$http_user_agent" "$http_x_forwarded_for"';

2 下載geo庫

# cd /etc/logstash
# wget https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
# tar xvf GeoLite2-City.tar.gz

3 增長logstash配置

# cat /etc/logstash/conf.d/nginx_access.conf

input {

    file {

        path => [ "/path/to/nginx/access.log" ]

        start_position => "beginning"

        ignore_older => 0

    }

}

filter {

    grok {

        match => { "message" => "%{IPORHOST:client_ip} (%{USER:ident}|-) (%{USER:auth}|-) \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} (%{NOTSPACE:request}|-)(?: HTTP/%{NUMBER:http_version})?|-)\" (%{NUMBER:response}|-) (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}" }

    }

    geoip {

      source => "client_ip"

      target => "geoip"

      database => "/etc/logstash/GeoLite2-City_20190122/GeoLite2-City.mmdb"

      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]

      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]

    }

    mutate {

      convert => [ "[geoip][coordinates]", "float" ]

      convert => [ "response","integer" ]

      convert => [ "bytes","integer" ]

      replace => { "type" => "nginx_access" }

      remove_field => "message"

    }

    date {

      match => [ "timestamp","dd/MMM/yyyy:HH:mm:ss Z"]

    }

    mutate {

      remove_field => "timestamp"

    }

}

output {

    elasticsearch {

        hosts => ["$es_server:9200"]

        index => "logstash-nginx-access-%{+YYYY.MM.dd}"

    }

    stdout {codec => rubydebug}

}

修改其中的nginx日誌路徑以及es的host；

若是你修改過nginx log format，還須要修改grok格式，grok格式能夠在這裏調試：http://grokdebug.herokuapp.com/

若是格式有誤，會報 _grokparsefailure

4 測試配置是否正常

$LOGSTASH_HOME/bin/logstash -t -f /etc/logstash/conf.d/nginx_access.conf

正常應該會打印Configuration OK

5 啓動logstash

$LOGSTASH_HOME/bin/logstash -f /etc/logstash/conf.d/nginx_access.conf

二 查看elasticsearch

# curl http://$es_server:9200/_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open logstash-nginx-access-2019.01.26 -szaQCE3THyq-fXbU18riQ 5 1 7875 0 862.7kb 465.2kb

此時es中應該有了索引

三 配置kibana

瀏覽器打開：http://$kibana_server:5000

配置索引pattern爲：logstash-nginx-access-*
配置Time-field爲：@timestamp

而後能夠配置各類visualization和dashboard