單節點elk+grafana搭建和採集nginx訪問日誌

需求：展現nginx打印的信息（如 pv uv http狀態碼 url前十）

此博客架構：es+kibana 在一臺，logstash+nginx 在另外一臺

1、

安裝包提供：

elasticsearch-7.2.0-linux-x86_64.tar.gz
logstash-7.2.0.tar.gz
kibana-7.2.0-linux-x86_64.tar.gz
grafana-7.2.0-1.x86_64.rpm

2、部署es前提條件

1. echo "vm.max_map_count=655360" >>/etc/sysctl.conf
2. sysctl -p
3. vim /etc/security/limits.conf
4.    * soft core unlimited
5.    * hard core unlimited
6.    * soft nofile 1048576
7.    * hard nofile 1048576
8.    * soft nproc 65536
9.    * hard nproc 65536
10.    * soft sigpending 255983
11.    * hard sigpending 255983
12.    * soft memlock unlimited
13.    * hard memlock unlimited
14. vim /etc/security/limits.d/20-nproc.conf
15. * soft nproc 65536

16. * hard nproc 65536

    3、部署es

17. useradd elastic #不能用root起服務，若是root起須要修改相關配置

18. 因爲咱們才用的是elk7.2 因此jdk要大於java8,本環境安裝java11

19. yum install java-11-openjdk -y

20. tar -zxvf elasticsearch-7.2.0-linux-x86_64.tar.gz
    mv elasticsearch-7.2.0 es
    mv elasticsearch.yml elasticsearch.yml.bak
    vim elasticsearch.yml
    [root@localhost config]# grep -vE '^#|^$'  elasticsearch.yml
    cluster.name: my-application
    node.name: node-1
    path.data: /data/es/data
    path.logs: /data/es/logs
    bootstrap.memory_lock: false
    bootstrap.system_call_filter: false
    network.host: 0.0.0.0
    cluster.initial_master_nodes: ["node-1"]
    
    chown -R elastic.elastic /data/es
    
    進入elastic啓動服務：

21. nohup ./bin/elasticsearch -d &

測試：curl -XGET 'localhost:9200/?pretty'

 查看index : curl 'localhost:9200/_cat/indices?v'

4、部署logstash

tar -zxvf logstash-7.2.0.tar.gz

測試： 輸入到控制檯  ../bin/logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'
      輸入到es     ../bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["localhost:9200"]} }'

5、收集nginx日誌

cat /etc/yum.repos.d/nginx.repo
#在文件中寫入如下內容：
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=0
enabled=1

yum install nginx -y

日誌格式：

  log_format main   '{"@timestamp":"$time_iso8601",'
                        '"@source":"$server_addr",'
                        '"hostname":"$hostname",'
                        '"ip":"$http_x_forwarded_for",'
                        '"client":"$remote_addr",'
                        '"request_method":"$request_method",'
                        '"scheme":"$scheme",'
                        '"domain":"$server_name",'
                        '"referer":"$http_referer",'
                        '"request":"$request_uri",'
                        '"args":"$args",'
                        '"size":$body_bytes_sent,'
                        '"status": $status,'
                        '"responsetime":$request_time,'
                        '"upstreamtime":"$upstream_response_time",'
                        '"upstreamaddr":"$upstream_addr",'
                        '"http_user_agent":"$http_user_agent",'
                        '"https":"$https",'
                        '"message":"$remote_addr $server_name $status $http_user_agent"'
                        '}';
                        
                        access_log  logs/access.log  main；   #改爲上面添加的名稱保存後重載Nginxnginx -s reload

配置收集nginx訪問日誌conf 添加收集nginx的配置vim testng.conf
input {
    file {
        path => [ "/data/log/nginx/*.access.log" ]
        ignore_older => 0
        type => "nginx-log"
        codec => json
    }
}

filter {
    mutate {
      convert => [ "status","integer" ]
      convert => [ "size","integer" ]
      convert => [ "upstreatime","float" ]
      remove_field => "message"
    }
    geoip {
        source => "ip"
    }

}
output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "nginx-log-%{+YYYY.MM.dd}"

    }
stdout{
         codec => rubydebug  #控制檯打印日誌
       }

}
}

能夠加上--configtest參數，測試下配置文件是否有語法錯誤或配置不當的地方/bin/logstash -f file.conf --configtest
放入後臺啓動 nohup ../bin/logstash -f testng.conf & 

curl localhost #這個時候應該控制檯就有日誌打印出來了，再去看es索引 應該就有了

6、部署grafana

yum install grafana-7.2.0-1.x86_64.rpm
#安裝插件
grafana-cli plugins install grafana-piechart-panel
grafana-cli plugins install grafana-worldmap-panel
etc/init.d/grafana-server start

導入模板 
 #能夠才用這個，不過要對應的改規則
適用於本博客的模板：

7、部署kibana

tar -zxvf kibana-7.2.0-linux-x86_64.tar.gz
cd kibana-7.2.0-linux-x86_64
cd config/
cp kibana.yml kibana.yml.bak
vim kibana.yml
nohup ./bin/kibana & #用非root用戶啓動

8、展現圖(模板右上角能夠直接跳轉到kibana，要在json裏面修改正確的地址)