filebeat收集nginx的json格式日誌

時間 2019-11-13

標籤 filebeat 收集 nginx json 格式日誌欄目 Nginx 简体版

原文原文鏈接

1、在nginx主機上安裝filebeat組件

[root@zabbix_server nginx]# cd /usr/local/src/
[root@zabbix_server src]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-x86_64.rpm
[root@zabbix_server src]# yum localinstall filebeat-6.2.4-x86_64.rpm

2、配置filebeat配置文件

[root@zabbix_server src]# vim /etc/filebeat/filebeat.yml 
filebeat.prospectors:
  - type: log
    enabled: true
    json.keys_under_root: true    #json格式收集
    json.overwrite_keys: true       #json格式收集
    paths:
      - /var/log/nginx/access.log  #須要收集的日誌文件路徑
    fields:
      log_topics: nginx-172.28.18.75 #設置日誌標題 
  
  output.logstash:
    hosts: ["172.28.18.69:10001"]  #輸出到logstash服務地址和端口

3、配置nginx.conf文件，設置json格式日誌

[root@zabbix_server src]# vim /etc/nginx/nginx.conf 
    log_format access_json_log  '{"@timestamp":"$time_local",'
                                  '"http_host":"$http_host",'
                                  '"clinetip":"$remote_addr",'
                                  '"request":"$request",'
                                  '"status":"$status",'
                                  '"size":"$body_bytes_sent",'
                                  '"upstream_addr":"$upstream_addr",'
                                  '"upstream_status":"$upstream_status",'
                                  '"upstream_response_time":"$upstream_response_time",'
                                  '"request_time":"$request_time",'
                                  '"http_referer":"$http_referer",'
                                  '"http_user_agent":"$http_user_agent",'
                                  '"http_x_forwarded_for":"$http_x_forwarded_for"}';
                                      
    access_log  /var/log/nginx/access.log  access_json_log;

4、檢查配置文件語法，並重載

[root@zabbix_server src]# nginx -tc /etc/nginx/nginx.conf 
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@zabbix_server src]# nginx -s reload

5、查看access.log日誌輸出

[root@zabbix_server src]# tail -f /var/log/nginx/access.log
{"@timestamp":"15/Jul/2019:10:33:23 +0800","http_host":"zabbix.9500.cn","clinetip":"219.239.8.14","request":"POST /zabbix.php?sid=311c76f6740bc2f7&action=widget.problems.view HTTP/1.1","status":"200","size":"33684","upstream_addr":"127.0.0.1:9000","upstream_status":"200","upstream_response_time":"1.103","request_time":"1.103","http_referer":"http://zabbix.9500.cn/zabbix.php?action=dashboard.view&ddreset=1","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.221 Safari/537.36 SE 2.X MetaSr 1.0","http_x_forwarded_for":"-"}
{"@timestamp":"15/Jul/2019:10:33:23 +0800","http_host":"zabbix.9500.cn","clinetip":"219.239.8.14","request":"POST /zabbix.php?sid=9203532d28b920b3&action=widget.problems.view HTTP/1.1","status":"200","size":"33856","upstream_addr":"127.0.0.1:9000","upstream_status":"200","upstream_response_time":"0.759","request_time":"0.759","http_referer":"http://zabbix.9500.cn/zabbix.php?action=dashboard.view&ddreset=1","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36","http_x_forwarded_for":"-"}

已經變成json格式的日誌輸出了。php

6、在logstash服務器上的/etc/logstash/conf.d/目錄下，新建一個nginx.conf文件

用於監聽10001端口，負責收集filebeat傳遞過來的日誌數據，並向elasticseatch服務發送日誌數據java

[root@server-1 conf.d]# vim /etc/logstash/conf.d/nginx.conf
input {
  beats {
   port=>10001   #監聽filebeat發送日誌的端口
  }
}

output {
  if[fields][log_topics]=="nginx-172.28.18.75"{   #判斷是不是filebeat配置設置的fields字段，是則發送到elastcisearch
    elasticsearch {
     hosts=>["172.28.18.69:9200"]
     index=>"nginx-172.28.18.75-%{+YYYY.MM.dd}"   #設置索引名字
    } 
  }
}

7、重啓logstash服務，並查看端口

[root@server-1 conf.d]# systemctl restart logstash
[root@server-1 conf.d]# netstat -tunlp|grep 10001
tcp6       0      0 :::10001                :::*                    LISTEN      26599/java

啓動成功nginx

8、重啓nginx主機上的filebeat服務

[root@zabbix_server src]# systemctl restart filebeat

9、查看elasticsearch服務器上是否有新建的索引

[root@zabbix_server etc]# curl 172.28.18.69:9200/_cat/indices?v
health status index                         uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana                       nQhtAX5YSSKzT1p0G-1XtA   1   0          4            0     25.2kb         25.2kb
yellow open   nginx-172.28.5.214-2019.07.15 tH8oVfZhQy-TdnIPHyFgvQ   5   1    2106272            0    668.8mb        668.8mb
yellow open   system-syslog-2019.07         REp7fM_gSaquo9PX2_sREQ   5   1   10772928            0      2.6gb          2.6gb