ELK實踐（二）：收集Nginx日誌

時間  2019-12-05
標籤 elk 實踐 收集 nginx 日誌 欄目 Nginx 简体版
原文   原文鏈接

Nginx訪問日誌

這裏補充下Nginx訪問日誌使用的說明。通常在nginx.conf主配置文件裏須要定義一種格式：php

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    '$status $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for" $request_time';

上面的格式我是基於默認的加了一個$request_time。html

而後子配置使用：nginx

access_log  logs/myapp.log  main;

便可。web

Filebeat採集日誌數據到ElasticSearch

配置：vim

su -e elk
cd /usr/local/elk
vim beats/filebeat/filebeat.test_nginx.yml

配置詳情：數組

filebeat.prospectors:
- type: log
  input_type: log
  paths:
    - /work/yphp/nginx/logs/*.log
  tags: ["ngx", "yujc"]
  fields:
    logIndex: nginx
    docType: nginx-access
  fields_under_root: true
  tail_files: false

output.elasticsearch:
  hosts: ["127.0.0.1:9200"]
  index: "test-nginx-%{+yyyy.MM.dd}"

配置說明：ruby

filebeat.prospectors:服務器

  • type 日誌類型，默認log
  • input_type 輸入類型，默認log
  • paths 採集的日誌，可使用通配符。支持多個
  • tags 自定義標籤，是個數組。自定義
  • fields 自定義字段
  • fields_under_root 自定義字段是否追加到根。若是爲false，fields配置的字段鍵名是fields
  • tail_files 是否從末尾開始採集
  • document_type 自定義字段，用於Logsatsh區分來源，在Logsatsh裏用變量type表示

output.elasticsearch:架構

  • hosts 配置ES節點，數組格式，支持多個。
  • index 配置ES索引。不配置使用默認的 filebeat-*
  • protocol 配置協議，例如http,https
  • username 配置ES用戶名，例如elastic
  • password 配置ES密碼，例如changeme

設置權限600，並啓動filebeat:app

chmod -R 600 beats/filebeat/filebeat.test_nginx.yml

./beats/filebeat/filebeat -c beats/filebeat/filebeat.test_nginx.yml

而後訪問Nginx應用，查看ES是否新增了一個索引：

$ curl http://127.0.0.1:9200/_cat/indices?v | grep test-nginx
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
105  1161  105  1161    0     0   123k      0 --:--:-- --:--:-- --:--:--  125k
yellow open   test-nginx-2018.09.24             ArxrVVOkTjG8ZlXJjb9bVg   5   1          1            0     11.6kb         11.6kb

咱們查看一條數據：

$ curl http://127.0.0.1:9200/test-nginx-2018.09.24/_search?q=*&size=1


{
  "_index": "test-nginx-2018.09.24",
  "_type": "doc",
  "_id": "AWYKkBqtJzfnbYlB_DRX",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2018-09-24T07:51:43.140Z",
    "beat": {
      "hostname": "2106567e5bce",
      "name": "2106567e5bce",
      "version": "5.6.2"
    },
    "docType": "nginx-access",
    "input_type": "log",
    "logIndex": "nginx",
    "message": "172.16.10.1 - - [24/Sep/2018:07:51:40 +0000] \"GET /?time=22 HTTP/1.1\" 200 97991 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36\" \"-\" 0.009",
    "offset": 5243,
    "source": "/work/yphp/nginx/logs/hello71.log",
    "tags": [
      "ngx",
      "yujc"
    ],
    "type": "log"
  },
  "fields": {
    "@timestamp": [
      1537775503140
    ]
  },
  "sort": [
    1537775503140
  ]
}

能夠看到已經有數據了。可是日誌內容做爲一個總體（字段是message）了。

Filebeat採集日誌數據，Logstash過濾發到ElasticSearch

配置：

su -e elk
cd /usr/local/elk
vim beats/filebeat/filebeat.test_nginx2.yml

配置詳情：

filebeat.prospectors:
- type: log
  input_type: log
  paths:
    - /work/yphp/nginx/logs/*.log
  tags: ["ngx", "yujc"]
  fields:
    logIndex: nginx
    docType: nginx-access
  fields_under_root: true
  tail_files: false

output.logstash:
  hosts: ["127.0.0.1:5044"]

配置logstash

su -e elk
cd /usr/local/elk
vim logstash/config/conf.d/filebeat.conf

配置詳情：

input {
    beats {
        port => 5044
    }
}

filter {
    grok {
        match => { "message" => "%{IPORHOST:remote_ip} - %{DATA:user_name} \[%{HTTPDATE:time}\] \"%{WORD:method} %{DATA:url} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:body_sent:bytes} \"%{DATA:referrer}\" \"%{DATA:agent}\" \"%{DATA:x_forwarded_for}\" %{NUMBER:request_time}" }
        remove_field => "message"
    }
}

output {
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
        index => "test-nginx2-%{type}-%{+YYYY.MM.dd}"
        document_type => "%{type}"
    }
    stdout { codec => rubydebug }
}

我使用的nginx日誌格式是在標準格式後面加了2個字段$http_x_forwarded_for和$request_time：

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" ' 
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" $request_time';

日誌示例：

172.16.10.1 - - [24/Sep/2018:09:04:40 +0000] "GET /?time=2244 HTTP/1.1" 200 98086 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36" "-" 0.002

上面的grok表達式是：

%{IPORHOST:remote_ip} - %{DATA:user_name} \[%{HTTPDATE:time}\] \"%{WORD:method} %{DATA:url} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:body_sent:bytes} \"%{DATA:referrer}\" \"%{DATA:agent}\" \"%{DATA:x_forwarded_for}\" %{NUMBER:request_time}

咱們先使用Grok Debugger 工具在線調試下，看看寫的grok是否正確。我以前沒有測試以前啓動，發現ES裏沒有grok裏解析出來的字段，後來在命令行看到filebeat的輸出（前臺運行）：

$ ./beats/filebeat/filebeat -c beats/filebeat/filebeat.test_nginx2.yml

{
    "@timestamp" => 2018-09-24T09:01:19.555Z,
      "logIndex" => "nginx",
        "offset" => 6467,
       "docType" => "nginx-access",
      "@version" => "1",
    "input_type" => "log",
          "beat" => {
            "name" => "2106567e5bce",
        "hostname" => "2106567e5bce",
         "version" => "5.6.2"
    },
          "host" => "2106567e5bce",
        "source" => "/work/yphp/nginx/logs/hello71.log",
       "message" => "172.16.10.1 - - [24/Sep/2018:09:01:14 +0000] \"GET /?time=2244 HTTP/1.1\" 200 98087 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36\" \"-\" 0.195",
          "type" => "log",
          "tags" => [
        [0] "ngx",
        [1] "yujc",
        [2] "beats_input_codec_plain_applied",
        [3] "_grokparsefailure"
    ]
}

最後面提示了_grokparsefailure，說明grok部分寫的有問題。因爲是參考的網上教程，也加上剛接觸，不知道怎麼配置，filebeat.conf調試了好久才生效。

咱們打開Grok Debugger，第一個輸入框輸入filebeat採集的消息原文message字段裏的內容，第二個輸入框輸入grok表達式：

點擊Go按鈕便可解析。若是下面的內容是{}說明解析失敗，而後能夠修改表達式，該工具會自動解析。最終解析結果：

{
  "remote_ip": [
    [
      "172.16.10.1"
    ]
  ],
  "HOSTNAME": [
    [
      "172.16.10.1"
    ]
  ],
  "IP": [
    [
      null
    ]
  ],
  "IPV6": [
    [
      null
    ]
  ],
  "IPV4": [
    [
      null
    ]
  ],
  "user_name": [
    [
      "-"
    ]
  ],
  "time": [
    [
      "24/Sep/2018:08:47:59 +0000"
    ]
  ],
  "MONTHDAY": [
    [
      "24"
    ]
  ],
  "MONTH": [
    [
      "Sep"
    ]
  ],
  "YEAR": [
    [
      "2018"
    ]
  ],
  "TIME": [
    [
      "08:47:59"
    ]
  ],
  "HOUR": [
    [
      "08"
    ]
  ],
  "MINUTE": [
    [
      "47"
    ]
  ],
  "SECOND": [
    [
      "59"
    ]
  ],
  "INT": [
    [
      "+0000"
    ]
  ],
  "method": [
    [
      "GET"
    ]
  ],
  "url": [
    [
      "/?time=2244"
    ]
  ],
  "http_version": [
    [
      "1.1"
    ]
  ],
  "BASE10NUM": [
    [
      "1.1",
      "200",
      "98086",
      "0.002"
    ]
  ],
  "response_code": [
    [
      "200"
    ]
  ],
  "body_sent": [
    [
      "98086"
    ]
  ],
  "referrer": [
    [
      "-"
    ]
  ],
  "agent": [
    [
      "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
    ]
  ],
  "x_forwarded_for": [
    [
      "-"
    ]
  ],
  "request_time": [
    [
      "0.002"
    ]
  ]
}

而後能夠啓動logstash了。

測試logstash配置是否經過：

./logstash/bin/logstash -f logstash/config/conf.d/filebeat.conf --config.test_and_exit

Config Validation Result: OK. Exiting Logstash
# 啓動logstash
./logstash/bin/logstash &

# 啓動filebeat
./beats/filebeat/filebeat -c beats/filebeat/filebeat.test_nginx2.yml

咱們再次訪問Nginx應用，而後咱們查看一條數據：

$ curl http://127.0.0.1:9200/test-nginx2-log-2018.09.24/_search?q=*&size=1&sort=@timestamp:desc

{
  "took": 14,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 3,
    "max_score": null,
    "hits": [
      {
        "_index": "test-nginx2-log-2018.09.24",
        "_type": "log",
        "_id": "AWYK0to8JzfnbYlB_DRx",
        "_score": null,
        "_source": {
          "response_code": "200",
          "agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36",
          "logIndex": "nginx",
          "offset": 6875,
          "method": "GET",
          "docType": "nginx-access",
          "user_name": "-",
          "input_type": "log",
          "http_version": "1.1",
          "source": "/work/yphp/nginx/logs/hello71.log",
          "message": """172.16.10.1 - - [24/Sep/2018:09:04:40 +0000] "GET /?time=2244 HTTP/1.1" 200 98086 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36" "-" 0.002""",
          "type": "log",
          "url": "/?time=2244",
          "tags": [
            "ngx",
            "yujc",
            "beats_input_codec_plain_applied"
          ],
          "x_forwarded_for": "-",
          "referrer": "-",
          "@timestamp": "2018-09-24T09:04:40.404Z",
          "remote_ip": "172.16.10.1",
          "request_time": "0.002",
          "@version": "1",
          "beat": {
            "name": "2106567e5bce",
            "hostname": "2106567e5bce",
            "version": "5.6.2"
          },
          "host": "2106567e5bce",
          "body_sent": "98086",
          "time": "24/Sep/2018:09:04:40 +0000"
        },
        "sort": [
          1537779880404
        ]
      }
    ]
  }
}

裏面就包含了全部咱們解析出來的字段。

kibana裏查看

打開kibana web地址：http://127.0.0.1:5601，依次打開：Management
-> Kibana -> Index Patterns ,選擇Create Index Pattern：

a. Index pattern 輸入：test-nginx2-* ；

b. Time Filter field name 選擇 @timestamp。

c. 點擊Create。

而後打開Discover，選擇 filebeat-* 就能看到日誌數據了。

能夠看到詳細字段：

參考

一、Logstash使用grok過濾nginx日誌（二） - Orgliny - 博客園 https://www.cnblogs.com/Orgliny/p/5592186.html 二、Rsyslog日誌服務搭建 - K‘e0llm - 博客園 http://www.cnblogs.com/Eivll0m/p/6700828.html 三、Logstash中如何處理到ElasticSearch的數據映射 - Cocowool - 博客園 https://www.cnblogs.com/cocowool/p/7347069.html 四、ELK 架構之 Logstash 和 Filebeat 安裝配置 - 田園裏的蟋蟀 - 博客園 http://www.cnblogs.com/xishuai/p/elk-logstash-filebeat.html 五、搭建ELK日誌分析平臺（下）—— 搭建kibana和logstash服務器-zero菌-51CTO博客 http://blog.51cto.com/zero01/2082794

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程