13 EBLK日誌分析

時間  2020-12-09
標籤 java node nginx web shell json vim tomcat curl elasticsearch 欄目 Java 简体版
原文   原文鏈接

1. EBLK的解釋:

  • E Elasticsearch java
  • B Filebeat Go
  • L Logstash java
  • K Kibana java

2. 日誌分析需求:

  1. 找出訪問排名前十的IP,URL
  2. 找出10點到12點之間排名前十的IP,URL
  3. 對比昨天這個時間段訪問狀況有什麼變化
  4. 對比上個星期同一天同一時間段的訪問變化
  5. 找出搜索引擎訪問的次數和每一個搜索引擎各訪問了多少次
  6. 指定域名的關鍵連接訪問次數,響應時間
  7. 網站HTTP狀態碼狀況
  8. 找出攻擊者的IP地址,這個IP訪問了什麼頁面,這個IP何時來的,何時走的,共訪問了多少次
  9. 5分鐘內告訴我結果

3. 各類配置

3.1 單機ES環境配置

[命令]java

systemctl stop elasticsearch

rm -rf /var/lib/elasticsearch/*

cat > /etc/elasticsearch/elasticsearch.yml << 'EOF'
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 127.0.0.1,10.0.0.51
http.port: 9200
discovery.seed_hosts: ["10.0.0.51"]
cluster.initial_master_nodes: ["10.0.0.51"]
EOF

systemctl start elasticsearch

3.2 收集普通格式的nginx日誌

[命令]node

vim /etc/filebeat/filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
output.elasticsearch:
  hosts: ["10.0.0.51:9200"]

3.3 收集JSON格式的nginx日誌

[命令]nginx

目前不完善的地方:
1.日誌字段不能拆分,不能單獨顯示
2.索引名稱不是自定義

咱們指望的結果:
1.日誌字段能夠單獨顯示

$remote_addr 	10.0.0.1
- 				-
$remote_user 	-
[$time_local] 	[08/Oct/2020:10:27:44 +0800]
$request		GET /zhangya HTTP/1.1
$status 		404
$body_bytes_sent	555
$http_referer		-
$http_user_agent	Chrome
$http_x_forwarded_for -


操做步驟:
1.中止filebeat和nginx
systemctl stop filebeat nginx

2.清空Nginx日誌
> /var/log/nginx/access.log

3.刪除ES索引

4.修改Nginx日誌爲json格式:
log_format json '{ "time_local": "$time_local", '
                          '"remote_addr": "$remote_addr", '
                          '"referer": "$http_referer", '
                          '"request": "$request", '
                          '"status": $status, '
                          '"bytes": $body_bytes_sent, '
                          '"agent": "$http_user_agent", '
                          '"x_forwarded": "$http_x_forwarded_for", '
                          '"up_addr": "$upstream_addr",'
                          '"up_host": "$upstream_http_host",'
                          '"upstream_time": "$upstream_response_time",'
                          '"request_time": "$request_time"'
    ' }';
    access_log  /var/log/nginx/access.log  json;
	


5.重啓nginx
nginx -t 
systemctl restart nginx 

6.訪問並測試
curl 127.0.0.1 
tail -f /var/log/nginx/access
# 修改後的日誌結果:
{ 
  "time_local": "08/Oct/2020:11:10:17 +0800", 
  "remote_addr": "127.0.0.1", 
  "referer": "-", 
  "request": "GET / HTTP/1.1", 
  "status": 200, 
  "bytes": 5, 
  "agent": "curl/7.29.0", 
  "x_forwarded": "-", 
  "up_addr": "-",
  "up_host": "-",
  "upstream_time": "-",
  "request_time": "0.000"
}

7.修改filebeat配置文件
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]

8.重啓filebeat
systemctl restart filebeat

9.訪問並測試

10.kibana刪除舊索引,建立新索引

3.4 自定義索引名稱

[命令]web

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "nginx-%{[agent.version]}-%{+yyyy.MM}"

setup.ilm.enabled: false
setup.template.enabled: false

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

3.5 日誌類型定義索引名稱

[命令]shell

方法1:囉嗦
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log

processors:
  - drop_fields:
      fields: ["ecs","log"] 

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "nginx-access-%{[agent.version]}-%{+yyyy.MM}"
      when.contains:
        log.file.path: "/var/log/nginx/access.log"

    - index: "nginx-error-%{[agent.version]}-%{+yyyy.MM}"
      when.contains:
        log.file.path: "/var/log/nginx/error.log"

setup.ilm.enabled: false
setup.template.enabled: false

logging.level: info
logging.to_files: true

方法2:優雅
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]

processors:
  - drop_fields:
      fields: ["ecs","log"]

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "nginx-access-%{[agent.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "access"

    - index: "nginx-error-%{[agent.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "error"

setup.ilm.enabled: false
setup.template.enabled: false

logging.level: info
logging.to_files: true

3.6 使用ES-pipeline轉換Nginx普通日誌

[命令]json

0.grok轉換語法:
127.0.0.1 							==> %{IP:clientip}
- 									==> -
- 									==> -
[08/Oct/2020:16:34:40 +0800] 		==> \\[%{HTTPDATE:nginx.access.time}\\]
"GET / HTTP/1.1" 					==> "%{DATA:nginx.access.info}"
200 								==> %{NUMBER:http.response.status_code:long} 
5 									==> %{NUMBER:http.response.body.bytes:long}
"-" 								==> "(-|%{DATA:http.request.referrer})"
"curl/7.29.0" 						==> "(-|%{DATA:user_agent.original})"
"-"									==> "(-|%{IP:clientip})"

1.修改nginx日誌爲普通格式
systemctl stop filebeat
> /var/log/nginx/access.log
vim /etc/nginx/nginx.conf
systemctl restart nginx
curl 127.0.0.1
cat /var/log/nginx/access.log

2.建立ES的pipeline
GET _ingest/pipeline
PUT  _ingest/pipeline/pipeline-nginx-access
{
  "description" : "nginx access log",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{IP:clientip} - - \\[%{HTTPDATE:nginx.access.time}\\] \"%{DATA:nginx.access.info}\" %{NUMBER:http.response.status_code:long} %{NUMBER:http.response.body.bytes:long} \"(-|%{DATA:http.request.referrer})\" \"(-|%{DATA:user_agent.original})\""]
      }
    },{
      "remove": {
        "field": "message"
      }
    }
  ]
}

3.修改filebeat配置文件
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  tags: ["access"]

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]

processors:
  - drop_fields:
      fields: ["ecs","log"]

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]

  pipelines:
    - pipeline: "pipeline-nginx-access"
      when.contains:
        tags: "access"

  indices:
    - index: "nginx-access-%{[agent.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "access"

    - index: "nginx-error-%{[agent.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "error"

setup.ilm.enabled: false
setup.template.enabled: false

logging.level: info
logging.to_files: true

3.7 收集tomcat的json日誌

[命令]vim

1.修改tomcat配置文件
[root@web01 ~]# /opt/tomcat/bin/shutdown.sh
[root@web01 ~]# vim /opt/tomcat/conf/server.xml 
	       pattern="{&quot;clientip&quot;:&quot;%h&quot;,&quot;ClientUser&quot;:&quot;%l&quot;,&quot;authenticated&quot;:&quot;%u&quot;,&quot;AccessTime&quot;:&quot;%t&quot;,&quot;method&quot;:&quot;%r&quot;,&quot;status&quot;:&quot;%s&quot;,&quot;SendBytes&quot;:&quot;%b&quot;,&quot;Query?string&quot;:&quot;%q&quot;,&quot;partner&quot;:&quot;%{Referer}i&quot;,&quot;AgentVersion&quot;:&quot;%{User-Agent}i&quot;}"/>



2.filebeat配置文件
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /opt/tomcat/logs/localhost_access_log.*.txt 
  json.keys_under_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "tomcat-%{[agent.version]}-%{+yyyy.MM}"

setup.ilm.enabled: false
setup.template.enabled: false

3.8 收集java多行日誌

[命令]tomcat

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/elasticsearch/elasticsearch.log

  multiline.pattern: ^\[
  multiline.negate: true
  multiline.match: after

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "es-%{[agent.version]}-%{+yyyy.MM}"

setup.ilm.enabled: false
setup.template.enabled: false
相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程