ELK安裝

時間 2019-11-24

標籤 elk 安裝简体版

原文原文鏈接

 官方elk介紹地址：
 
 https://demo.elastic.co

具體圖解：css

提早準備html

 yum install java-1.8.0-openjdk.x86_64 -y
 yum install nodejs npm openssl screen -y

elasticserarch的用法java

 es-head相關說明：
 Node.js     #es-head是用node.js開發的
 npm       #node.js的包管理工具
 npm install   #es-head項目裏面的依賴下載下來
 npm run start & #放在後臺啓動

安裝elasticsearchnode

 [root@oldboy ~]# rpm -ivh elasticsearch-6.6.0.rpm
 [root@oldboy ~]# vim /usr/lib/systemd/system/elasticsearch.service
 7 [Service]    在server段下寫入第8行數據
 8 LimitMEMLOCK=infinity
 [root@oldboy ~]# grep -v "^#" /etc/elasticsearch/elasticsearch.yml
 node.name: node-1
 path.data: /var/lib/elasticsearch
 path.logs: /var/log/elasticsearch
 bootstrap.mem ory_lock: true
 network.host: 10.0.0.11,127.0.0.1
 http.port: 9200
 [root@oldboy ~]# systemctl daemon-reload
 [root@oldboy ~]# systemctl start elasticsearch.service
 [root@oldboy ~]# systemctl enable elasticsearch.service
 [root@oldboy ~]# netstat -tanpl |grep 9200
 tcp6       0      0 10.0.0.11:9200          :::*                    LISTEN      16901/java
 tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      16901/java
 [root@oldboy ~]#curl 127.0.0.1:9200
 {
   "name" : "node-1",
   "cluster_name" : "elasticsearch",
   "cluster_uuid" : "OX-iOIDlS5-fbondE9rPWw",
   "version" : {
     "number" : "6.6.0",
     "build_flavor" : "default",
     "build_type" : "rpm",
     "build_hash" : "a9861f4",
     "build_date" : "2019-01-24T11:27:09.439740Z",
     "build_snapshot" : false,
     "lucene_version" : "7.6.0",
     "minimum_wire_compatibility_version" : "5.6.0",
     "minimum_index_compatibility_version" : "5.0.0"
   },
   "tagline" : "You Know, for Search"
 }

 tar xf elasticsearch-head.tar.gz
 cd elasticsearch-head/
 npm run start &


 
 修改es配置文件支持跨域

[root@oldboy elasticsearch-head]# vim /etc/elasticsearch/elasticsearch.yml
 最後兩行添加以下參數
 http.cors.enabled: true 
 http.cors.allow-origin: "*"

 重啓elasticsearch

[root@oldboy elasticsearch-head]# systemctl restart elasticsearch.service  
 [root@oldboy elasticsearch-head]# netstat -tanpl |grep 9200
 tcp6       0      0 10.0.0.11:9200          :::*                    LISTEN      17559/java
 tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      17559/java

頁面訪問：mysql

安裝kibananginx

 [root@oldboy ~]# rpm -ivh kibana-6.6.0-x86_64.rpm
 [root@oldboy ~]# grep "^[a-z]" /etc/kibana/kibana.yml
 server.port: 5601
 server.host: "10.0.0.11"
 elasticsearch.hosts: ["http://localhost:9200"]    ###elasticsearch裝在哪一臺就是用哪一臺的ip
 kibana.index: ".kibana"
 [root@oldboy ~]# systemctl start kibana.service
 [root@oldboy ~]# netstat -tanpl |grep 5601
 tcp        0      0 10.0.0.11:5601          0.0.0.0:*               LISTEN      17741/node

頁面訪問：web

安裝filebeatredis

[root@oldboy ~]# rpm -ivh filebeat-6.6.0-x86_64.rpm
 [root@oldboy elasticsearch-head]# rpm -qc filebeat
 /etc/filebeat/filebeat.yml
 /etc/filebeat/modules.d/apache2.yml.disabled
 /etc/filebeat/modules.d/auditd.yml.disabled
 /etc/filebeat/modules.d/elasticsearch.yml.disabled
 /etc/filebeat/modules.d/haproxy.yml.disabled
 /etc/filebeat/modules.d/icinga.yml.disabled
 /etc/filebeat/modules.d/iis.yml.disabled
 /etc/filebeat/modules.d/kafka.yml.disabled
 /etc/filebeat/modules.d/kibana.yml.disabled
 /etc/filebeat/modules.d/logstash.yml.disabled
 /etc/filebeat/modules.d/mongodb.yml.disabled
 /etc/filebeat/modules.d/mysql.yml.disabled
 /etc/filebeat/modules.d/nginx.yml.disabled
 /etc/filebeat/modules.d/osquery.yml.disabled
 /etc/filebeat/modules.d/postgresql.yml.disabled
 /etc/filebeat/modules.d/redis.yml.disabled
 /etc/filebeat/modules.d/suricata.yml.disabled
 /etc/filebeat/modules.d/system.yml.disabled
 /etc/filebeat/modules.d/traefik.yml.disabled
  修改配置文件
 [root@oldboy elasticsearch-head]# egrep -v "#|^$" /etc/filebeat/filebeat.yml
 filebeat.inputs:
 - type: log
   enabled: true    ####
   paths:
     - /var/log/nginx/access.log    ###
 filebeat.config.modules:
   path: ${path.config}/modules.d/*.yml
   reload.enabled: true
 setup.template.settings:
   index.number_of_shards: 3
 setup.kibana:
 output.elasticsearch:
   hosts: ["10.0.0.11:9200"]      ####
 processors:
   - add_host_metadata: ~
   - add_cloud_metadata: ~
 [root@oldboy elasticsearch-head]# systemctl start filebeat.service

頁面測試：sql

添加mongodb

[root@oldboy ~]# yum -y install httpd-tools nginx
 [root@oldboy ~]# ab -c 10 -n 100 10.0.0.11/      ###這個頁面存在
 This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
 Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
 Licensed to The Apache Software Foundation, http://www.apache.org/
 
 Benchmarking 10.0.0.11 (be patient).....done
 
 
 Server Software:        nginx/1.12.2
 Server Hostname:        10.0.0.11
 Server Port:            80
 
 Document Path:          /
 Document Length:        3700 bytes
 
 Concurrency Level:      10
 Time taken for tests:   0.011 seconds
 Complete requests:      100
 Failed requests:        0
 Write errors:           0
 Total transferred:      393400 bytes
 HTML transferred:       370000 bytes
 Requests per second:    9355.41 [#/sec] (mean)
 Time per request:       1.069 [ms] (mean)
 Time per request:       0.107 [ms] (mean, across all concurrent requests)
 Transfer rate:          35941.59 [Kbytes/sec] received
 
 Connection Times (ms)
               min  mean[+/-sd] median   max
 Connect:        0    0   0.1      0       0
 Processing:     0    1   0.4      1       2
 Waiting:        0    1   0.4      1       2
 Total:          1    1   0.4      1       2
 
 Percentage of the requests served within a certain time (ms)
   50%      1
   66%      1
   75%      1
   80%      1
   90%      2
   95%      2
   98%      2
   99%      2
  100%      2 (longest request)
   
  [root@oldboy ~]# ab -c 10 -n 100 10.0.0.11/test.html    這個頁面不存在
 This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
 Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
 Licensed to The Apache Software Foundation, http://www.apache.org/
 
 Benchmarking 10.0.0.11 (be patient).....done
 
 
 Server Software:        nginx/1.12.2
 Server Hostname:        10.0.0.11
 Server Port:            80
 
 Document Path:          /test.html
 Document Length:        3650 bytes
 
 Concurrency Level:      10
 Time taken for tests:   0.010 seconds
 Complete requests:      100
 Failed requests:        0
 Write errors:           0
 Non-2xx responses:      100
 Total transferred:      382300 bytes
 HTML transferred:       365000 bytes
 Requests per second:    10031.10 [#/sec] (mean)
 Time per request:       0.997 [ms] (mean)
 Time per request:       0.100 [ms] (mean, across all concurrent requests)
 Transfer rate:          37450.08 [Kbytes/sec] received
 
 Connection Times (ms)
               min  mean[+/-sd] median   max
 Connect:        0    0   0.1      0       0
 Processing:     0    1   0.3      1       1
 Waiting:        0    1   0.2      1       1
 Total:          1    1   0.3      1       1
 
 Percentage of the requests served within a certain time (ms)
   50%      1
   66%      1
   75%      1
   80%      1
   90%      1
   95%      1
   98%      1
   99%      1
  100%      1 (longest request)
查看訪問日誌
 Try 'tail --help' for more information.
 [root@oldboy ~]# tail -f /var/log/nginx/access.log
 10.0.0.11 - - [10/Jun/2019:11:47:40 +0800] "GET /test.html HTTP/1.0" 404 3650 "-" "ApacheBench/2.3" "-"
 10.0.0.11 - - [10/Jun/2019:11:47:40 +0800] "GET /test.html HTTP/1.0" 404 3650 "-" "ApacheBench/2.3" "-"
 10.0.0.11 - - [10/Jun/2019:11:47:40 +0800] "GET /test.html HTTP/1.0" 404 3650 "-" "ApacheBench/2.3" "-"
 10.0.0.11 - - [10/Jun/2019:11:47:40 +0800] "GET /test.html HTTP/1.0" 404 3650 "-" "ApacheBench/2.3" "-"
 10.0.0.11 - - [10/Jun/2019:11:47:40 +0800] "GET /test.html HTTP/1.0" 404 3650 "-" "ApacheBench/2.3" "-"
 10.0.0.11 - - [10/Jun/2019:11:47:40 +0800] "GET /test.html HTTP/1.0" 404 3650 "-" "ApacheBench/2.3" "-"
 10.0.0.11 - - [10/Jun/2019:11:47:40 +0800] "GET /test.html HTTP/1.0" 404 3650 "-" "ApacheBench/2.3" "-"
 10.0.0.11 - - [10/Jun/2019:11:47:40 +0800] "GET /test.html HTTP/1.0" 404 3650 "-" "ApacheBench/2.3" "-"
 10.0.0.11 - - [10/Jun/2019:11:47:40 +0800] "GET /test.html HTTP/1.0" 404 3650 "-" "ApacheBench/2.3" "-"
 10.0.0.11 - - [10/Jun/2019:11:47:40 +0800] "GET /test.html HTTP/1.0" 404 3650 "-" "ApacheBench/2.3" "-"

 [root@oldboy filebeat]# vi /etc/filebeat/filebeat.yml
 filebeat.inputs:
 - type: log
   enabled: true
   paths:
     - /var/log/nginx/access.log
 output.elasticsearch:
   hosts: ["10.0.0.11:9200"]
   index: "nginx-%{[beat.version]}-%{+yyyy.MM.dd}"
 setup.template.name: "nginx"
 setup.template.pattern: "nginx-*"
 setup.template.enabled: false
 setup.template.overwrite: true
 [root@oldboy filebeat]# systemctl restart filebeat.service

刷新nginx訪問頁面，以後看下面頁面，會出現nginx-6*開頭的監控項

根據index索引添加監控項

添加以後的監控圖像

這裏以message爲主添加監控篩選條件

如圖

nginx日誌轉換成json格式

 nginx轉換成json操做步驟：
 
 0.中止filebeat
 systemctl stop filebeat
 
 1.修改nginx日誌格式爲json
 vim /etc/nginx/nginx.conf
     log_format json '{ "time_local": "$time_local", '
                            '"remote_addr": "$remote_addr", '
                            '"referer": "$http_referer", '
                            '"request": "$request", '
                            '"status": $status, '
                            '"bytes": $body_bytes_sent, '
                            '"agent": "$http_user_agent", '
                            '"x_forwarded": "$http_x_forwarded_for", '
                            '"up_addr": "$upstream_addr",'
                            '"up_host": "$upstream_http_host",'
                            '"upstream_time": "$upstream_response_time",'
                            '"request_time": "$request_time"' '}';
 
 
     access_log  /var/log/nginx/access.log  json;
 
 2.清空原有nginx日誌
 >/var/log/nginx/access.log
 
 3.檢查nginx配置是否正確
 nginx -t
 
 4.重啓nginx並訪問
 systemctl restart nginx
 curl localhost/
 
 5.檢查日誌是否爲json
 tail -a /var/log/nginx/access.log
 
 6.修改filebeat配置文件，增長json配置
 vim /etc/filebeat/filebeat.yml 
 filebeat.inputs:
 - type: log
   enabled: true 
   paths:
     - /var/log/nginx/access.log
   json.keys_under_root: true
   json.overwrite_keys: true
 output.elasticsearch:
   hosts: ["10.0.0.101:9200"]
   index: "nginx-json-%{[beat.version]}-%{+yyyy.MM.dd}"
 setup.template.name: "nginx"
 setup.template.pattern: "nginx-*"
 setup.template.enabled: false
 setup.template.overwrite: true
 
 7.重啓filebeat
 systemctl restart filebeat
 
 8.es-head查看是否生成新的索引
 
 9.kiabina添加新生成的json的索引
 
 10.過濾展現

監控tomcat日誌

個人步驟：

 yum install tomcat tomcat-webapps tomcat-admin-webapps tomcat-docs-webapp tomcat-javadoc -y
 
 [root@oldboy opt]# vim /etc/tomcat/server.xml
 刪掉原來139行的數據，插入新的帶有json格式的數據
 138                prefix="localhost_access_log." suffix=".txt"
 139                pattern="{"clientip":"%h","ClientUser":&quo    t;%l","authenticated":"%u","AccessTime":"%t&quot    ;,"method":"%r","status":"%s","SendBytes&qu    ot;:"%b","Query?string":"%q","partner":"%{R    eferer}i","AgentVersion":"%{User-Agent}i"}"/>
 [root@oldboy filebeat]# systemctl start tomcat
   
 [root@oldboy filebeat]# vim  /etc/filebeat/filebeat.yml 
   filebeat.inputs:
 - type: log
   enabled: true
   paths:
     - /var/log/tomcat/localhost_access_log.*
   json.keys_under_root: true
   json.overwrite_keys: true
 
 output.elasticsearch:
   hosts: ["10.0.0.11:9200"]
   index: "tomcat-json-%{[beat.version]}-%{+yyyy.MM.dd}"
 setup.template.name: "tomcat"
 setup.template.pattern: "tomcat-*"
 setup.template.enabled: false
 setup.template.overwrite: true
 
 [root@oldboy tomcat]# systemctl start filebeat.service

老師的步驟

 TOCMCAT日誌收集步驟
 1.正常安裝配置tomcat
 yum install tomcat tomcat-webapps tomcat-admin-webapps tomcat-docs-webapp tomcat-javadoc -y
 2.啓動訪問tomcat測試
 systemctl start tomcat 
 3.tomcat日誌修改成json格式
 #中止tomcat
 systemctl stop tomcat
 #清空原有tomcat日誌
 >/var/log/tomcat/localhost_access_log.2019-06-10.txt
 #編輯tomcat配置文件139行，替換爲json格式
 vim  /etc/tomcat/server.xml
 
 pattern="{"clientip":"%h","ClientUser":"%l","authenticated":"%u","AccessTime":"%t","method":"%r","status":"%s","SendBytes":"%b","Query?string":"%q","partner":"%{Referer}i","AgentVersion":"%{User-Agent}i"}"/>
 
 #重啓tomcat
 systemctl restart tomcat
 
 #編輯filebeat配置文件
 filebeat.inputs:
 - type: log
   enabled: true 
   paths:
     - /var/log/tomcat/localhost_access_log.*
   json.keys_under_root: true
   json.overwrite_keys: true
 
 output.elasticsearch:
   hosts: ["10.0.0.101:9200"]
   index: "tomcat-json-%{[beat.version]}-%{+yyyy.MM.dd}"
 setup.template.name: "tomcat"
 setup.template.pattern: "tomcat-*"
 setup.template.enabled: false
 setup.template.overwrite: true
 
 #重啓filebeat
 systemctl restart filebeat

多訪問幾回tomcat界面，讓其產生訪問日誌

而後訪問elasticsearch

[root@oldboy /]# vim  /etc/filebeat/filebeat.yml
 filebeat.inputs:
 - type: log
   enabled: true 
   paths:
     - /root/elasticsearch.log   ####這裏的文件是老師發的
   multiline.pattern: '^\['
   multiline.negate: true
   multiline.match: after
 output.elasticsearch:
   hosts: ["10.0.0.101:9200"]
   index: "es-java-%{[beat.version]}-%{+yyyy.MM.dd}"
 setup.template.name: "es"
 setup.template.pattern: "es-*"
 setup.template.enabled: false
 setup.template.overwrite: true
 [root@oldboy ~]# systemctl restart filebeat.service

查看elasticsearch

操做步驟：

1.nginx修改成普通日誌格式
2.清空nginx日誌
3.修改filebeat配置文件，添加模塊相關參數
4.使用filebeat命令打開模塊開關
5.重啓filebeat
6.訪問nginx產生數據

注意：解決elasticsearch老是打不開網頁的問題

 systemctl stop elasticsearch
 systemctl stop kibana
 
 rm -rf /var/lib/elasticsearch/*
 rm -rf /var/lib/kibana/*
 
 systemctl start elasticsearch
 systemctl start kibana

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。