elasticsearch+kibana+logstash搭建日誌監控系統

時間 2019-11-24

標籤 elasticsearch+kibana+logstash elasticsearch kibana logstash 搭建日誌監控系統欄目日誌分析简体版

原文原文鏈接

1. 下載php

https://www.elastic.co/downloads/kibanahtml

https://www.elastic.co/downloads/elasticsearchjson

https://www.elastic.co/downloads/logstashruby

2. 解壓到制定的文件夾下面並重命名服務器

mv elasticsearch-2.3.3 elasticsearchcurl

mv kibana-4.5.1 kibanaelasticsearch

mv logstash-2.3.2 logstashtcp

3. 安裝插件ide

$cd elasticsearch
head
$bin/plugin install mobz/elasticsearch-head
watcher
$bin/plugin install license
$bin/plugin install watcher
shield
$bin/plugin install shield

4. 配置shield插件測試

#建立管理員用戶
$bin/shield/esusers useradd es_admin -r admin
建立Logstash用戶
$bin/shield/esusers useradd logstashserver -r logstash
建立kibana用戶
$bin/shield/esusers useradd kibanaserver -r kibana4_server

5. 配置kibana

$cd cabana
$ vi config/kibana.yml
設置server.host #例如 "127.0.0.1"
設置elasticsearch.username # 例如： kibanaserver
設置elasticsearch.password
保存退出

6. 啓動

$cd elasticsearch
$bin/elasticsearch

$cd kibana
$bin/kibana

7. 測試

es: http://127.0.0.1:9200/_plugin/head/

kibana: http://127.0.0.1:5601

輸入管理員帳號和密碼 es_admin/password

若是一切正常。說明elasticsearch 和 kibana搭建成功

8. 配置logstash

cd logstash

(1) hello world

vi logstash-simple.conf

input {

stdin { }

}

output {

elasticsearch {

hosts => ["localhost:9200"]

user => "logstashserver"

password => "newpass"

}

stdout {

codec => rubydebug }

}

bin/logstash -f logstash-simple.conf

接着輸入Hello World

去http://127.0.0.1:9200/_plugin/head/能夠看到以logstash開頭的indices

(2) 複雜點的logstash配置

vi logstash-filter.conf

input { stdin { } }

filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }

date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] } }

output { elasticsearch {

hosts => ["localhost:9200"]

user => "logstashserver"

password => "newpass"

}

stdout { codec => rubydebug }

}

bin/logstash -f logstash-filter.conf

輸入如下

127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"

(3) syslog

vi logstash-syslog.conf

input {

tcp {

port => 5000

type => syslog

}

udp {

port => 5000

type => syslog

} }

filter {

if [type] == "syslog」

{grok {

match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }

add_field => [ "received_at", "%{@timestamp}" ]

add_field => [ "received_from", "%{host}" ] }

date { match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]

} } }

output { elasticsearch {

hosts => ["localhost:9200」]

user => "logstashserver"

password => "newpass"

}

stdout { codec => rubydebug } }

再另外打開一個終端

telnet localhost 5000

輸入hello logstash

(4) File Input

logstash-tutorial-dataset 文件下載連接

https://download.elastic.co/demos/logstash/gettingstarted/logstash-tutorial.log.gz

vi logstash-first.conf

input {

file {

path => "/Users/fdrong/LogProject/logstash-tutorial-dataset"

start_position => beginning

}

filter {

grok {

match => { "message" => "%{COMBINEDAPACHELOG}"}

}

geoip {

source => "clientip"

}

output {

elasticsearch {

hosts => ["localhost:9200"]

user => "logstashserver"

password => "newpass"

}

stdout {

codec => rubydebug }

}

測試如下語法是否有錯

bin/logstash -f logstash-first.conf —configtest

若是沒有錯誤

bin/logstash -f logstash-first.conf

若是看到控制檯打印解析後的日誌（JSON格式）說明配置成功

9. 配置Watcher

(1). add a condition that simply checks to see if the search input returned any hits

curl -u es_admin —XPUT 'http://localhost:9200/_watcher/watch/log_error_watch' -d '{

"trigger" : { "schedule" : { "interval" : "10s" } },

"input" : {

"search" : {

"request" : {

"indices" : [ "logs" ],

"body" : {

"query" : {

"match" : { "message": "error" }}}}}},

"condition" : {

"compare" : { "ctx.payload.hits.total" : { "gt" : 0 }}

}

(2) 觀察結果

curl —u es_admin -XGET 'http://localhost:9200/.watch_history*/_search?pretty' -d '{

"query" : {

"bool" : {

"must" : [

{ "match" : { "result.condition.met" : true }},

{ "range" : { "result.execution_time" : { "from" : "now-10s"}}}

]

}

（3）添加錯誤日誌

curl -u es_admin —XPOST 'http://localhost:9200/logs/event' -d '{

"timestamp" : "2015-05-17T18:12:07.613Z",

"request" : "GET index.html",

"status_code" : 404,

"message" : "Error: File not found"

}’

而後再用第二步的命令查看出現不少hints說明執行成功

（4）添加郵件提醒功能

a. 首先cd elasticsearch

vi config/elasticsearch.yml

在文件的最末尾添加發件服務器設置

watcher.actions.email.service.account:

account:

profile: qq

email_defaults:

from: '<xxxxxx@qq.com>'

smtp:

auth: true

starttls.enable: true

host: smtp.qq.com

port: 25

user: xxxxx

password: xxxxxx

b. 添加郵件提醒event

curl -u es_admin —XPUT 'http://localhost:9200/_watcher/watch/log_error_watch_email' -d '{

"trigger" : { "schedule" : { "interval" : "10s" } },

"input" : {

"search" : {

"request" : {

"indices" : [ "logs" ],

"body" : {

"query" : {

"match" : { "message": "error" }}}}}},

"condition" : {

"compare" : { "ctx.payload.hits.total" : { "gt" : 0 }}

}

"actions" : {

"email_administrator" : {

"throttle_period": "15m", #發送郵件間隔

"email" : {

"to" : "接收郵件地址",

"subject" : "found {{ctx.payload.hits.total}} errors in logs",

"body" : "Too many error in the system, see attached data",

"attachments" : {

"attached_data" : {

"data" : {

"format" : "json"

}

"priority" : "high"

}

40秒以後就會收到一個提醒郵件，提示發現錯誤日誌

（4）刪除

curl -XDELETE 'http://localhost:9200/_watcher/watch/log_error_watch'

curl -XDELETE 'http://localhost:9200/_watcher/watch/log_error_watch_email'

10. 相關連接

logstash document

https://www.elastic.co/guide/index.html

config

http://kibana.logstash.es/content/logstash/plugins/output/elasticsearch.html

elastic download

https://www.elastic.co/downloads

wacher

https://www.elastic.co/guide/en/watcher/current/watch-log-data.html#log-add-condition

shield

https://eligao.com/shield-on-elasticsearch/

http://blog.sina.com.cn/s/blog_8ea8e9d50102wudw.html http://blog.csdn.net/july_2/article/details/24481935

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。