zabbix-proxy及ELK

時間 2019-11-17

標籤 zabbix proxy elk 欄目 Zabbix 简体版

原文原文鏈接

一、添加tomcat監控模版php

yum install java-1.8.0-openjdk tomcat-webapps tomcat-admin-webapps tomcat-docs-webapp -y #在被監控節點安裝tomcat包html

與jmx接口通訊須要使用特定的客戶端，還須要在其餘節點或者主節點安裝一個特定的客戶端組件zabbix-gateway，這裏在主節點直接安裝上java

yum install zabbix-java-gateway -y 在主節點安裝node

systemctl start zabbix-java-gateway #啓動服務mysql

修改主節點配置文件，加入JavaGateway的地址端口及進程數nginx

修改tomcat配置文件支持jmxgit

CATALINA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.managemegithub

nt.jmxremote.port=12345 -Dcom.sun.management.jmxremote.ssh=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmweb

i.server.hostname=192.168.1.197"redis

添加jmx主機

添加模板

二、搭建zabbix-proxy，並測試

在代理端安裝zabbix-proxy包，mariadb-server存儲監控數據

vim /etc/my.cnf#在proxy節點修改配置數據庫文件

skip_name_resolve=ON

以後建立數據庫，受權帳號，導入表結構，

MariaDB [(none)]> create database zabbix character set utf8 collate utf8_bin;

Query OK, 1 row affected (0.01 sec)

MariaDB [(none)]> grant all privileges on zabbix.* to zabbix@'192.168.1.%' identified by '123456';

Query OK, 0 rows affected (0.00 sec)

zcat /usr/share/doc/zabbix-proxy-mysql-3.0.28/schema.sql.gz | mysql -uzabbix -h 192.168.1.197 -p zabbix

ServerActive=192.168.1.197 #修改agent端配置，更換server地址

Server=192.168.1.196,192.168.1.197

vim /etc/zabbix/zabbix_proxy.conf

Server=192.168.1.196

Hostname=node2

DBName=zabbix

DBUser=zabbix

DBPassword=123456

HeartbeatFrequency=60 #每60秒向主節點發送心跳

ConfigFrequency=5 #主節點每幾秒過來proxy拖去配置文件

DataSenderFrequency=1#每1秒向主節點發送監控項

主節點添加proxy

建立主機，將node1，和node2自身都加入node2的proxy代理統一來監控

添加模板

三、添加redis監控模版（自動發現的方式，一臺主機，多個redis端口）

1.安裝Zabbix Server端與數據庫

腳本內容：

# vim redis.sh

#!/bin/bash

redis_status(){

R_PORT=$1

R_COMMAND=$2

(echo -en "INFO \r\n";sleep 1;) | nc 127.0.0.1 "$R_PORT" > /usr/local/zabbix/redis_"$R_PORT".tmp

REDIS_STAT_VALUE=$(grep ""$R_COMMAND":" /usr/local/zabbix/redis_"$R_PORT".tmp | cut -d ':' -f2)

echo $REDIS_STAT_VALUE

}

help(){

echo "${0} + redis_status + PORT + COMMAND"

}

main(){

case $1 in

redis_status)

redis_status $2 $3

;;

help

;;

esac

}

main $1 $2 $3

Server端驗證腳本：

# bash redis.sh redis_status 6379 connected_clients

# zabbix_get -s 192.168.15.205 -p 10050 -k "redis_status[redis_status,6379,connected_clients]"

加入自動發現

定義條件

四、搭建ELK日誌收集系統

yum install java-1.8.0-openjdk-devel -y #elasticsearch用java開發，依賴於Jvm須要安裝jdk組件

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.12.rpm#下載官網製做好的rpm包

ansible all -m yum -a 'name=elasticsearch-5.6.12.rpm state=installed' #贊成安裝包

三臺主機搭建集羣，所有須要安裝上述elk安裝包

192.168.1.195 node1

192.168.1.196 node2

192.168.1.198 node3

vim /etc/elasticsearch/elasticsearch.yml #編輯配置文件

cluster.name: myels #配置集羣名稱

node.name: node1 #節點名稱

path.data: /els/data #存儲路徑

path.logs: /els/logs

network.host: 192.168.1.196 #對外通訊主機名

http.port: 9200#監聽端口

discovery.zen.ping.unicast.hosts: ["node1","node2", "node3"] #集羣節點發現能夠是ip也能夠是主機名

discovery.zen.minimum_master_nodes: 1 #內部協調時的主節點，對外els沒有主節點這一稱呼

#修改文件屬主

ansible all -m shell -a 'mkdir -p /els/{data,logs}'

ansible all -m shell -a 'chown elasticsearch:elasticsearch /els/*'

開啓啓動成功

9200位客戶端提供服務，9300位集羣內部端口

測試訪問

查詢節點

RESTful API:

curl -X<VERB> '<PROTOCOL>://<HOST>:<PORT>/<PATH>?<QUERY_STRING>' -d '<BODY>'

<BODY>：json格式的請求主體；

curl -XPUT http://node1:9200/myindex/students/1 -d ' #建立文檔

> {"name":"GUO","age":17,"kf":"ab"}'

{"_index":"myindex","_type":"students","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true}[root@centos7 els]#

[root@centos7 els]# curl http://node1:9200/myindex/students/1?pretty #請求數據

{

"_index" : "myindex",

"_type" : "students",

"_id" : "1",

"_version" : 1,

"found" : true,

"_source" : {

"name" : "GUO",

"age" : 17,

"kf" : "ab"

}

[root@centos7 els]#

/_search：搜索全部的索引和類型；

/INDEX_NAME/_search：搜索指定的單個索引；

/INDEX1,INDEX2/_search：搜索指定的多個索引；

/s*/_search：搜索全部以s開頭的索引；

/INDEX_NAME/TYPE_NAME/_search：搜索指定的單個索引的指定類型；

Logstash組件：

簡歷web服務。收集web服務日誌

安裝jdk，下載logstash包

wget https://artifacts.elastic.co/downloads/logstash/logstash-6.8.1.rpm #下載logstash包

yum install logstash-6.8.1.rpm #本地安裝

/usr/share/logstash/bin/logstash #將服務程序文件寫入path變量中

[root@node2 ~]# vim /etc/profile.d/logstash.sh

export PATH=$PATH:/usr/share/logstash/bin/

###添加配置文件，增長輸入輸出插件配置段，input段，output段，以及fiter段（過濾數據）

https://www.elastic.co/guide/en/logstash/5.6/index.html官方文檔說明

cd /etc/logstash/con.d/

[root@node2 conf.d]# vim ss.conf #添加一個簡單配置，從交互輸入到json格式輸出

input {

stdin {}

}

output {

stdout {

codec => rubydebug

}

須要給定配置文件路徑，我這裏在當前目錄因此不須要給出

以上格式就能夠保存至elasticsearch中

#修改配置文件從web日誌中加載信息

input {

file {

path => '["/var/log/httpd/access_log"]'

start_position => "beginning"

}

ouput {

stdout {

codec => rubydebug

}

數據獲取成功。接下來使用fiter字段作數據加工

使用grok插件

使用表達式匹配字段。使用系統自帶的匹配模式，大寫字母爲變量。表示證的表達式

%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}

less /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns #logstash系統自帶的表達式文件

配置示例，match能將message字段中使用某個模式將字段分段處理好

1.input { file { path => "/var/log/http.log" } } filter { grok { match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" } } }

2.input {

file {

path => ["/var/log/httpd/access_log"]

start_position => "beginning"

}

filter {

grok {

match => { "message" => "%{HTTPD_COMBINEDLOG}" }

}

output {

stdout {

codec => rubydebug

}

將message字段切分開，將結果加上一個key

filter {

grok {

match => { "message" => "%{HTTPD_COMBINEDLOG}" }

remove_field => " message"

}

加工數據字段後將message字段刪除掉在grok中加入remove_field => " message"

geoip插件。GeoIP過濾器根據來自Maxmind GeoLite2數據庫的數據添加有關IP地址地理位置的信息

wget https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz 下載maxmind數據庫在本地，

配置示例。放在filter字段中

geoop {

source => "clientip" #須要匹配的字段

target => "geoip" #將匹配到的字段替換成這裏的字段

database => "/etc/logstash/maxmind/geolite-city.mmdb" #mixmind數據庫位置mmdb爲特定壓縮格式

}

tar xf GeoLite2-City.tar.gz

mkdir /etc/logstash/maxmind/

[root@centos7 ~]# mv GeoLite2-City_20191015/ /etc/logstash/maxmind/

[root@centos7 ~]# cd /etc/logstash/maxmind/

[root@centos7 maxmind]# ln -s GeoLite2-City_20191015/GeoLite2-City.mmdb ./

將clientip替換成geoip。而且提供時區。城市。國家等等詳細信息

mutate插件：變異篩選器可以讓您對字段執行常規變異。您能夠重命名，刪除，替換和修改事件中的字段。

過濾器{ mutate { split => [ 「主機名」，「。] add_field => { 「 shortHostname」 => 「％{hostname [0]}」 } } mutate { 重命名=> [ 「 shortHostname」，「 hostname」 ] } }

filter { mutate { split => ["hostname", "."] add_field => { "shortHostname" => "%{hostname[0]}" } } mutate { rename => ["shortHostname", "hostname" ] }}

elasticsearch 插件

output { #將輸出從屏幕保存至elasticsearch

elasticsearch {

hosts => ["http://node1:9200","http://node2:9200"] #添加主機名

index => "logstash-%{+YYYY.MM.dd}" #添加索引信息

document_type => "httpd_access_logs"#添加類型描述

}

查看索引狀態

curl node2:9200/logstash-*/_search?q=157.84.* | jq . #搜索logstash開頭的索引，基於ip地址搜索

五、用filebeat收集系統日誌併發送到ELK，展現出來

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.8-x86_64.rpm #下載安裝filebeat組件。

vim filebeat.yml

- /var/log/httpd/access_log

hosts: ["192.168.1.197:5044"] #修改logstash主機地址，

修改logstash的配置文件，將輸入替換爲beats。端口改成5044，若是不在同一臺機器也可加hosts => "IPADDR"

input {

beats {

port => 5044

}

logstash -f beatels.conf #啓動logstash

啓動filebeat

經過redis作消息隊列服務

vim /etc/redis.conf#修改一下地址及認證口令

bind 0.0.0.0

requirepass 123456

修改filebeat文件

修改logstash入站插件爲redis

input {

redis {

host => "192.168.1.197"

port => 6379

password => "centos"

db => 0

key => "filebeat"

data_type => "list"

}

在咱們沒啓動logstash以前filebeat拿到的日誌文件會存在redis中，

啓動以後logstash會在redis拿去數據存放至elasticsearch中

elasticsearch的存儲的數據會不斷增長

數據將會被分片至其餘節點

表達式斷定字段

編輯filebeat配置文件。加入兩個字段

fields:

log_type: access

#- c:\programdata\elasticsearch\logs\*

- paths:

- /var/log/httpd/error_log

fields:

log_type: errors

修改logstash配置文件，將錯誤日誌和訪問日誌分開存放

input {

redis {

host => "192.168.1.197"

port => 6379

password => "centos"

db => 0

key => "filebeat"

data_type => "list"

}

filter {

if [fields][log_type] == "access"{

grok {

match => { "message" => "%{HTTPD_COMBINEDLOG}" }

remove_field => "message"

}

geoip {

source => "clientip"

target => "geoip"

database => "/etc/logstash/maxmind/GeoLite2-City.mmdb"

}

output {

if [fields][log_type] == "access" {

elasticsearch {

hosts => ["http://node1:9200/","http://node2:9200/","http://node3:9200"]

index => "logstash-%{+YYYY.MM.dd}"

document_type => "httpd_access_logs"

}

} else {

elasticsearch {

hosts => ["http://node1:9200/","http://node2:9200/","http://node3:9200"]

index => "logstash-%{+YYYY.MM.dd}"

document_type => "httpd_error_logs"

}

六、收集nginx，tomcat日誌，php錯誤日誌，並用kibana畫出php錯誤日誌圖

wget https://artifacts.elastic.co/downloads/kibana/kibana-5.6.8-x86_64.rpm#下載安裝kibana圖形界面

修改配置文件裏的基礎信息

1 nginx 日誌格式配置

[root@elk-5-10 config]# cd /usr/local/nginx/conf/

[root@elk-5-10 conf]# vi nginx.conf

log_format access '$http_host $remote_addr - $remote_user [$time_local] "$request" '

'$status $body_bytes_sent "$http_referer" '

'"$http_user_agent" $http_x_forwarded_for';

2 日誌格式數據樣品

2.1 訪問日誌：

ss00.xxxxxx.me 150.138.154.157 - - [25/Jul/2017:03:02:35 +0800] "GET /csm/7_527.html HTTP/1.1" 304 0 "http://www.twww.com/tetris/page/64000159042/?ad_id=62928537191&cid=62928889880&req_id=0" "Mozilla/5.0 (Linux; Android 6.0.1; Redmi 4X Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/55.0.2883.91 Mobile Safari/537.36" 183.204.183.118

2.2 錯誤日誌

2017/07/25 15:23:29 [error] 24881#0: *33 open() "/usr/local/nginx-1.12.0/html/favicon.ico" failed (2: No such file or directory), client: 192.168.1.103, server: www.zyb.com, request: "GET /favicon.ico HTTP/1.1", host: "www.zyb.com"

3 logstash 配置文件

input {

file {

type => "nginx-access"

path => "/data/weixin.sys.mingyaohui.com.log"

start_position => beginning

}

file {

type => "nginx-error"

path => "/data/nginx_error.log"

start_position => beginning

}

filter {

if [type] == "nginx-access" {

grok {

match => ["message","%{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth}

"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent} %{IPORHOST:forwordip}" ]}

}

} else if [type] == "nginx-error" {

grok {

match => [ "message" , "(?<timestamp>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME})

%{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:errormessage}(?:, client: (?<clientip>%{IP}|%{HOSTNAME}))(?:, server: %{IPORHOST:server}?)(?:, request: %{QS:request})?(?:, upstream: (?<upstream>\"%{URI}\"|%{QS}))?(?:, host: %{QS:request_host})?(?:, referrer: \"%{URI:referrer}\")?"]

}

# add geo-location info

geoip {

source => "clientip"

}

output {

elasticsearch {

hosts => ["10.0.0.10"]

index => "%{type}-%{+YYYY.MM.dd}"

}

4 kibana分析效果圖

參考資料