ELKStack之極速入門（上）
#ELKStack之極速入門（上）css
連接：https://pan.baidu.com/s/1V2aYpB86ZzxL21Hf-AF1rA 提取碼：7izv 複製這段內容後打開百度網盤手機App，操做更方便哦html
##1. EKL介紹 ###1.1 需求背景java
- 通常咱們須要進行日誌分析場景：直接在日誌文件中 grep、awk 就能夠得到本身想要的信息。但在規模較大的場景中，此方法效率低下，面臨問題包括日誌量太大如何歸檔、文本搜索太慢怎麼辦、如何多維度查詢。須要集中化的日誌管理，全部服務器上的日誌收集彙總。常看法決思路是創建集中式日誌收集系統，將全部節點上的日誌統一收集，管理，訪問。
- 通常大型系統是一個分佈式部署的架構，不一樣的服務模塊部署在不一樣的服務器上，問題 出現時，大部分狀況須要根據問題暴露的關鍵信息，定位到具體的服務器和服務模塊， 構建一套集中式日誌系統，能夠提升定位問題的效率。
- 一個完整的集中式日誌系統，須要包含如下幾個主要特色：
(1)收集－可以採集多種來源的日誌數據
(2)傳輸－可以穩定的把日誌數據傳輸到中央系統
(3)存儲－如何存儲日誌數據
(4)分析－能夠支持 UI 分析
(5)警告－可以提供錯誤報告，監控機制- ELK提供了一整套解決方案，而且都是開源軟件，之間互相配合使用，完美銜接，高效的 知足了不少場合的應用。目前主流的一種日誌系統。 ###1.2 ELKStack介紹
- Logstash : 開源的服務器端數據處理管道，可以同時從多個來源採集數據，轉換數據， 而後將數據存儲到數據庫中。
- Elasticsearch：搜索，分析和存儲數據，分佈式數據庫。
- Kibana：數據可視化。
- Beats：輕量型採集器的平臺，從邊緣機器向Logstash和Elasticsearch發送數據。
- Filebeat：輕量型日誌採集器。
###1.3 ELK Stack架構
- Input：輸入，輸出數據能夠是Stdin，File，TCP，Redis，Syslog等
- Filter：過濾，將日誌格式化。有豐富的過濾插件：Grok正則捕獲，Date時間處 理，Json編碼解碼，Mutate數據修改等
- Output：輸出，輸出目標能夠是Stdout，File，TCP，Redis，ES等 ##2. elk的安裝與部署 ###2.1 準備工做 | 主機名 | 主機IP | | :--: | :--: | | ELK-kibana | 192.168.200.15 |
###2.2 安裝環境node
[root@ELK-Kibana ~]# cat /etc/redhat-release CentOS Linux release 7.6.1810 (Core) [root@ELK-Kibana ~]# uname -r 3.10.0-957.12.1.el7.x86_64 [root@ELK-Kibana ~]# [root@ELK-Kibana ~]# systemctl stop firewalld [root@ELK-Kibana ~]# systemctl disable firewalld [root@ELK-Kibana ~]# setenforce 0 setenforce: SELinux is disabled [root@ELK-Kibana ~]# ll /root/yuanmabao/ 總用量 462332 -rw-r--r-- 1 root root 9621331 8月 23 14:46 apache-tomcat-8.5.33.tar.gz -rw-r--r-- 1 root root 29050159 8月 23 14:46 elasticsearch-6.2.3.tar.gz -rw-r--r-- 1 root root 30948737 8月 23 14:46 GeoLite2-City.tar.gz -rw-r--r-- 1 root root 181238643 8月 23 14:46 jdk-8u60-linux-x64.tar.gz -rw-r--r-- 1 root root 83426328 8月 23 14:46 kibana-6.2.3-linux-x86_64.tar.gz -rw-r--r-- 1 root root 138221072 8月 23 14:46 logstash-6.2.3.tar.gz -rw-r--r-- 1 root root 910812 8月 23 14:46 nginx-1.10.2.tar.gz
###2.3 安裝jdk環境linux
[root@ELK-Kibana ~]# cd yuanmabao/ [root@ELK-Kibana yuanmabao]# tar xf jdk-8u60-linux-x64.tar.gz -C /usr/local/ [root@ELK-Kibana yuanmabao]# mv /usr/local/jdk1.8.0_60 /usr/local/jdk [root@ELK-Kibana yuanmabao]# ll -d /usr/local/jdk drwxr-xr-x 8 10 143 255 8月 5 2015 /usr/local/jdk
###2.4 配置java環境變量nginx
[root@ELK-Kibana yuanmabao]# sed -i.ori '$a export JAVA_HOME=/usr/local/jdk\nexport PATH=$PATH:$JAVA_HOME/bin\nexport CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$CLASSPATH' /etc/profile [root@ELK-Kibana yuanmabao]# tail -3 /etc/profile export JAVA_HOME=/usr/local/jdk export PATH=$PATH:$JAVA_HOME/bin export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$CLASSPATH [root@ELK-Kibana yuanmabao]# source /etc/profile [root@ELK-Kibana yuanmabao]# java -version java version "1.8.0_60" Java(TM) SE Runtime Environment (build 1.8.0_60-b27) Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
##3. kibana的安裝和啓動 ###3.1 kibana簡介web
- kibana的用途:主要用來展示數據，它自己不存儲數據
- kibana的下載地址:https://artifacts.elastic.co/downloads/kibana/kibana-6.2.3-linux-x86_64.tar.gz
###3.2 編譯部署kibana正則表達式
[root@ELK-Kibana yuanmabao]# useradd -s /sbin/nologin -M elk [root@ELK-Kibana yuanmabao]# id elk uid=1000(elk) gid=1000(elk) 組=1000(elk) [root@ELK-Kibana yuanmabao]# tar xf kibana-6.2.3-linux-x86_64.tar.gz -C /usr/local/ [root@ELK-Kibana yuanmabao]# mv /usr/local/kibana-6.2.3-linux-x86_64 /usr/local/kibana [root@ELK-Kibana yuanmabao]# ll -d /usr/local/kibana drwxrwxr-x 12 elk elk 232 3月 13 2018 /usr/local/kibana
###3.3 kibana配置文件修改redis
#修改前的配置文件
[root@ELK-Kibana yuanmabao]# cat -n /usr/local/kibana/config/kibana.yml | sed -n '2p;7p;21p;39p;40p'
2 #server.port: 5601
7 #server.host: "localhost"
21 #elasticsearch.url: "http://localhost:9200"
39 #elasticsearch.username: "user"
40 #elasticsearch.password: "pass"
#修改後的配置文件
[root@ELK-Kibana yuanmabao]# vim /usr/local/kibana/config/kibana.yml
[root@ELK-Kibana yuanmabao]# cat -n /usr/local/kibana/config/kibana.yml | sed -n '2p;7p;21p;39p;40p'
2 server.port: 5601 #暫時就先修改這兩行
7 server.host: "0.0.0.0" #暫時就先修改這兩行
21 #elasticsearch.url: "http://localhost:9200"
39 #elasticsearch.username: "user"
40 #elasticsearch.password: "pass"
###3.4 將kibana目錄的屬主，屬組修改成elk數據庫
[root@ELK-Kibana yuanmabao]# chown -R elk:elk /usr/local/kibana/
###3.5 新增啓動腳本
[root@ELK-Kibana yuanmabao]# vim /usr/local/kibana/bin/start.sh [root@ELK-Kibana yuanmabao]# cat /usr/local/kibana/bin/start.sh nohup /usr/local/kibana/bin/kibana >> /tmp/kibana.log 2>> /tmp/kibana.log & [root@ELK-Kibana yuanmabao]# chmod a+x /usr/local/kibana/bin/start.sh [root@ELK-Kibana yuanmabao]# ll -d /usr/local/kibana/bin/start.sh -rwxr-xr-x 1 root root 76 8月 23 15:19 /usr/local/kibana/bin/start.sh
###3.6 用普通用戶降權啓動kibana
[root@ELK-Kibana yuanmabao]# su -s /bin/bash elk '/usr/local/kibana/bin/start.sh' [root@ELK-Kibana yuanmabao]# ps -ef | grep elk | grep -v grep elk 7051 1 19 15:20 pts/0 00:00:02 /usr/local/kibana/bin/../node/bin/node --no-warnings /usr/local/kibana/bin/../src/cli
###3.7 查看錯誤日誌
[root@ELK-Kibana yuanmabao]# cat /tmp/kibana.log | grep warning | head
{"type":"log","@timestamp":"2019-08-23T07:20:53Z","tags":["warning","elasticsearch","admin"],"pid":7051,"message":"Unable to revive connection: http://localhost:9200/"}
{"type":"log","@timestamp":"2019-08-23T07:20:53Z","tags":["warning","elasticsearch","admin"],"pid":7051,"message":"No living connections"}
{"type":"log","@timestamp":"2019-08-23T07:20:56Z","tags":["warning","elasticsearch","admin"],"pid":7051,"message":"Unable to revive connection: http://localhost:9200/"}
{"type":"log","@timestamp":"2019-08-23T07:20:56Z","tags":["warning","elasticsearch","admin"],"pid":7051,"message":"No living connections"}
{"type":"log","@timestamp":"2019-08-23T07:20:58Z","tags":["warning","elasticsearch","admin"],"pid":7051,"message":"Unable to revive connection: http://localhost:9200/"}
{"type":"log","@timestamp":"2019-08-23T07:20:58Z","tags":["warning","elasticsearch","admin"],"pid":7051,"message":"No living connections"}
{"type":"log","@timestamp":"2019-08-23T07:21:01Z","tags":["warning","elasticsearch","admin"],"pid":7051,"message":"Unable to revive connection: http://localhost:9200/"}
{"type":"log","@timestamp":"2019-08-23T07:21:01Z","tags":["warning","elasticsearch","admin"],"pid":7051,"message":"No living connections"}
{"type":"log","@timestamp":"2019-08-23T07:21:03Z","tags":["warning","elasticsearch","admin"],"pid":7051,"message":"Unable to revive connection: http://localhost:9200/"}
{"type":"log","@timestamp":"2019-08-23T07:21:03Z","tags":["warning","elasticsearch","admin"],"pid":7051,"message":"No living connections"}
###3.8 訪問kibana http://192.168.200.15:5601
##4. 使用nginx來限制訪問kibana ###4.1 使用nginx轉發kibana
因爲kibana沒有權限控制，能夠藉助nginx來部署認證和進行ip控制 ####4.1.1 修改kibana的配置文件，改成監聽127.0.0.1
[root@ELK-Kibana yuanmabao]# vim /usr/local/kibana/config/kibana.yml [root@ELK-Kibana yuanmabao]# sed -n '7p' /usr/local/kibana/config/kibana.yml server.host: "127.0.0.1"
####4.1.2 重啓kibana服務
[root@ELK-Kibana yuanmabao]# ps -ef | grep elk | grep -v grep elk 7051 1 0 15:24 pts/0 00:00:10 /usr/local/kibana/bin/../node/bin/node --no-warnings /usr/local/kibana/bin/../src/cli [root@ELK-Kibana yuanmabao]# kill -9 7051 [root@ELK-Kibana yuanmabao]# su -s /bin/bash elk '/usr/local/kibana/bin/start.sh' [root@ELK-Kibana yuanmabao]# ps -ef | grep elk | grep -v grep elk 7143 1 69 16:25 pts/1 00:00:02 /usr/local/kibana/bin/../node/bin/node --no-warnings /usr/local/kibana/bin/../src/cli
###4.2 藉助nginx來限制訪問，控制源IP的訪問 ####4.2.1 編譯安裝nginx
#安裝依賴包 [root@ELK-Kibana yuanmabao]# yum -y install pcre-devel openssl-devel [root@ELK-Kibana yuanmabao]# rpm -qa pcre-devel openssl-devel pcre-devel-8.32-17.el7.x86_64 openssl-devel-1.0.2k-16.el7_6.1.x86_64
#編譯安裝nginx [root@ELK-Kibana yuanmabao]# useradd -s /sbin/nologin -M nginx [root@ELK-Kibana yuanmabao]# id nginx uid=1001(nginx) gid=1001(nginx) 組=1001(nginx) [root@ELK-Kibana yuanmabao]# tar xf nginx-1.10.2.tar.gz -C /usr/src/ [root@ELK-Kibana yuanmabao]# cd /usr/src/nginx-1.10.2/ [root@ELK-Kibana nginx-1.10.2]# ./configure --user=nginx --group=nginx --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module #如下省略若干。。。 [root@ELK-Kibana nginx-1.10.2]# make && make install #如下省略若干。。。
[root@ELK-Kibana nginx-1.10.2]# ln -s /usr/local/nginx/sbin/* /usr/local/sbin/ [root@ELK-Kibana nginx-1.10.2]# nginx -V nginx version: nginx/1.10.2 built by gcc 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) built with OpenSSL 1.0.2k-fips 26 Jan 2017 TLS SNI support enabled configure arguments: --user=nginx --group=nginx --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module
####4.2.2 編輯nginx配置文件，進行訪問控制，並啓動nginx
[root@ELK-Kibana nginx-1.10.2]# cd /usr/local/nginx/
[root@ELK-Kibana nginx]# cp conf/nginx.conf{,.bak}
[root@ELK-Kibana nginx]# vim conf/nginx.conf
[root@ELK-Kibana nginx]# cat conf/nginx.conf
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request"'
'$status $body_bytes_sent "$http_referer"'
'"$http_user_agent""$http_x_forwarded_for"';
sendfile on;
keepalive_timeout 65;
server {
listen 5609;
access_log /usr/local/nginx/logs/kibana_access.log main;
error_log /usr/local/nginx/logs/kibana_error.log error;
location / {
allow 192.168.200.1;
deny all;
proxy_pass http://127.0.0.1:5601;
}
}
}
[root@ELK-Kibana nginx]# nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
[root@ELK-Kibana nginx]# nginx
[root@ELK-Kibana nginx]# netstat -antup | grep nginx
tcp 0 0 0.0.0.0:5609 0.0.0.0:* LISTEN 9709/nginx: master
###4.3 訪問nginx監聽端口5609:http://192.168.200.15:5609
###4.4 若是訪問IP常常改變，受權就會很麻煩。能夠利用nginx的訪問驗證功能來認證訪問 ####4.4.1 修改nginx配置文件
[root@ELK-Kibana nginx]# vim conf/nginx.conf
[root@ELK-Kibana nginx]# cat conf/nginx.conf
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request"'
'$status $body_bytes_sent "$http_referer"'
'"$http_user_agent""$http_x_forwarded_for"';
sendfile on;
keepalive_timeout 65;
server {
listen 5609;
access_log /usr/local/nginx/logs/kibana_access.log main;
error_log /usr/local/nginx/logs/kibana_error.log error;
location / {
auth_basic "elk auth"; #修改這兩行
auth_basic_user_file /usr/local/nginx/conf/htpasswd; #修改這兩行
proxy_pass http://127.0.0.1:5601;
}
}
}
[root@ELK-Kibana nginx]# nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
[root@ELK-Kibana nginx]# nginx -s reload
####4.4.2 配置nginx登陸密碼
[root@ELK-Kibana nginx]# yum -y install httpd [root@ELK-Kibana nginx]# htpasswd -c /usr/local/nginx/conf/htpasswd admin #帳號 New password: Re-type new password: Adding password for user admin #密碼
####4.4.3 再次訪問nginx網址:http://192.168.200.15:5609
##5. elasticsearch的安裝和啓動
elasticsearch未安裝以前，kibana網頁上報錯，提示找不到elasticsearch
###5.1 elasticsearch簡介
- elasticsearch用途:elastic search主要用來存儲數據，供kibana調取並進行展示
- elasticsearch下載地址:https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.3.tar.gz ###5.2 解壓部署elasticsearch
[root@ELK-Kibana nginx]# cd /root/yuanmabao/ [root@ELK-Kibana yuanmabao]# tar xf elasticsearch-6.2.3.tar.gz -C /usr/local/ [root@ELK-Kibana yuanmabao]# mv /usr/local/elasticsearch-6.2.3 /usr/local/elasticsearch [root@ELK-Kibana yuanmabao]# ll -d /usr/local/elasticsearch drwxr-xr-x 8 root root 143 3月 13 2018 /usr/local/elasticsearch
###5.3 elasticsearch配置
#修改前的配置文件中的代碼
[root@ELK-Kibana yuanmabao]# cat -n /usr/local/elasticsearch/config/elasticsearch.yml | sed -n '33p;37p;55p;59p'
33 #path.data: /path/to/data
37 #path.logs: /path/to/logs
55 #network.host: 192.168.0.1
59 #http.port: 9200
#修改後的配置文件中的代碼
[root@ELK-Kibana yuanmabao]# vim /usr/local/elasticsearch/config/elasticsearch.yml
[root@ELK-Kibana yuanmabao]# cat -n /usr/local/elasticsearch/config/elasticsearch.yml | sed -n '33p;37p;55p;59p'
33 path.data: /usr/local/elasticsearch/data
37 path.logs: /usr/local/elasticsearch/logs
55 network.host: 127.0.0.1
59 http.port: 9200
###5.4 把elasticsearch目錄的用戶和屬主都更新爲elk
[root@ELK-Kibana yuanmabao]# chown -R elk:elk /usr/local/elasticsearch/
###5.5 更改jvm的內存限制（看我的配置,這裏未作修改）
[root@elk config]# cat jvm.options | sed -n "22p;23p" -Xms1g -Xmx1g
###5.6 編輯elasticsearch啓動腳本，並在後臺啓動
[root@ELK-Kibana yuanmabao]# cd /usr/local/elasticsearch/bin/ [root@ELK-Kibana bin]# vim start.sh [root@ELK-Kibana bin]# cat start.sh #/bin/bash #--Mr.yang-- /usr/local/elasticsearch/bin/elasticsearch -d >> /tmp/elasticsearch.log 2>> /tmp/elasticsearch.log [root@ELK-Kibana bin]# chmod a+x start.sh [root@ELK-Kibana bin]# ll -d start.sh -rwxr-xr-x 1 root root 124 8月 23 17:24 start.sh
###5.7 降權啓動elasticsearch
[root@ELK-Kibana bin]# su -s /bin/bash elk start.sh [root@ELK-Kibana bin]# ps -ef | grep elk | grep -v grep elk 7143 1 0 16:25 pts/1 00:00:14 /usr/local/kibana/bin/../node/bin/node --no-warnings /usr/local/kibana/bin/../src/cli elk 9852 1 36 17:25 pts/1 00:00:02 /usr/local/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.kNCsRtq1 -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/usr/local/elasticsearch -Des.path.conf=/usr/local/elasticsearch/config -cp /usr/local/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -d
###5.8 觀察日誌，看看kibana日誌還會不會報elasticsearch錯誤
[root@ELK-Kibana bin]# tail -f /tmp/kibana.log
{"type":"response","@timestamp":"2019-08-23T09:28:55Z","tags":[],"pid":7143,"method":"get","statusCode":200,"req":{"url":"/app/kibana","method":"get","headers":{"host":"127.0.0.1:5601","connection":"close","cache-control":"max-age=0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3","referer":"http://192.168.200.15:5609/app/kibana","accept-encoding":"gzip, deflate","accept-language":"zh-CN,zh;q=0.9"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.200.15:5609/app/kibana"},"res":{"statusCode":200,"responseTime":52,"contentLength":9},"message":"GET /app/kibana 200 52ms - 9.0B"}
##6. logstash的安裝和啓動 ###6.1 logstash簡介
- logstash用途：用來讀取日誌，正則分析日誌，發送給elasticsearch數據庫
- logstash下載地址：https://artifacts.elastic.co/downloads/logstash/logstash-6.2.3.tar.gz ###6.2 解壓部署logstash
[root@ELK-Kibana bin]# cd /root/yuanmabao/ [root@ELK-Kibana yuanmabao]# tar xf logstash-6.2.3.tar.gz -C /usr/local/ [root@ELK-Kibana yuanmabao]# mv /usr/local/logstash-6.2.3 /usr/local/logstash [root@ELK-Kibana yuanmabao]# ll -d /usr/local/logstash drwxr-xr-x 11 root root 237 8月 23 17:55 /usr/local/logstash
###6.3 logstash jvm配置簡介，加入內存限制(這裏看我的配置,這裏沒有修改)
[root@ELK-Kibana yuanmabao]# cat -n /usr/local/logstash/config/jvm.options | sed -n '6p;7p;'
6 -Xms1g
7 -Xmx1g
###6.4 新建logstash配置文件（默認沒有）
[root@ELK-Kibana yuanmabao]# vim /usr/local/logstash/config/logstash.conf
[root@ELK-Kibana yuanmabao]# cat /usr/local/logstash/config/logstash.conf
#/bin/bash
#--Mr.yang--
input {
file {
path => "/usr/local/nginx/logs/kibana_access.log" #讀取日誌路徑
}
}
output {
elasticsearch {
hosts => ["http://127.0.0.1:9200"] #保存日誌
}
}
###6.5 新建logstash的啓動腳本
[root@ELK-Kibana yuanmabao]# cd /usr/local/logstash/bin/ [root@ELK-Kibana bin]# vim start.sh [root@ELK-Kibana bin]# cat start.sh #/bin/bash #--Mr.yang-- nohup /usr/local/logstash/bin/logstash -f /usr/local/logstash/config/logstash.conf >> /tmp/logstash.log 2>>/tmp/logstash.log & [root@ELK-Kibana bin]# chmod a+x start.sh [root@ELK-Kibana bin]# ll -d start.sh -rwxr-xr-x 1 root root 152 8月 23 18:13 start.sh
###6.6 啓動logstash
#logstash並無監聽端口，所以不須要用elk用戶來啓動 [root@ELK-Kibana bin]# /usr/local/logstash/bin/start.sh [root@ELK-Kibana bin]# ps -ef | grep logstash | grep -v grep root 9964 1 23 18:15 pts/2 00:00:54 /usr/local/jdk/bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/local/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/local/logstash/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/local/logstash/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/local/logstash/logstash-core/lib/jars/google-java-format-1.5.jar:/usr/local/logstash/logstash-core/lib/jars/guava-22.0.jar:/usr/local/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/local/logstash/logstash-core/lib/jars/jackson-annotations-2.9.1.jar:/usr/local/logstash/logstash-core/lib/jars/jackson-core-2.9.1.jar:/usr/local/logstash/logstash-core/lib/jars/jackson-databind-2.9.1.jar:/usr/local/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.1.jar:/usr/local/logstash/logstash-core/lib/jars/janino-3.0.8.jar:/usr/local/logstash/logstash-core/lib/jars/javac-shaded-9-dev-r4023-3.jar:/usr/local/logstash/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar:/usr/local/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/local/logstash/logstash-core/lib/jars/log4j-api-2.9.1.jar:/usr/local/logstash/logstash-core/lib/jars/log4j-core-2.9.1.jar:/usr/local/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar:/usr/local/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/local/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash -f /usr/local/logstash/config/logstash.conf
特別提示:logstash啓動的比較慢，須要多等一下子。若是在kibana的Discover裏能看到添加索引就說明logstash啓動好了
###6.7 在kibana上配置索引，展示獲取的kibana日誌數據
進行數據展示字段的篩選
對nginx的kibana_access.log進行數據追蹤，對比分析
[root@ELK-Kibana bin]# tail -f /usr/local/nginx/logs/kibana_access.log 192.168.200.1 - admin [23/Aug/2019:18:26:40 +0800] "GET /ui/fonts/open_sans/open_sans_v13_latin_700.woff2 HTTP/1.1"304 0 "http://192.168.200.15:5609/app/kibana""Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36""-" 192.168.200.1 - admin [23/Aug/2019:18:27:28 +0800] "GET /api/saved_objects/?type=index-pattern&fields=title&per_page=10000 HTTP/1.1"200 168 "http://192.168.200.15:5609/app/kibana""Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36""-" 192.168.200.1 - admin [23/Aug/2019:18:27:29 +0800] "GET /bundles/448c34a56d699c29117adc64c43affeb.woff2 HTTP/1.1"200 18028 "http://192.168.200.15:5609/bundles/commons.style.css?v=16602""Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36""-" 192.168.200.1 - admin [23/Aug/2019:18:27:29 +0800] "POST /elasticsearch/_msearch HTTP/1.1"200 3691 "http://192.168.200.15:5609/app/kibana""Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36""-" 192.168.200.1 - admin [23/Aug/2019:18:27:46 +0800] "GET /bundles/6199ab7146b1eb859db9ffd2b9b786d3.svg HTTP/1.1"304 0 "http://192.168.200.15:5609/bundles/kibana.style.css?v=16602""Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36""-" #數據是一致的
##7. logstash使用詳解
[root@ELK-Kibana bin]# ps -ef | grep logstash | grep -v grep root 9964 1 4 18:15 pts/2 00:01:27 /usr/local/jdk/bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/local/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/local/logstash/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/local/logstash/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/local/logstash/logstash-core/lib/jars/google-java-format-1.5.jar:/usr/local/logstash/logstash-core/lib/jars/guava-22.0.jar:/usr/local/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/local/logstash/logstash-core/lib/jars/jackson-annotations-2.9.1.jar:/usr/local/logstash/logstash-core/lib/jars/jackson-core-2.9.1.jar:/usr/local/logstash/logstash-core/lib/jars/jackson-databind-2.9.1.jar:/usr/local/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.1.jar:/usr/local/logstash/logstash-core/lib/jars/janino-3.0.8.jar:/usr/local/logstash/logstash-core/lib/jars/javac-shaded-9-dev-r4023-3.jar:/usr/local/logstash/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar:/usr/local/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/local/logstash/logstash-core/lib/jars/log4j-api-2.9.1.jar:/usr/local/logstash/logstash-core/lib/jars/log4j-core-2.9.1.jar:/usr/local/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar:/usr/local/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/local/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash -f /usr/local/logstash/config/logstash.conf [root@ELK-Kibana bin]# kill -9 9964
[root@ELK-Kibana bin]# /usr/local/logstash/bin/logstash -e "" #命令執行的時間有點長，請耐心等待(執行前須要關閉logstash服務)
Sending Logstash's logs to /usr/local/logstash/logs which is now configured via log4j2.properties
[2019-08-23T18:50:01,638][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/local/logstash/modules/fb_apache/configuration"}
[2019-08-23T18:50:01,665][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/local/logstash/modules/netflow/configuration"}
[2019-08-23T18:50:02,403][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-08-23T18:50:03,232][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.3"}
[2019-08-23T18:50:03,823][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-08-23T18:50:06,822][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-08-23T18:50:22,288][INFO ][logstash.pipeline ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x7c6ef79b run>"}
The stdin plugin is now waiting for input:
[2019-08-23T18:50:22,386][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}
welcome #在這裏輸入"welcome"
{
"type" => "stdin",
"@timestamp" => 2019-08-23T10:53:57.807Z,
"host" => "ELK-Kibana",
"message" => "welcome",
"@version" => "1"
}
###7.1 logstash工做原理
- Logstash使用管道方式進行日誌的蒐集處理和輸出。有點相似於管道命令xxx|ccc|ddd，xxx執行完了會執行ccc，而後執行ddd。
- 在logstash中，包括了三個階段：輸入input ---> 處理filter（不是必須的） ---> 輸出output
- 每一個階段都有不少的插件配合工做，好比file，elasticsearch，redis等
- 每一個階段也能夠指定多種方式，好比輸出既能夠輸出到elasticsearch中，也能夠指定到 stdout在控制檯打印。
- 因爲這種插件式的組織方式，使得logstash變得易於擴展和定製
###7.2 命令行中經常使用的參數 | 參數 | 用途 | | :--: | :--: | | -f | 經過這個命令能夠指定Logstash的配置文件，根據配置文件配置logstash | | -e | 後面跟着字符串，該字符串能夠被看成logstash的配置（若是是""則默認使用stdin 做爲默認輸入，stdout做爲默認輸出） | | -l | 日誌輸出的地址（默認就是stdout直接在控制檯中輸出） | | -t | 測試配置文件是否正確，而後退出。| ####7.3 配置文件說明
前面介紹過logstash基本上由三部分組成，input，output以及用戶須要才添加的 filter，所以標準的配置文件格式以下：
input {...}
filter {...}
output {...}
在每一個部分中，也能夠指定多個訪問方式，例如我想要指定兩個日誌來源文件，則能夠 這樣寫：
input {
file { path => "/var/log/messages" type => "syslog" }
file { path => "/var/log/apache/access.log" type => "apache" }
}
相似的，若是在filter中添加了多種處理規則，則按照它的順序----處理，可是有一些插 件並非線程安全的。
好比在filter中指定了兩個同樣的插件，這兩個任務並不能保證準確的按順序執行，所以 官方也推薦避免在filter中重複使用插件。 ##8. 利用logstash的正則進行日誌信息的抓取測試
###8.1 測試性抓取日誌字段
#logstash提取數據段配置文件模板詳解
[root@ELK-Kibana bin]# cd /usr/local/logstash/config/
[root@ELK-Kibana config]# vim logstash.conf
[root@elk config]# cat logstash.conf
#/bin/bash
#--Mr.yang--
input {
stdin{} #從標準輸入讀取數據
}
filter {
grok {
match => {
"message" => '(?<字段名>正則表達式).*'
}
}
}
output {
elasticsearch { #若是要輸入到elasticsearch裏，那麼須要註釋掉stdout{}
hosts => ["http://127.0.0.1:9200"]
}
stdout { #只將信息輸出到屏幕上
codec => rubydebug #用於正則提取測試，將正則抓取結果輸出到屏幕上
}
}
####8.1.1 修改logstash配置文件
[root@ELK-Kibana config]# vim logstash.conf
[root@ELK-Kibana config]# cat logstash.conf
#/bin/bash
#--Mr.yang--
input {
stdin{ }
}
filter {
grok {
match => {
"message" => '(?<mydate>[a-zA-Z]+ [0-9]+ [0-9:]+) (?<hostname >[a-zA-Z]+).*'
}
}
}
output {
elasticsearch {
hosts => ["http://127.0.0.1:9200"]
}
stdout {
codec => rubydebug
}
}
####8.1.2 爲了測試方便，咱們暫時交互式啓動logstash
[root@ELK-Kibana config]# /usr/local/logstash/bin/logstash -f /usr/local/logstash/config/logstash.conf
####8.1.3 輸出結果以下
#輸入內容 Aug 16 18:29:49 ELK systemd: Startup finished in 789ms (kernel) + 1.465s (initrd) +18.959s (userspace) = 21.214s.
{
"@version" => "1",
"message" => "Aug 16 18:29:49 ELK systemd: Startup finished in 789ms (kernel) + 1.465s (initrd) +18.959s (userspace) = 21.214s.",
"host" => "elk",
"hostname " => "ELK", #這就是抓取的字段
"@timestamp" => 2019-04-01T05:14:55.982Z,
"mydate" => "Aug 16 18:29:49" #這就是抓取的字段
}
###8.2 將抓取的字段信息輸入到elasticsearch並顯示在kibana上 ####8.2.1 修改logstash配置文件
[root@ELK-Kibana config]# vim logstash.conf
[root@ELK-Kibana config]# cat logstash.conf
#/bin/bash
#--Mr.yang--
input {
stdin{ }
}
filter {
grok {
match => {
"message" => '(?<mydate>[a-zA-Z]+ [0-9]+ [0-9:]+) (?<hostname >[a-zA-Z]+).*'
}
}
}
output {
elasticsearch {
hosts => ["http://127.0.0.1:9200"]
}
# stdout { #將這三行給註釋掉
# codec => rubydebug #將這三行給註釋掉
# } #將這三行給註釋掉
}
####8.2.2 交互式啓動logstash
[root@ELK-Kibana config]# /usr/local/logstash/bin/logstash -f /usr/local/logstash/config/logstash.conf #用於進行測試輸入的日誌內容 Aug 16 18:29:49 ELK systemd: Startup finished in 789ms (kernel) + 1.465s (initrd) + 18.959s (userspace) = 21.214s.
###9. ELK技術全量分析nginx日誌
logstash若是直接把一整行日誌直接發送給elasticsearch，kibana顯示出來就沒有什麼 意義，咱們須要提取本身想要的字段。假如說咱們想要提取響應碼，用戶訪問url，響應 時間等，就得依靠正則來提取。
#logstash提取數據段配置文件模板詳解
input { #日誌輸入來源函數
file {
path => "/usr/local/nginx/logs/kibana_access.log"
}
}
filter { #字段數據提取函數
grok {
match => {
"message" => '(?<字段名>正則表達式).*'
}
}
}
output { #數據輸出目的地函數
elasticsearch {
hosts => ["http://127.0.0.1:9200"]
}
}
###9.1 利用正則從message中提取kibana訪問日誌的IP地址 ####9.1.1 修改logstash配置文件
[root@ELK-Kibana config]# vim logstash.conf
[root@ELK-Kibana config]# cat logstash.conf
#/bin/bash
#--Mr.yang--
input {
file {
path => "/usr/local/nginx/logs/kibana_access.log"
}
}
filter {
grok {
match => {
"message" => '(?<IP>[0-9.]+) .*'
}
}
}
output {
elasticsearch {
hosts => ["http://127.0.0.1:9200"]
}
}
####9.1.2 交互式啓動logstash
[root@ELK-Kibana config]# /usr/local/logstash/bin/logstash -f /usr/local/logstash/config/logstash.conf
####9.1.3 檢查測試結果
###9.2 從message中提取kibana訪問日誌的time，url，返 回碼，字節大小 ####9.2.1 修改logstash配置文件
[root@ELK-Kibana config]# vim logstash.conf
[root@ELK-Kibana config]# cat logstash.conf
#/bin/bash
#--Mr.yang--
input {
file {
path => "/usr/local/nginx/logs/kibana_access.log"
}
}
filter {
grok {
match => {
"message" => '(?<IP>[0-9.]+) .*HTTP/[0-9.]+"(?<mark>[0-9]+) (?<size>[0-9]+)[ "]+(?<url>[a-zA-Z]+://[0-9.]+:[0-9]+/[a-zA-Z/]+)".*'
}
}
}
output {
elasticsearch {
hosts => ["http://127.0.0.1:9200"]
}
}
####9.2.2 交互式啓動logstash
[root@ELK-Kibana config]# /usr/local/logstash/bin/logstash -f /usr/local/logstash/config/logstash.conf
####9.2.3 檢查測試結果
相關文章
- 1. 企業級日誌分析平臺（八）：ELKStack之極速入門（上）
- 2. 【急速入門】Spring 極速入門
- 3. MyBatis 極速入門
- 4. springBoot極速入門
- 5. python3.8.1極速入門
- 6. Noark入門之極速體驗
- 7. CI 極速入門學習
- 8. Go 語言極速入門
- 9. Cocos Creator 極速入門
- 10. Dagger 2 極速入門
- 更多相關文章...
- • Memcached入門教程 - NoSQL教程
- • Neo4j數據庫入門教程 - NoSQL教程
- • YAML 入門教程
- • Java Agent入門實戰（一）-Instrumentation介紹與使用
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
歡迎關注本站公眾號,獲取更多信息
相關文章
- 1. 企業級日誌分析平臺（八）：ELKStack之極速入門（上）
- 2. 【急速入門】Spring 極速入門
- 3. MyBatis 極速入門
- 4. springBoot極速入門
- 5. python3.8.1極速入門
- 6. Noark入門之極速體驗
- 7. CI 極速入門學習
- 8. Go 語言極速入門
- 9. Cocos Creator 極速入門
- 10. Dagger 2 極速入門