ELK——ElasticSearch、Logstash、Kibana、Beats
ELK
1、ELK介紹
需求背景：隨着業務發展愈來愈大，服務器會愈來愈多，那麼，各類日誌量（好比，訪問日誌、應用日誌、錯誤日誌等）會愈來愈多。 所以，開發人員排查問題，須要到服務器上查看日誌，很不方便。而運維人員也須要一些數據，因此也要到服務器分析日誌，很麻煩。java
對於日誌來講，最多見的需求就是收集、存儲、查詢、展現，開源社區正好有相對應的開源項目：logstash（收集）、elasticsearch（存儲+搜索）、kibana（展現），咱們將這三個組合起來的技術稱之爲ELKStack，因此說ELKStack指的是Elasticsearch、Logstash、Kibana技術棧的結合。node
ELK Stack （5.0版本以後）Elastic Stack == （ELK Stack + Beats）nginx
ELK Stack包含：ElasticSearch、Logstash、Kibanaweb
ElasticSearch是一個搜索引擎，用來搜索、分析、存儲日誌。它是分佈式的，也就是說能夠橫向擴容，能夠自動發現，索引自動分片，總之很強大。json
Logstash用來收集日誌，把日誌解析爲json格式交給ElasticSearch。bootstrap
Kibana是一個數據可視化組件，把處理後的結果經過web界面展現vim
Beats在這裏是一個輕量級日誌收集器，其實Beats家族有5個成員(不斷增長) 早期的ELK架構中使用Logstash收集、解析日誌，可是Logstash對內存、cpu、io等資源消耗比較高。相比 Logstash，Beats所佔系統的CPU和內存幾乎能夠忽略不計。 x-pack對Elastic Stack提供了安全、警報、監控、報表、圖表於一身的擴展包，是收費的；瀏覽器
ELK的架構圖安全
2、ELK安裝準備工做
構建集羣的機器角色的分配：ruby
IP hostname 節點定義 openjdk elasticsearch kibana logstash beats 192.168.112.150 ying04 主節點 安裝 安裝 安裝 —— —— 192.168.112.151 ying05 數據節點 安裝 安裝 —— 安裝 —— 192.168.112.152 ying06 數據節點 安裝 安裝 —— —— 安裝
把三臺主機的IP 域名增長到/etc/hosts
[root@ying04 ~]# vim /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.112.150 ying04 192.168.112.151 ying05 192.168.112.152 ying06
三臺機器須要安裝openjdk；
[root@ying04 ~]# java -version //查無openjdk -bash: java: 未找到命令 [root@ying06 ~]# yum install -y java-1.8.0-openjdk //安裝openjdk [root@ying04 ~]# java -version openjdk version "1.8.0_181" OpenJDK Runtime Environment (build 1.8.0_181-b13) OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
3、配置elasticsearch
三臺機器都安裝elasticsearch
按照官方文檔；建立建立yum源的elastic文件；而後就能夠yum安裝elasticsearch
[root@ying04 ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch //導入官方祕鑰 [root@ying04 ~]# vim /etc/yum.repos.d/elastic.repo //建立yum源的elastic文件 [elasticsearch-6.x] name=Elasticsearch repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md
[root@ying04 ~]# yum install -y elasticsearch ### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd sudo systemctl daemon-reload sudo systemctl enable elasticsearch.service ### You can start elasticsearch service by executing sudo systemctl start elasticsearch.service Created elasticsearch keystore in /etc/elasticsearch 驗證中 : elasticsearch-6.4.2-1.noarch 1/1 已安裝: elasticsearch.noarch 0:6.4.2-1 完畢！ [root@ying04 ~]# echo $? 0
ying04機器上的配置文件按以下編輯：
[root@ying04 ~]# vim /etc/elasticsearch/elasticsearch.yml # ---------------------------------- Cluster ----------------------------------- #cluster.name: my-application cluster.name: fengstory //集羣的名稱 # ------------------------------------ Node ------------------------------------ #node.name: node-1 node.name: ying04 //節點的名稱 #node.attr.rack: r1 node.master: true //此機器是master節點 node.date: false //此機器不是數據節點 # ---------------------------------- Network ----------------------------------- #network.host: 192.168.0.1 network.host: 192.168.112.150 //綁定的IP，意思在哪一個IP上監聽 #http.port: 9200 http.port: 9200 //端口 # --------------------------------- Discovery ---------------------------------- #discovery.zen.ping.unicast.hosts: ["host1", "host2"] discovery.zen.ping.unicast.hosts: ["feng04", "feng05", "feng06"] //參與集羣的角色
ying05機器上的配置文件按以下編輯：
[root@ying05 ~]# vim /etc/elasticsearch/elasticsearch.yml # ---------------------------------- Cluster ----------------------------------- #cluster.name: my-application cluster.name: fengstory //集羣的名稱 # ------------------------------------ Node ------------------------------------ #node.name: node-1 node.name: ying05 //節點的名稱 #node.attr.rack: r1 node.master: false //ying05不是master節點 node.date: true //ying05是數據節點，多覺得true # ---------------------------------- Network ----------------------------------- #network.host: 192.168.0.1 network.host: 192.168.112.151 //綁定的IP，意思在哪一個IP上監聽 #http.port: 9200 http.port: 9200 //端口 # --------------------------------- Discovery ---------------------------------- #discovery.zen.ping.unicast.hosts: ["host1", "host2"] discovery.zen.ping.unicast.hosts: ["feng04", "feng05", "feng06"] //參與集羣的角色
ying06機器上的配置文件按以下編輯：
[root@ying06 ~]# vim /etc/elasticsearch/elasticsearch.yml # ---------------------------------- Cluster ----------------------------------- #cluster.name: my-application cluster.name: fengstory //集羣的名稱 # ------------------------------------ Node ------------------------------------ #node.name: node-1 node.name: ying06 //節點的名稱 #node.attr.rack: r1 node.master: false //ying05不是master節點 node.date: true //ying05是數據節點，多覺得true # ---------------------------------- Network ----------------------------------- #network.host: 192.168.0.1 network.host: 192.168.112.152 //綁定的IP，意思在哪一個IP上監聽 #http.port: 9200 http.port: 9200 //端口 # --------------------------------- Discovery ---------------------------------- #discovery.zen.ping.unicast.hosts: ["host1", "host2"] discovery.zen.ping.unicast.hosts: ["feng04", "feng05", "feng06"] //參與集羣的角色
三臺機器都開啓elasticsearch服務
[root@ying04 ~]# systemctl start elasticsearch
此時安裝配置正確的狀況下，每臺機器其端口會出現9200和9300
[root@ying04 ~]# ps aux |grep elastic elastic+ 1163 79.7 28.6 1550360 538184 ? Ssl 17:05 0:03 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.uXgx3jDC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/elasticsearch/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=rpm -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet root 1207 0.0 0.0 112720 984 pts/0 R+ 17:05 0:00 grep --color=auto elastic [root@ying04 ~]# netstat -lnpt |grep java tcp6 0 0 192.168.112.150:9200 :::* LISTEN 1163/java tcp6 0 0 192.168.112.150:9300 :::* LISTEN 1163/java
4、curl查看elasticsearch
集羣健康檢查：
[root@ying04 ~]# curl '192.168.112.150:9200/_cluster/health?pretty'
{
"cluster_name" : "fengstory",
"status" : "green", //green 表示正確
"timed_out" : false, //未超時
"number_of_nodes" : 3, //節點3個
"number_of_data_nodes" : 2, //數據節點2個
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
查看集羣詳細信息：curl '192.168.112.150:9200/_cluster/state?pretty'
[root@ying04 ~]# curl '192.168.112.150:9200/_cluster/state?pretty' |head //集羣詳細信息，只顯示前10行
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{
"cluster_name" : "fengstory", //集羣名字
"compressed_size_in_bytes" : 9577, //文件太大，約幾千行
"cluster_uuid" : "5pI8vvn0RXWBmGmj7Lj54A",
"version" : 5,
"state_uuid" : "Mc803-QnRQ-pkw4UWC7Gqw",
"master_node" : "0nBvsj3DTTmYSdGyiI1obg",
"blocks" : { },
"nodes" : {
"-gLGPb6tTEecUPPhlUlUuA" : {
62 126k 62 81830 0 0 498k 0 --:--:-- --:--:-- --:--:-- 502k
curl: (23) Failed writing body (90 != 16384)
5、安裝kibana
注意：kibana只須要在ying04 （192.168112.150）上安裝
由於已經建立yum源，所以直接用yum安裝kibana
[root@ying04 ~]# yum install -y kibana
編輯kibana的配置文件：/etc/kibana/kibana.yml
[root@ying04 ~]# vim /etc/kibana/kibana.yml //添加如下配置；帶#的是系統舉例；爲了清晰規整，則須要配置的內容，寫在相應的說明下； #server.port: 5601 server.port: 5601 #server.host: "localhost" server.host: 192.168.112.150 //只監控主機 150 #elasticsearch.url: "http://localhost:9200" elasticsearch.url: "http://192.168.112.150:9200" #logging.dest: stdout logging.dest: /var/log/kibana.log //定義日誌的路徑
由於kibana的配置文件中，指定日誌的文件，所以須要建立此文件，並給予777權限；
[root@ying04 ~]# touch /var/log/kibana.log; chmod 777 /var/log/kibana.log [root@ying04 ~]# ls -l /var/log/kibana.log -rwxrwxrwx 1 root root 10075 10月 13 18:25 /var/log/kibana.log
啓動kibana服務，並查看是否有進程以及5601端口；
[root@ying04 ~]# systemctl start kibana [root@ying04 ~]# ps aux |grep kibana kibana 1968 25.2 6.8 1076360 128712 ? Rsl 18:24 0:06 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml root 1980 5.0 0.0 112720 984 pts/0 R+ 18:24 0:00 grep --color=auto kibana [root@ying04 ~]# netstat -lnpt Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 536/rpcbind tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 966/nginx: master p tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 820/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1031/master tcp 0 0 192.168.112.150:5601 0.0.0.0:* LISTEN 1968/node //出現5601端口 tcp6 0 0 :::111 :::* LISTEN 536/rpcbind tcp6 0 0 192.168.112.150:9200 :::* LISTEN 1870/java tcp6 0 0 192.168.112.150:9300 :::* LISTEN 1870/java tcp6 0 0 :::22 :::* LISTEN 820/sshd tcp6 0 0 ::1:25 :::* LISTEN 1031/master
此時在瀏覽器裏，訪問http://192.168.112.150:5601
可以顯示以上頁面，說明配置成功；
6、安裝logstash
只須要在ying05（192.168.112.151）機器上安裝；
[root@ying05 ~]# yum install -y logstash
在/etc/logstash/conf.d/目錄下，專門存放收集相關定義日誌的配置文件
[root@ying05 ~]# ls /etc/logstash/ conf.d jvm.options log4j2.properties logstash-sample.conf logstash.yml pipelines.yml startup.options [root@ying05 ~]# ls /etc/logstash/conf.d/ //自定義的配置文件，放在此目錄下，纔會被加載
先建立syslog.conf，專門來收集系統日誌；
[root@ying05 ~]# vim /etc/logstash/conf.d/syslog.conf
input { //輸入配置
syslog {
type => "system-syslog" //定義日誌類型
port => 10514 //定義端口
}
}
output { //輸出配置
stdout { //標準輸出
codec => rubydebug
}
}
檢測一下配置是否正確，顯示OK，則配置正確；
./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
釋義：
- --path.settings /etc/logstash/ ：指定logstash配置文件的目錄；系統會按照這個目錄尋找；
- -f /etc/logstash/conf.d/syslog.conf ：自定義配置與logstash相關的文件；
- --config.test_and_exit ：檢測此文件；若是不要exit，則不能自動退出，會直接啓動logstash；
[root@ying05 ~]# cd /usr/share/logstash/bin
[root@ying05 bin]# ls
benchmark.sh dependencies-report logstash logstash-keystore logstash.lib.sh logstash-plugin.bat pqrepair setup.bat
cpdump ingest-convert.sh logstash.bat logstash-keystore.bat logstash-plugin pqcheck ruby system-install
[root@ying05 bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2018-10-13T19:06:58,327][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"}
[2018-10-13T19:06:58,337][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/logstash/dead_letter_queue"}
[2018-10-13T19:06:58,942][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2018-10-13T19:07:01,595][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
修改此rsyslog的配置文件
[root@ying05 bin]# vim /etc/rsyslog.conf #### RULES #### *.* @@192.168.112.150:10514 //*.*，代表全部的日誌；
重啓rsyslog
[root@ying05 bin]# systemctl restart rsyslog
如今啓動logstash，此時會把 所定義的日誌所有打印到屏幕上，並且不會自動退出來；至關於開了另外一個終端
[root@ying05 bin]# ./logstash --path.settings /etc/logstash/ -f
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2018-10-13T22:41:00,835][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"5039884c-a106-4370-8bb3-fcab8227a8d6", :path=>"/var/lib/logstash/uuid"}
[2018-10-13T22:41:01,662][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.4.2"}
[2018-10-13T22:41:05,042][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-10-13T22:41:05,838][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x9b429e run>"}
[2018-10-13T22:41:06,101][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2018-10-13T22:41:06,179][INFO ][logstash.inputs.syslog ] Starting syslog udp listener {:address=>"0.0.0.0:10514"}
[2018-10-13T22:41:06,209][INFO ][logstash.inputs.syslog ] Starting syslog tcp listener {:address=>"0.0.0.0:10514"}
[2018-10-13T22:41:06,757][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-10-13T22:41:17,106][INFO ][logstash.inputs.syslog ] new connection {:client=>"192.168.112.151:60140"}
{
"logsource" => "ying05",
"message" => "DHCPDISCOVER on ens37 to 255.255.255.255 port 67 interval 19 (xid=0x3a663c52)\n",
"@timestamp" => 2018-10-13T14:41:16.000Z,
"severity_label" => "Informational",
"priority" => 30,
"severity" => 6,
"host" => "192.168.112.151",
"pid" => "2163",
"facility" => 3,
"program" => "dhclient",
"type" => "system-syslog",
"timestamp" => "Oct 13 22:41:16",
"facility_label" => "system",
"@version" => "1"
}
{
"logsource" => "ying05",
"message" => "<info> [1539441739.5305] device (ens37): state change: failed -> disconnected (reason 'none') [120 30 0]\n",
"@timestamp" => 2018-10-13T14:42:19.000Z,
"severity_label" => "Informational",
"priority" => 30,
"severity" => 6,
"host" => "192.168.112.151",
"pid" => "559",
"facility" => 3,
"program" => "NetworkManager",
"type" => "system-syslog",
"timestamp" => "Oct 13 22:42:19",
"facility_label" => "system",
"@version" => "1"
}
因爲此處是終端顯示，不可以退出；會不停的刷新日誌狀況；此時須要查看其主機的信息，須要複製一個終端（簡稱B終端）；
在B重點,查看端口，此時有10514；
[root@ying05 ~]# netstat -lnpt Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 550/rpcbind tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 930/nginx: master p tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 821/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1041/master tcp6 0 0 :::111 :::* LISTEN 550/rpcbind tcp6 0 0 192.168.112.151:9200 :::* LISTEN 1391/java tcp6 0 0 :::10514 :::* LISTEN 2137/java tcp6 0 0 192.168.112.151:9300 :::* LISTEN 1391/java tcp6 0 0 :::22 :::* LISTEN 821/sshd tcp6 0 0 ::1:25 :::* LISTEN 1041/master tcp6 0 0 127.0.0.1:9600 :::* LISTEN 2137/java
回到以前的終端（成爲A端口），Ctrl + C強制退出，此時再查看監聽端口，10514也不會存在；
^C[2018-10-13T23:52:23,187][WARN ][logstash.runner ] SIGINT received. Shutting down.
[2018-10-13T23:52:23,498][INFO ][logstash.inputs.syslog ] connection error: stream closed
[2018-10-13T23:52:23,651][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x671ad8a0 run>"}
[root@ying05 bin]# netstat -lnpt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 550/rpcbind
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 930/nginx: master p
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 821/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1041/master
tcp6 0 0 :::111 :::* LISTEN 550/rpcbind
tcp6 0 0 192.168.112.151:9200 :::* LISTEN 1391/java
tcp6 0 0 192.168.112.151:9300 :::* LISTEN 1391/java
tcp6 0 0 :::22 :::* LISTEN 821/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1041/master
此時在A端口上，再次啓動logstash；
[root@ying05 bin]# ./logstash --path.settings /etc/logstash/ -f //此啓動方式爲，前臺啓動
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2018-10-13T23:54:27,377][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.4.2"}
[2018-10-13T23:54:30,556][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-10-13T23:54:31,118][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x7bcbeded run>"}
[2018-10-13T23:54:31,182][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2018-10-13T23:54:31,217][INFO ][logstash.inputs.syslog ] Starting syslog udp listener {:address=>"0.0.0.0:10514"}
[2018-10-13T23:54:31,243][INFO ][logstash.inputs.syslog ] Starting syslog tcp listener {:address=>"0.0.0.0:10514"}
[2018-10-13T23:54:31,525][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-10-13T23:58:47,450][INFO ][logstash.inputs.syslog ] new connection {:client=>"192.168.112.151:60152"}
[2018-10-13T23:58:47,785][INFO ][logstash.inputs.syslog ] new connection {:client=>"192.168.112.151:60154"}
{
"facility" => 3,
"severity_label" => "Informational",
"program" => "systemd",
"timestamp" => "Oct 13 23:58:47",
"@timestamp" => 2018-10-13T15:58:47.000Z,
"type" => "system-syslog",
"logsource" => "ying05",
"message" => "Stopping System Logging Service...\n",
"severity" => 6,
"facility_label" => "system",
"priority" => 30,
"host" => "192.168.112.151",
"@version" => "1"
}
在B終段上，查看端口，一直沒有10514，以後出現，說明以前一直在加載，啓動後，就會監聽10514端口
[root@ying05 ~]# netstat -lnpt |grep 10514 [root@ying05 ~]# netstat -lnpt |grep 10514 [root@ying05 ~]# netstat -lnpt |grep 10514 [root@ying05 ~]# netstat -lnpt |grep 10514 tcp6 0 0 :::10514 :::* LISTEN 2535/java [root@ying05 ~]#
7、配置logstash
剛纔只是把日誌顯示在屏幕上，方便測試，如今須要把日誌輸入到elasticsearch;
現編輯配置文件syslog.conf
[root@ying05 bin]# vim /etc/logstash/conf.d/syslog.conf
input {
syslog {
type => "system-syslog"
port => 10514
}
}
output {
elasticsearch {
hosts => ["192.168.112.150:9200"] //指向master機器ying04，因爲是分佈式的能夠指向15一、152
index => "system-syslog-%{+YYYY.MM}" //定義日誌索引
}
}
檢測配置文件否正常；輸出OK則爲配置成功；
[root@ying05 bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties [2018-10-14T00:16:21,163][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK [2018-10-14T00:16:23,242][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
如今啓動服務
[root@ying05 bin]# systemctl start logstash
可是查看日誌，卻久久不可以被寫入，由於權限問題
[root@ying05 bin]# ls -l /var/log/logstash/logstash-plain.log -rw-r--r-- 1 root root 624 10月 14 00:16 /var/log/logstash/logstash-plain.log [root@ying05 bin]# chown logstash /var/log/logstash/logstash-plain.log [root@ying05 bin]# ls -l /var/log/logstash/logstash-plain.log -rw-r--r-- 1 logstash root 624 10月 14 00:16 /var/log/logstash/logstash-plain.log [root@ying05 bin]# ls -l /var/lib/logstash/ //這個也須要更改 總用量 4 drwxr-xr-x 2 root root 6 10月 13 19:06 dead_letter_queue drwxr-xr-x 2 root root 6 10月 13 19:06 queue -rw-r--r-- 1 root root 36 10月 13 22:41 uuid [root@ying05 bin]# chown -R logstash /var/lib/logstash/ [root@ying05 bin]# systemctl restart logstash
同時9600端口監聽，改成192.168.112.151
[root@ying05 bin]# vim /etc/logstash/logstash.yml # ------------ Metrics Settings -------------- # # Bind address for the metrics REST endpoint # # http.host: "127.0.0.1" http.host: "192.168.112.151" //添加主機IP #
此時已經出現10514端口和9600，說明啓動成功
[root@ying05 bin]# systemctl restart logstash //重啓logstash服務； [root@ying05 bin]# netstat -lnpt Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 550/rpcbind tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 930/nginx: master p tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 821/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1041/master tcp6 0 0 :::111 :::* LISTEN 550/rpcbind tcp6 0 0 192.168.112.151:9200 :::* LISTEN 1391/java tcp6 0 0 :::10514 :::* LISTEN 4828/java tcp6 0 0 192.168.112.151:9300 :::* LISTEN 1391/java tcp6 0 0 :::22 :::* LISTEN 821/sshd tcp6 0 0 ::1:25 :::* LISTEN 1041/master tcp6 0 0 192.168.112.151:9600 :::* LISTEN 4828/java
到master機器（ying04）上，查看logstash收集的日誌傳到可以傳到elasticsearch中；現查看其日誌索引，能夠看到system-syslog-2018.10
[root@ying04 ~]# curl '192.168.112.150:9200/_cat/indices?v' //看到索引，說明logstash與elasticsearch通訊正常 health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open system-syslog-2018.10 uP2TM4UFTdSx7fbvLD1IsQ 5 1 82 0 773.8kb 361.9kb
如今咱們從feng06（192.168.112.152）機器，由ssh登陸到feng05（192.168.112.151）機器，會產生日誌；
[root@ying06 ~]# ssh 192.168.112.151 The authenticity of host '192.168.112.151 (192.168.112.151)' can't be established. ECDSA key fingerprint is SHA256:ZQlXi+kieRwi2t64Yc5vUhPPWkMub8f0CBjnYRlX2Iw. ECDSA key fingerprint is MD5:ff:9f:37:87:81:89:fc:ed:af:c6:62:c6:32:53:7a:ad. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.112.151' (ECDSA) to the list of known hosts. root@192.168.112.151's password: Last login: Sun Oct 14 13:55:30 2018 from 192.168.112.1
在feng05機器上查看，這個時間段發生的日誌；
[root@ying05 ~]# less /var/log/messages ......截取14.25.04這個時間點 Oct 14 14:25:04 ying05 systemd: Started Session 14 of user root. Oct 14 14:25:04 ying05 systemd-logind: New session 14 of user root. Oct 14 14:25:04 ying05 systemd: Starting Session 14 of user root.
在瀏覽器上，刷新kibana；會出現日誌，就是虛擬機顯示的日誌；注意時間點的對照
8、收集nginx日誌
在/etc/logstash/conf.d/目錄下建立一個收集nginx的配置文件；
[root@ying05 ~]# cd /etc/logstash/conf.d/
[root@ying05 conf.d]# ls
syslog.conf
[root@ying05 conf.d]# vim nginx.conf //建立一個收集nginx的配置文件
input {
file {
path => "/tmp/elk_access.log" //產生日誌的路徑
start_position => "beginning" //從哪裏開始
type => "nginx" //類型標記
}
}
filter { //有grok過濾器進行解析字段
grok {
match => { "message" => "%{IPORHOST:http_host} %{IPORHOST:clientip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:response} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time:float}"}
}
geoip {
source => "clientip"
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["192.168.112.151:9200"] //主機ying05
index => "nginx-test-%{+YYYY.MM.dd}"
}
}
對此配置進行檢查；生成OK，就是正確的；
[root@ying05 conf]# cd /usr/share/logstash/bin [root@ying05 bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/nginx.conf --config.test_and_exit Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties [2018-10-15T08:31:42,427][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK [2018-10-15T08:31:47,080][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
須要你的機器上安裝nginx；下面得知，nginx已經運行，並且也在佔用80端口，OK；
[root@ying05 ~]# netstat -lnpt Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 550/rpcbind tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 930/nginx: master p tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 821/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1041/master tcp6 0 0 :::111 :::* LISTEN 550/rpcbind tcp6 0 0 192.168.112.151:9200 :::* LISTEN 1391/java tcp6 0 0 :::10514 :::* LISTEN 4828/java tcp6 0 0 192.168.112.151:9300 :::* LISTEN 1391/java tcp6 0 0 :::22 :::* LISTEN 821/sshd tcp6 0 0 ::1:25 :::* LISTEN 1041/master tcp6 0 0 192.168.112.151:9600 :::* LISTEN 4828/java
在nginx配置文件中，添加代理kibana（安裝在feng04的主機）的虛擬主機文件；
[root@ying05 ~]# cd /usr/local/nginx/conf/
[root@ying05 conf]# vim nginx.conf
server {
listen 80;
server_name elk.ying.com;
location / {
proxy_pass http://192.168.112.150:5601; //真實服務器
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
access_log /tmp/elk_access.log main2; //定義日誌格式爲main2
}
因爲定義的格式爲main2，所以須要在nginx配置文件中也須要 添加main2格式
[root@ying05 conf]# vim nginx.conf //跟上面爲同一個配置
......默認
log_format combined_realip '$remote_addr $http_x_forwarded_for [$time_local]'
' $host "$request_uri" $status'
' "$http_referer" "$http_user_agent"'; //此爲默認配置
...... 默認
log_format main2 '$http_host $remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$upstream_addr" $request_time'; //此爲main2格式定義
重啓nginx服務；
[root@ying05 conf]# /usr/local/nginx/sbin/nginx -t nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful [root@ying05 conf]# /usr/local/nginx/sbin/nginx -s reload
再到 C:\Windows\System32\drivers\etc目錄下，hosts下，配置定義的域名；
192.168.112.151 elk.ying.com
再到ying05上，重啓logstash 服務
[root@ying05 conf]# systemctl restart logstash
稍等片刻後，到ying04機器上，查看生成的索引 nginx-test-2018.10.15；
[root@ying04 ~]# curl '192.168.112.150:9200/_cat/indices?v' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open .kibana aO3JiaT_TKWt3OJhDjPOvg 1 0 3 0 17.8kb 17.8kb yellow open nginx-test-2018.10.15 taXOvQTyTFely-_oiU_Y2w 5 1 60572 0 6mb 6mb yellow open system-syslog-2018.10 uP2TM4UFTdSx7fbvLD1IsQ 5 1 69286 0 10.7mb 10.7mb
到瀏覽器上，輸入elk.ying.com；並點擊 index patterns
把nginx-test-2018.10.15 這個索引填入到圖示位置；
填寫好以後，點擊 creat，就能夠看到 success! 添加索引成功；點擊下一步；
此時點擊左側Management，會出現兩個索引，其中一個就是剛纔添加的 nginx-test-2018.10.15;
在ying05機器上查看/tmp/elk_access.log日誌，9.15分的日誌，與kibana顯示的對比；結果確定是同樣的；
[root@ying05 bin]# less /tmp/elk_access.log elk.ying.com 192.168.112.1 - - [15/Oct/2018:09:05:18 +0800] "POST /elasticsearch/_msearch HTTP/1.1" 200 5919 "http://elk.ying.com/app/kibana" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko" "192.168.112.150:5601" 1.287
請看下面的message；注意時間是否一致；測試成功；
9、使用beats採集日誌
目前還有一種日誌收集器beats；
- beats是輕量級採集日誌，耗用資源少，可擴展；
- logstash耗費資源；
先yum安裝filebeat
[root@ying06 ~]yum list |grep filebeat filebeat.x86_64 6.4.2-1 @elasticsearch-6.x filebeat.i686 6.4.2-1 elasticsearch-6.x [root@ying06 ~]yum install -y filebeat
按下面編輯filebeat.yml的配置文件，目的是爲了測試，此配置；
[root@ying06 ~]# vim /etc/filebeat/filebeat.yml
#=========================== Filebeat inputs =============================
# Change to true to enable this input configuration.
# enabled: false //默認範例
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
# - /var/log/*.log //默認範例
- /var/log/messages //指定日誌路徑
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
# Array of hosts to connect to.
# hosts: ["localhost:9200"] //範例
output.console:
enable: true
此時會顯示很對的日誌，不斷刷屏，此時顯示的日誌，就是"source":"/var/log/messages"
[root@ying06 ~]/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml
{"@timestamp":"2018-10-15T07:32:06.322Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.4.2"},"beat":{"version":"6.4.2","name":"ying06","hostname":"ying06"},"host":{"name":"ying06"},"source":"/var/log/messages","offset":1253647,"message":"Oct 15 15:32:04 ying06 NetworkManager[558]: \u003cwarn\u003e [1539588724.3946] device (ens37): Activation: failed for connection '有線鏈接 1'","prospector":{"type":"log"},"input":{"type":"log"}}
{"@timestamp":"2018-10-15T07:32:06.322Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.4.2"},"host":{"name":"ying06"},"source":"/var/log/messages","offset":1253784,"message":"Oct 15 15:32:04 ying06 NetworkManager[558]: \u003cinfo\u003e [1539588724.3958] device (ens37): state change: failed -\u003e disconnected (reason 'none') [120 30 0]","prospector":{"type":"log"},"input":{"type":"log"},"beat":{"name":"ying06","hostname":"ying06","version":"6.4.2"}}
^C[root@ying06 ~]#
此兩個message與上面採集的日誌，是對應的；
[root@ying06 ~]# less /var/log/messages Oct 15 15:32:04 ying06 NetworkManager[558]: <warn> [1539588724.3946] device (ens37): Activation: failed for connection '有線鏈接 1' Oct 15 15:32:04 ying06 NetworkManager[558]: <info> [1539588724.3958] device (ens37): state change: failed -> disconnected (reason 'none') [120 30 0]
如今只是把日誌顯示在屏幕上，如今須要把日誌，傳到elasticsearch，再經過kibana可視化顯示；
[root@ying06 ~]# vim /etc/filebeat/filebeat.yml
#=========================== Filebeat inputs =============================
# Paths that should be crawled and fetched. Glob based paths.
paths:
# - /var/log/*.log
- /var/log/elasticsearch/fengstory.log //如今收集的此日誌
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch: //輸出到elasticsearch中
# Array of hosts to connect to.
hosts: ["192.168.112.150:9200"] //指向feng04主機
#output.console: //以前測試的 不用加載
# enable: true
開啓filebeat服務，並查看其進程；
[root@ying06 ~]# systemctl start filebeat [root@ying06 ~]# ps aux|grep filebeat root 1599 0.0 0.8 309872 16528 ? Ssl 16:20 0:00 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat root 1704 0.0 0.0 112720 980 pts/0 R+ 16:47 0:00 grep --color=auto filebeat [root@ying06 ~]# netstat -lnpt Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 556/rpcbind tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 964/nginx: master p tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 827/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1034/master tcp6 0 0 :::111 :::* LISTEN 556/rpcbind tcp6 0 0 192.168.112.152:9200 :::* LISTEN 1711/java tcp6 0 0 192.168.112.152:9300 :::* LISTEN 1711/java tcp6 0 0 :::22 :::* LISTEN 827/sshd tcp6 0 0 ::1:25 :::* LISTEN 1034/master
在feng04（192.168.112.150）機器上,查看日誌是否到達elasticsearch中；出現 filebeat字樣，說明已經到達；
[root@ying04 ~]# curl '192.168.112.150:9200/_cat/indices?v' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open nginx-test-2018.10.15 taXOvQTyTFely-_oiU_Y2w 5 1 202961 0 36.1mb 18.5mb green open .kibana aO3JiaT_TKWt3OJhDjPOvg 1 1 4 0 80.3kb 40.1kb green open filebeat-6.4.2-2018.10.15 m7Biv3QMTXmRR5u-cxIAoQ 3 1 73 0 153.3kb 95.4kb green open system-syslog-2018.10 uP2TM4UFTdSx7fbvLD1IsQ 5 1 211675 0 41.9mb 21.4mb
feng04上端口也有5601,此時能夠在瀏覽器上查看；
[root@ying04 ~]# netstat -lnpt Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 543/rpcbind tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 907/nginx: master p tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 820/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1042/master tcp 0 0 192.168.112.150:5601 0.0.0.0:* LISTEN 1420/node tcp6 0 0 :::111 :::* LISTEN 543/rpcbind tcp6 0 0 192.168.112.150:9200 :::* LISTEN 1255/java tcp6 0 0 192.168.112.150:9300 :::* LISTEN 1255/java tcp6 0 0 :::22 :::* LISTEN 820/sshd tcp6 0 0 ::1:25 :::* LISTEN 1042/master
此時在瀏覽器訪問：elk.ying.com
建立成功後，就能夠看到日誌 /var/log/elasticsearch/fengstory.log
此時查看日誌來源，經過對比，能夠得出二者同樣
[root@ying06 ~]# less /var/log/elasticsearch/fengstory.log
[2018-10-15T16:49:08,548][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [ying06] publish_address {192.168.112.152:9200}, bound_addresses {192.168.112.152:9200}
[2018-10-15T16:49:08,548][INFO ][o.e.n.Node ] [ying06] started
相關文章
- 1. ELK（Elasticsearch、Logstash、Kibana）
- 2. ELK(elasticsearch+logstash+kibana)
- 3. rhel6.5-elk(elasticsearch、logstash、kibana)
- 4. CENTOS7 ELK 5.3.2 elasticsearch logstash kibana filebeat
- 5. ELK (Elasticsearch+Logstash+Kibana)部署
- 6. ELK（elasticsearch+logstash+kibana)安裝
- 7. 初識ELK(ElasticSearch-LogStash-Kibana)
- 8. Centos6.4 安裝elk (elasticsearch logstash kibana)
- 9. ELK搭建（filebeat、elasticsearch、logstash、kibana）
- 10. windows 安裝 ELK(Elasticsearch,Logstash,kibana)
- 更多相關文章...
相關標籤/搜索
-
elasticsearch+logstash+kibana
kibana+logstash+elasticsearch
elasticsearch+kibana+logstash+ik
elasticsearch+kibana+logstash
filebeat+logstash+elasticsearch+kibana
logstash&redis&elasticsearch&kibana
elasticsearch+kibana+logstash+filebeat
elasticsearch+logstash+kibana+redis
elasticsearch+logstash+filebeat+kibana
logstash+elasticsearch+kibana
日誌分析
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
歡迎關注本站公眾號,獲取更多信息
相關文章
- 1. ELK（Elasticsearch、Logstash、Kibana）
- 2. ELK(elasticsearch+logstash+kibana)
- 3. rhel6.5-elk(elasticsearch、logstash、kibana)
- 4. CENTOS7 ELK 5.3.2 elasticsearch logstash kibana filebeat
- 5. ELK (Elasticsearch+Logstash+Kibana)部署
- 6. ELK（elasticsearch+logstash+kibana)安裝
- 7. 初識ELK(ElasticSearch-LogStash-Kibana)
- 8. Centos6.4 安裝elk (elasticsearch logstash kibana)
- 9. ELK搭建（filebeat、elasticsearch、logstash、kibana）
- 10. windows 安裝 ELK(Elasticsearch,Logstash,kibana)