Top
NSD ARCHITECTURE DAY04
- 案例1:導入數據
- 案例2:綜合練習
1 案例1:導入數據
1.1 問題
本案例要求批量導入數據:css
1.2 步驟
實現此案例須要按照以下步驟進行。html
步驟一:導入數據java
使用POST方式批量導入數據,數據格式爲json,url 編碼使用data-binary導入含有index配置的json文件git
- [root@room9pc01 ~]# scp /var/ftp/elk
2)使用GET查詢結果github
- [root@kibana ~]# curl -XGET 'http://192.168.1.61:9200/_mget?pretty' -d '{
- "docs":[
- {
- "_index":"shakespeare",
- "_type:":"act",
- "_id":0
- },
- {
- "_index":"shakespeare",
- "_type:":"line",
- "_id":0
- },
- {
- "_index":"xixi",
- "_type:":"haha",
- "_id":25
- }
- ]
- }'
- {
- "docs" : [ {
- "_index" : "shakespeare",
- "_type" : "act",
- "_id" : "0",
- "_version" : 1,
- "found" : true,
- "_source" : {
- "line_id" : 1,
- "play_name" : "Henry IV",
- "speech_number" : "",
- "line_number" : "",
- "speaker" : "",
- "text_entry" : "ACT I"
- }
- }, {
- "_index" : "shakespeare",
- "_type" : "act",
- "_id" : "0",
- "_version" : 1,
- "found" : true,
- "_source" : {
- "line_id" : 1,
- "play_name" : "Henry IV",
- "speech_number" : "",
- "line_number" : "",
- "speaker" : "",
- "text_entry" : "ACT I"
- }
- }, {
- "_index" : "xixi",
- "_type" : "haha",
- "_id" : "25",
- "_version" : 1,
- "found" : true,
- "_source" : {
- "account_number" : 25,
- "balance" : 40540,
- "firstname" : "Virginia",
- "lastname" : "Ayala",
- "age" : 39,
- "gender" : "F",
- "address" : "171 Putnam Avenue",
- "employer" : "Filodyne",
- "email" : "virginiaayala@filodyne.com",
- "city" : "Nicholson",
- "state" : "PA"
- }
- } ]
- }
步驟二:使用kibana查看數據是否導入成功正則表達式
1)數據導入之後查看logs是否導入成功,如圖-1所示:apache
- [root@se5 ~]# firefox http:
圖-1json
2)kibana導入數據,如圖-2所示:vim
- [root@kibana ~]# firefox http:
圖-2瀏覽器
3)成功建立會有logstash-*,如圖-3所示:
/
圖-3
4)導入成功以後選擇Discover,如圖-4所示:
圖-4
注意: 這裏沒有數據的緣由是導入日誌的時間段不對,默認配置是最近15分鐘,在這能夠修改一下時間來顯示
5)kibana修改時間,選擇Lsat 15 miuntes,如圖-5所示:
圖-5
6)選擇Absolute,如圖-6所示:
圖-6
7)選擇時間2015-5-15到2015-5-22,如圖-7所示:
圖-7
8)查看結果,如圖-8所示:
圖-8
9)除了柱狀圖,Kibana還支持不少種展現方式 ,如圖-9所示:
圖-9
10)作一個餅圖,選擇Pie chart,如圖-10所示:
圖-10
11)選擇from a new serach,如圖-11所示:
圖-11
12)選擇Spilt Slices,如圖-12所示:
圖-12
13)選擇Trems,Memary(也能夠選擇其餘的,這個不固定),如圖-13所示:
圖-13
14)結果,如圖-14所示:
圖-14
15)保存後能夠在Dashboard查看,如圖-15所示:
圖-15
2 案例2:綜合練習
2.1 問題
本案例要求:
- 練習插件
- 安裝一臺Apache服務並配置
- 使用filebeat收集Apache服務器的日誌
- 使用grok處理filebeat發送過來的日誌
- 存入elasticsearch
2.2 步驟
實現此案例須要按照以下步驟進行。
步驟一:安裝logstash
1)配置主機名,ip和yum源,配置/etc/hosts(請把se1-se5和kibana主機配置和logstash同樣的/etc/hosts)
- [root@logstash ~]# vim /etc/hosts
- 192.168.1.61 se1
- 192.168.1.62 se2
- 192.168.1.63 se3
- 192.168.1.64 se4
- 192.168.1.65 se5
- 192.168.1.66 kibana
- 192.168.1.67 logstash
2)安裝java-1.8.0-openjdk和logstash
- [root@logstash ~]# yum -y install java-1.8.0-openjdk
- [root@logstash ~]# yum -y install logstash
- [root@logstash ~]# java -version
- openjdk version "1.8.0_131"
- OpenJDK Runtime Environment (build 1.8.0_131-b12)
- OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)
- [root@logstash ~]# touch /etc/logstash/logstash.conf
- [root@logstash ~]# /opt/logstash/bin/logstash --version
- logstash 2.3.4
- [root@logstash ~]# /opt/logstash/bin/logstash-plugin list
- ...
- logstash-input-stdin
- logstash-output-stdout
- ...
- [root@logstash ~]# vim /etc/logstash/logstash.conf
- input{
- stdin{
- }
- }
- filter{
- }
- output{
- stdout{
- }
- }
- [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
- Settings: Default pipeline workers: 2
- Pipeline main started
- aa
- 2018-09-15T06:19:28.724Z logstash aa
備註:若不會寫配置文件能夠找幫助,插件文檔的位置:
https://github.com/logstash-plugins
3)codec類插件
- [root@logstash ~]# vim /etc/logstash/logstash.conf
- input{
- stdin{
- codec => "json"
- }
- }
- filter{
- }
- output{
- stdout{
- codec => "rubydebug"
- }
- }
- [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
- Settings: Default pipeline workers: 2
- Pipeline main started
- {"a":1}
- {
- "a" => 1,
- "@version" => "1",
- "@timestamp" => "2018-09-15T06:34:14.538Z",
- "host" => "logstash"
- }
4)file模塊插件
- [root@logstash ~]# vim /etc/logstash/logstash.conf
- input{
- file {
- path => [ "/tmp/a.log", "/var/tmp/b.log" ]
- sincedb_path => "/var/lib/logstash/sincedb"
- start_position => "beginning"
- type => "testlog"
- }
- }
- filter{
- }
- output{
- stdout{
- codec => "rubydebug"
- }
- }
- [root@logstash ~]# touch /tmp/a.log
- [root@logstash ~]# touch /var/tmp/b.log
- [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
另開一個終端:寫入數據
- [root@logstash ~]# echo a1 > /tmp/a.log
- [root@logstash ~]# echo b1 > /var/tmp/b.log
以前終端查看:
- [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
- Settings: Default pipeline workers: 2
- Pipeline main started
- {
- "message" => "a1",
- "@version" => "1",
- "@timestamp" => "2018-09-15T06:44:30.671Z",
- "path" => "/tmp/a.log",
- "host" => "logstash",
- "type" => "testlog"
- }
- {
- "message" => "b1",
- "@version" => "1",
- "@timestamp" => "2018-09-15T06:45:04.725Z",
- "path" => "/var/tmp/b.log",
- "host" => "logstash",
- "type" => "testlog"
- }
-
5)tcp、udp模塊插件
- [root@logstash ~]# vim /etc/logstash/logstash.conf
- input{
- file {
- path => [ "/tmp/a.log", "/var/tmp/b.log" ]
- sincedb_path => "/var/lib/logstash/sincedb"
- start_position => "beginning"
- type => "testlog"
- }
- tcp {
- host => "0.0.0.0"
- port => "8888"
- type => "tcplog"
- }
- udp {
- host => "0.0.0.0"
- port => "9999"
- type => "udplog"
- }
- }
- filter{
- }
- output{
- stdout{
- codec => "rubydebug"
- }
- }
- [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
另開一個終端查看,能夠看到端口
- [root@logstash tmp]# netstat -antup | grep 8888
- tcp6 0 0 :::8888 :::* LISTEN 22191/java
- [root@logstash tmp]# netstat -antup | grep 9999
- udp6 0 0 :::9999 :::* 22191/java
在另外一臺主機上寫一個腳本,發送數據,使啓動的logstash能夠接收到數據
- [root@se5 ~]# vim tcp.sh
- function sendmsg(){
- if [[ "$1" == "tcp" ]];then
- exec 9<>/dev/tcp/192.168.1.67/8888
- else
- exec 9<>/dev/udp/192.168.1.67/9999
- fi
- echo "$2" >&9
- exec 9<&-
- }
- [root@se5 ~]# . tcp.sh
- [root@se5 ~]# sendmsg udp "is tcp test"
- [root@se5 ~]# sendmsg udp "is tcp ss"
logstash主機查看結果
- [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
- Settings: Default pipeline workers: 2
- Pipeline main started
- {
- "message" => "is tcp test\n",
- "@version" => "1",
- "@timestamp" => "2018-09-15T07:45:00.638Z",
- "type" => "udplog",
- "host" => "192.168.1.65"
- }
- {
- "message" => "is tcp ss\n",
- "@version" => "1",
- "@timestamp" => "2018-09-15T07:45:08.897Z",
- "type" => "udplog",
- "host" => "192.168.1.65"
- }
6)syslog插件練習
- [root@logstash ~]# systemctl list-unit-files | grep syslog
- rsyslog.service enabled
- syslog.socket static
- [root@logstash ~]# vim /etc/logstash/logstash.conf
- start_position => "beginning"
- type => "testlog"
- }
- tcp {
- host => "0.0.0.0"
- port => "8888"
- type => "tcplog"
- }
- udp {
- host => "0.0.0.0"
- port => "9999"
- type => "udplog"
- }
- syslog {
- port => "514"
- type => "syslog"
- }
- }
- filter{
- }
- output{
- stdout{
- codec => "rubydebug"
- }
- }
另外一個終端查看是否檢測到514
- [root@logstash ~]# netstat -antup | grep 514
- tcp6 0 0 :::514 :::* LISTEN 22728/java
- udp6 0 0 :::514 :::* 22728/java
另外一臺主機上面操做,本地寫的日誌本地能夠查看
- [root@se5 ~]# vim /etc/rsyslog.conf
- local0.info /var/log/mylog
- [root@se5 ~]# systemctl restart rsyslog
- [root@se5 ~]# ll /var/log/mylog
- ls: cannot access /var/log/mylog: No such file or directory
- [root@se5 ~]# logger -p local0.info -t nsd "elk"
- [root@se5 ~]# ll /var/log/mylog
- -rw------- 1 root root 29 Sep 15 16:23 /var/log/mylog
- [root@se5 ~]# tail /var/log/mylog
- Sep 15 16:23:25 se5 nsd: elk
- [root@se5 ~]# tail /var/log/messages
- ...
- Sep 15 16:23:25 se5 nsd: elk
把本地的日誌發送給遠程1.67
- [root@se5 ~]# vim /etc/rsyslog.conf
- local0.info @192.168.1.67:514
- [root@se5 ~]# systemctl restart rsyslog
- [root@se5 ~]# logger -p local0.info -t nds "001 elk"
- [root@logstash bin]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
- {
- "message" => "001 elk",
- "@version" => "1",
- "@timestamp" => "2018-09-05T09:15:47.000Z",
- "type" => "syslog",
- "host" => "192.168.1.65",
- "priority" => 134,
- "timestamp" => "Jun 5 17:15:47",
- "logsource" => "kibana",
- "program" => "nds1801",
- "severity" => 6,
- "facility" => 16,
- "facility_label" => "local0",
- "severity_label" => "Informational"
- }
rsyslog.conf配置向遠程發送數據,遠程登錄1.65的時侯,把登錄日誌的信息(/var/log/secure)轉發給logstash即1.67這臺機器
- [root@se5 ~]# vim /etc/rsyslog.conf
- 57 authpriv.* @@192.168.1.67:514
- [root@se5 ~]# systemctl restart rsyslog
- [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
- Settings: Default pipeline workers: 2
- Pipeline main started
- {
- "message" => "Accepted password for root from 192.168.1.254 port 33780 ssh2\n",
- "@version" => "1",
- "@timestamp" => "2018-09-15T08:40:57.000Z",
- "type" => "syslog",
- "host" => "192.168.1.65",
- "priority" => 86,
- "timestamp" => "Sep 15 16:40:57",
- "logsource" => "se5",
- "program" => "sshd",
- "pid" => "26133",
- "severity" => 6,
- "facility" => 10,
- "facility_label" => "security/authorization",
- "severity_label" => "Informational"
- }
- {
- "message" => "pam_unix(sshd:session): session opened for user root by (uid=0)\n",
- "@version" => "1",
- "@timestamp" => "2018-09-15T08:40:57.000Z",
- "type" => "syslog",
- "host" => "192.168.1.65",
- "priority" => 86,
- "timestamp" => "Sep 15 16:40:57",
- "logsource" => "se5",
- "program" => "sshd",
- "pid" => "26133",
- "severity" => 6,
- "facility" => 10,
- "facility_label" => "security/authorization",
- "severity_label" => "Informational"
7)filter grok插件
grok插件:
解析各類非結構化的日誌數據插件
grok使用正則表達式把飛結構化的數據結構化
在分組匹配,正則表達式須要根據具體數據結構編寫
雖然編寫困難,但適用性極廣
- [root@logstash ~]# vim /etc/logstash/logstash.conf
- input{
- stdin{ codec => "json" }
- file {
- path => [ "/tmp/a.log", "/var/tmp/b.log" ]
- sincedb_path => "/var/lib/logstash/sincedb"
- start_position => "beginning"
- type => "testlog"
- }
- tcp {
- host => "0.0.0.0"
- port => "8888"
- type => "tcplog"
- }
- udp {
- host => "0.0.0.0"
- port => "9999"
- type => "udplog"
- }
- syslog {
- port => "514"
- type => "syslog"
- }
- }
- filter{
- grok{
- match => ["message", "(?<key>reg)"]
- }
- }
- output{
- stdout{
- codec => "rubydebug"
- }
- }
- [root@se5 ~]# yum -y install httpd
- [root@se5 ~]# systemctl restart httpd
- [root@se5 ~]# vim /var/log/httpd/access_log
- 192.168.1.254 - - [15/Sep/2018:18:25:46 +0800] "GET / HTTP/1.1" 403 4897 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0"
複製/var/log/httpd/access_log的日誌到logstash下的/tmp/a.log
- [root@logstash ~]# vim /tmp/a.log
- 192.168.1.254 - - [15/Sep/2018:18:25:46 +0800] "GET / HTTP/1.1" 403 4897 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0"
- [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
- Settings: Default pipeline workers: 2
- Pipeline main started
- {
- "message" => ".168.1.254 - - [15/Sep/2018:18:25:46 +0800] \"GET / HTTP/1.1\" 403 4897 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\"",
- "@version" => "1",
- "@timestamp" => "2018-09-15T10:26:51.335Z",
- "path" => "/tmp/a.log",
- "host" => "logstash",
- "type" => "testlog",
- "tags" => [
- [0] "_grokparsefailure"
- ]
- }
若要解決沒有解析的問題,一樣的方法把日誌複製到/tmp/a.log,logstash.conf配置文件裏面修改grok
查找正則宏路徑
- [root@logstash ~]# cd /opt/logstash/vendor/bundle/ \
- jruby/1.9/gems/logstash-patterns-core-2.0.5/patterns/
- [root@logstash ~]# vim grok-patterns
- COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
- [root@logstash ~]# vim /etc/logstash/logstash.conf
- ...
- filter{
- grok{
- match => ["message", "%{COMBINEDAPACHELOG}"]
- }
- }
- ...
解析出的結果
- [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
- Settings: Default pipeline workers: 2
- Pipeline main started
- {
- "message" => "192.168.1.254 - - [15/Sep/2018:18:25:46 +0800] \"GET /noindex/css/open-sans.css HTTP/1.1\" 200 5081 \"http://192.168.1.65/\" \"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\"",
- "@version" => "1",
- "@timestamp" => "2018-09-15T10:55:57.743Z",
- "path" => "/tmp/a.log",
- "host" => "logstash",
- "type" => "testlog",
- "clientip" => "192.168.1.254",
- "ident" => "-",
- "auth" => "-",
- "timestamp" => "15/Sep/2018:18:25:46 +0800",
- "verb" => "GET",
- "request" => "/noindex/css/open-sans.css",
- "httpversion" => "1.1",
- "response" => "200",
- "bytes" => "5081",
- "referrer" => "\"http://192.168.1.65/\"",
- "agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\""
- }
步驟二:安裝Apache服務,用filebeat收集Apache服務器的日誌,存入elasticsearch
1)在以前安裝了Apache的主機上面安裝filebeat
- [root@se5 ~]# yum -y install filebeat
- [root@se5 ~]# vim/etc/filebeat/filebeat.yml
- paths:
- - /var/log/httpd/access_log
- document_type: apachelog
- elasticsearch:
- hosts: ["localhost:9200"]
- logstash:
- hosts: ["192.168.1.67:5044"]
- [root@se5 ~]# systemctl start filebeat
- [root@logstash ~]# vim /etc/logstash/logstash.conf
- input{
- stdin{ codec => "json" }
- beats{
- port => 5044
- }
- file {
- path => [ "/tmp/a.log", "/var/tmp/b.log" ]
- sincedb_path => "/dev/null"
- start_position => "beginning"
- type => "testlog"
- }
- tcp {
- host => "0.0.0.0"
- port => "8888"
- type => "tcplog"
- }
- udp {
- host => "0.0.0.0"
- port => "9999"
- type => "udplog"
- }
- syslog {
- port => "514"
- type => "syslog"
- }
- }
- filter{
- if [type] == "apachelog"{
- grok{
- match => ["message", "%{COMBINEDAPACHELOG}"]
- }}
- }
- output{
- stdout{ codec => "rubydebug" }
- if [type] == "filelog"{
- elasticsearch {
- hosts => ["192.168.1.61:9200", "192.168.1.62:9200"]
- index => "filelog"
- flush_size => 2000
- idle_flush_time => 10
- }}
- }
- [root@logstash logstash]# /opt/logstash/bin/logstash \
- -f /etc/logstash/logstash.conf
打開另外一終端查看5044是否成功啓動
- [root@logstash ~]# netstat -antup | grep 5044
- tcp6 0 0 :::5044 :::* LISTEN 23776/java
- [root@se5 ~]# firefox 192.168.1.65
回到原來的終端,有數據
2)修改logstash.conf文件
- [root@logstash logstash]# vim logstash.conf
- ...
- output{
- stdout{ codec => "rubydebug" }
- if [type] == "apachelog"{
- elasticsearch {
- hosts => ["192.168.1.61:9200", "192.168.1.62:9200"]
- index => "apachelog"
- flush_size => 2000
- idle_flush_time => 10
- }}
- }
瀏覽器訪問Elasticsearch,有apachelog,如圖-16所示:
圖-16