Logstash安裝和設置（圖文詳解）（多節點的ELK集羣安裝在一個節點就好） Elasticsearch-2.4.3的下載（圖文詳解） Elasticsearch-2.4.3的單節點安裝（多種方式圖

 

 

 

 

前提

Elasticsearch-2.4.3的下載（圖文詳解）

Elasticsearch-2.4.3的單節點安裝（多種方式圖文詳解）

Elasticsearch-2.4.3的3節點安裝（多種方式圖文詳解）

Logstash-2.4.1的下載（圖文詳解）

 

 

 　　　　Logstash是一個管理日誌和事件的工具。

 

 

 

　我這裏的機器集羣狀況分別是：

　　HadoopMaster（192.168.80.10）、HadoopSlave1（192.168.80.11）和HadoopSlave2（192.168.80.12）。

 

 

 

一、上傳logstash-2.4.1.tar.gz壓縮包

[hadoop@HadoopMaster app]$ ll
total 16832
drwxrwxr-x.  9 hadoop hadoop     4096 Feb 22 06:05 elasticsearch-2.4.3
-rw-r--r--.  1 hadoop hadoop   908862 Jan 10 11:38 elasticsearch-head-master.zip
-rw-r--r--.  1 hadoop hadoop  2228252 Jan 10 11:38 elasticsearch-kopf-master.zip
drwxr-xr-x.  2 hadoop hadoop     4096 Mar 27 01:36 filebeat-1.3.1-x86_64
drwxr-xr-x. 10 hadoop hadoop     4096 Oct 31 17:15 hadoop-2.6.0
drwxr-xr-x. 15 hadoop hadoop     4096 Nov 14  2014 hadoop-2.6.0-src
drwxrwxr-x.  8 hadoop hadoop     4096 Nov  2 18:20 hbase-1.2.3
drwxr-xr-x.  8 hadoop hadoop     4096 Apr 11  2015 jdk1.7.0_79
drwxrwxr-x. 11 hadoop hadoop     4096 Nov  4 23:24 kibana-4.6.3-linux-x86_64
-rw-r--r--.  1 hadoop hadoop 10162116 Mar 25 10:00 marvel-2.4.4.tar.gz
-rw-r--r--.  1 hadoop hadoop  2332033 Jan 16 17:25 shield-2.4.3.zip
drwxrwxr-x.  9 hadoop hadoop     4096 Feb 25 19:18 tomcat-7.0.73
-rw-r--r--.  1 hadoop hadoop  1556618 Jan 16 17:22 watcher-2.4.3.zip
drwxr-xr-x. 10 hadoop hadoop     4096 Nov  1 23:39 zookeeper-3.4.6
[hadoop@HadoopMaster app]$ rz

[hadoop@HadoopMaster app]$ ll
total 98864
drwxrwxr-x.  9 hadoop hadoop     4096 Feb 22 06:05 elasticsearch-2.4.3
-rw-r--r--.  1 hadoop hadoop   908862 Jan 10 11:38 elasticsearch-head-master.zip
-rw-r--r--.  1 hadoop hadoop  2228252 Jan 10 11:38 elasticsearch-kopf-master.zip
drwxr-xr-x.  2 hadoop hadoop     4096 Mar 27 01:36 filebeat-1.3.1-x86_64
drwxr-xr-x. 10 hadoop hadoop     4096 Oct 31 17:15 hadoop-2.6.0
drwxr-xr-x. 15 hadoop hadoop     4096 Nov 14  2014 hadoop-2.6.0-src
drwxrwxr-x.  8 hadoop hadoop     4096 Nov  2 18:20 hbase-1.2.3
drwxr-xr-x.  8 hadoop hadoop     4096 Apr 11  2015 jdk1.7.0_79
drwxrwxr-x. 11 hadoop hadoop     4096 Nov  4 23:24 kibana-4.6.3-linux-x86_64
-rw-r--r--.  1 hadoop hadoop 83999654 Feb 25 11:06 logstash-2.4.1.tar.gz
-rw-r--r--.  1 hadoop hadoop 10162116 Mar 25 10:00 marvel-2.4.4.tar.gz

-rw-r--r--. 1 hadoop hadoop 2332033 Jan 16 17:25 shield-2.4.3.zip
drwxrwxr-x. 9 hadoop hadoop 4096 Feb 25 19:18 tomcat-7.0.73
-rw-r--r--. 1 hadoop hadoop 1556618 Jan 16 17:22 watcher-2.4.3.zip
drwxr-xr-x. 10 hadoop hadoop 4096 Nov 1 23:39 zookeeper-3.4.6
[hadoop@HadoopMaster app]$

 

 

 

 

 

 二、解壓

[hadoop@HadoopMaster app]$ ll
total 98864
drwxrwxr-x.  9 hadoop hadoop     4096 Feb 22 06:05 elasticsearch-2.4.3
-rw-r--r--.  1 hadoop hadoop   908862 Jan 10 11:38 elasticsearch-head-master.zip
-rw-r--r--.  1 hadoop hadoop  2228252 Jan 10 11:38 elasticsearch-kopf-master.zip
drwxr-xr-x.  2 hadoop hadoop     4096 Mar 27 01:36 filebeat-1.3.1-x86_64
drwxr-xr-x. 10 hadoop hadoop     4096 Oct 31 17:15 hadoop-2.6.0
drwxr-xr-x. 15 hadoop hadoop     4096 Nov 14  2014 hadoop-2.6.0-src
drwxrwxr-x.  8 hadoop hadoop     4096 Nov  2 18:20 hbase-1.2.3
drwxr-xr-x.  8 hadoop hadoop     4096 Apr 11  2015 jdk1.7.0_79
drwxrwxr-x. 11 hadoop hadoop     4096 Nov  4 23:24 kibana-4.6.3-linux-x86_64
-rw-r--r--.  1 hadoop hadoop 83999654 Feb 25 11:06 logstash-2.4.1.tar.gz
-rw-r--r--.  1 hadoop hadoop 10162116 Mar 25 10:00 marvel-2.4.4.tar.gz
-rw-r--r--.  1 hadoop hadoop  2332033 Jan 16 17:25 shield-2.4.3.zip
drwxrwxr-x.  9 hadoop hadoop     4096 Feb 25 19:18 tomcat-7.0.73
-rw-r--r--.  1 hadoop hadoop  1556618 Jan 16 17:22 watcher-2.4.3.zip
drwxr-xr-x. 10 hadoop hadoop     4096 Nov  1 23:39 zookeeper-3.4.6
[hadoop@HadoopMaster app]$ tar -zxvf logstash-2.4.1.tar.gz 

 

 

 

 

 

第三步：刪除安裝包，並修改所屬組和用戶

 

-rw-r--r--.  1 hadoop hadoop  2228252 Jan 10 11:38 elasticsearch-kopf-master.zip
drwxr-xr-x.  2 hadoop hadoop     4096 Mar 27 01:36 filebeat-1.3.1-x86_64
drwxr-xr-x. 10 hadoop hadoop     4096 Oct 31 17:15 hadoop-2.6.0
drwxr-xr-x. 15 hadoop hadoop     4096 Nov 14  2014 hadoop-2.6.0-src
drwxrwxr-x.  8 hadoop hadoop     4096 Nov  2 18:20 hbase-1.2.3
drwxr-xr-x.  8 hadoop hadoop     4096 Apr 11  2015 jdk1.7.0_79
drwxrwxr-x. 11 hadoop hadoop     4096 Nov  4 23:24 kibana-4.6.3-linux-x86_64
drwxrwxr-x.  5 hadoop hadoop     4096 Mar 27 03:58 logstash-2.4.1
-rw-r--r--.  1 hadoop hadoop 83999654 Feb 25 11:06 logstash-2.4.1.tar.gz
-rw-r--r--.  1 hadoop hadoop 10162116 Mar 25 10:00 marvel-2.4.4.tar.gz
-rw-r--r--.  1 hadoop hadoop  2332033 Jan 16 17:25 shield-2.4.3.zip
drwxrwxr-x.  9 hadoop hadoop     4096 Feb 25 19:18 tomcat-7.0.73
-rw-r--r--.  1 hadoop hadoop  1556618 Jan 16 17:22 watcher-2.4.3.zip
drwxr-xr-x. 10 hadoop hadoop     4096 Nov  1 23:39 zookeeper-3.4.6
[hadoop@HadoopMaster app]$ rm logstash-2.4.1.tar.gz 
[hadoop@HadoopMaster app]$ ll
total 16836
drwxrwxr-x.  9 hadoop hadoop     4096 Feb 22 06:05 elasticsearch-2.4.3
-rw-r--r--.  1 hadoop hadoop   908862 Jan 10 11:38 elasticsearch-head-master.zip
-rw-r--r--.  1 hadoop hadoop  2228252 Jan 10 11:38 elasticsearch-kopf-master.zip
drwxr-xr-x.  2 hadoop hadoop     4096 Mar 27 01:36 filebeat-1.3.1-x86_64
drwxr-xr-x. 10 hadoop hadoop     4096 Oct 31 17:15 hadoop-2.6.0
drwxr-xr-x. 15 hadoop hadoop     4096 Nov 14  2014 hadoop-2.6.0-src
drwxrwxr-x.  8 hadoop hadoop     4096 Nov  2 18:20 hbase-1.2.3
drwxr-xr-x.  8 hadoop hadoop     4096 Apr 11  2015 jdk1.7.0_79
drwxrwxr-x. 11 hadoop hadoop     4096 Nov  4 23:24 kibana-4.6.3-linux-x86_64
drwxrwxr-x.  5 hadoop hadoop     4096 Mar 27 03:58 logstash-2.4.1
-rw-r--r--.  1 hadoop hadoop 10162116 Mar 25 10:00 marvel-2.4.4.tar.gz
-rw-r--r--.  1 hadoop hadoop  2332033 Jan 16 17:25 shield-2.4.3.zip
drwxrwxr-x.  9 hadoop hadoop     4096 Feb 25 19:18 tomcat-7.0.73
-rw-r--r--.  1 hadoop hadoop  1556618 Jan 16 17:22 watcher-2.4.3.zip
drwxr-xr-x. 10 hadoop hadoop     4096 Nov  1 23:39 zookeeper-3.4.6

 

 

 

 

第四步：認識目錄結構

[hadoop@HadoopMaster app]$ cd logstash-2.4.1/
[hadoop@HadoopMaster logstash-2.4.1]$ pwd
/home/hadoop/app/logstash-2.4.1
[hadoop@HadoopMaster logstash-2.4.1]$ ll
total 160
drwxrwxr-x. 2 hadoop hadoop   4096 Mar 27 03:58 bin
-rw-rw-r--. 1 hadoop hadoop 102879 Nov 14 10:04 CHANGELOG.md
-rw-rw-r--. 1 hadoop hadoop   2249 Nov 14 10:04 CONTRIBUTORS
-rw-rw-r--. 1 hadoop hadoop   5084 Nov 14 10:07 Gemfile
-rw-rw-r--. 1 hadoop hadoop  23015 Nov 14 10:04 Gemfile.jruby-1.9.lock
drwxrwxr-x. 4 hadoop hadoop   4096 Mar 27 03:58 lib
-rw-rw-r--. 1 hadoop hadoop    589 Nov 14 10:04 LICENSE
-rw-rw-r--. 1 hadoop hadoop    149 Nov 14 10:04 NOTICE.TXT
drwxrwxr-x. 4 hadoop hadoop   4096 Mar 27 03:58 vendor
[hadoop@HadoopMaster logstash-2.4.1]$ 

 

 

 

 　Filebeat啊，根據input來監控數據，根據output來使用數據！！！

 　對應於，Logstash啊，有input、filter和output。

 

 

 

最簡單的Logstash測試（即，輸入什麼，直接在console打印輸出）

 

[hadoop@HadoopMaster logstash-2.4.1]$ pwd
/home/hadoop/app/logstash-2.4.1
[hadoop@HadoopMaster logstash-2.4.1]$ ll
total 160
drwxrwxr-x. 2 hadoop hadoop   4096 Mar 27 03:58 bin
-rw-rw-r--. 1 hadoop hadoop 102879 Nov 14 10:04 CHANGELOG.md
-rw-rw-r--. 1 hadoop hadoop   2249 Nov 14 10:04 CONTRIBUTORS
-rw-rw-r--. 1 hadoop hadoop   5084 Nov 14 10:07 Gemfile
-rw-rw-r--. 1 hadoop hadoop  23015 Nov 14 10:04 Gemfile.jruby-1.9.lock
drwxrwxr-x. 4 hadoop hadoop   4096 Mar 27 03:58 lib
-rw-rw-r--. 1 hadoop hadoop    589 Nov 14 10:04 LICENSE
-rw-rw-r--. 1 hadoop hadoop    149 Nov 14 10:04 NOTICE.TXT
drwxrwxr-x. 4 hadoop hadoop   4096 Mar 27 03:58 vendor
[hadoop@HadoopMaster logstash-2.4.1]$ bin/logstash -e 'input { stdin { } } output { stdout {} }'
Settings: Default pipeline workers: 1
Pipeline main started
（輸入回車） 2017-03-26T21:01:02.849Z HadoopMaster （顯示回車） abcd 2017-03-26T21:01:10.559Z HadoopMaster abcd

　　以上是最簡單的Logstash測試（即，輸入什麼，直接在console打印輸出）。

 

 

^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>"main"}
Received shutdown signal, but pipeline is still waiting for in-flight events
to be processed. Sending another ^C will force quit Logstash, but this may cause
data loss. {:level=>:warn}
^CSIGINT received. Terminating immediately.. {:level=>:fatal}
[hadoop@HadoopMaster logstash-2.4.1]$ 

 　

 

 

 

 

　　Logstash能夠以指定某種格式來輸入。好比以下：

[hadoop@HadoopMaster logstash-2.4.1]$ pwd
/home/hadoop/app/logstash-2.4.1
[hadoop@HadoopMaster logstash-2.4.1]$ ll
total 160
drwxrwxr-x. 2 hadoop hadoop   4096 Mar 27 03:58 bin
-rw-rw-r--. 1 hadoop hadoop 102879 Nov 14 10:04 CHANGELOG.md
-rw-rw-r--. 1 hadoop hadoop   2249 Nov 14 10:04 CONTRIBUTORS
-rw-rw-r--. 1 hadoop hadoop   5084 Nov 14 10:07 Gemfile
-rw-rw-r--. 1 hadoop hadoop  23015 Nov 14 10:04 Gemfile.jruby-1.9.lock
drwxrwxr-x. 4 hadoop hadoop   4096 Mar 27 03:58 lib
-rw-rw-r--. 1 hadoop hadoop    589 Nov 14 10:04 LICENSE
-rw-rw-r--. 1 hadoop hadoop    149 Nov 14 10:04 NOTICE.TXT
drwxrwxr-x. 4 hadoop hadoop   4096 Mar 27 03:58 vendor
[hadoop@HadoopMaster logstash-2.4.1]$ bin/logstash -e 'input { stdin { } } output { stdout { codec => json} }'

Settings: Default pipeline workers: 1
Pipeline main started
{"message":"","@version":"1","@timestamp":"2017-03-26T21:13:09.879Z","host":"HadoopMaster"}hjjjk
{"message":"hjjjk","@version":"1","@timestamp":"2017-03-26T21:13:23.484Z","host":"HadoopMaster"}^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>"main"}
^CSIGINT received. Terminating immediately.. {:level=>:fatal}
[hadoop@HadoopMaster logstash-2.4.1]$ 

　　咱們能夠看到，咱們輸入什麼內容logstash按照某種格式輸出，其中-e參數參數容許Logstash直接經過命令行接受設置。這點尤爲快速的幫助咱們反覆的測試配置是否正確而不用寫配置文件。使用Ctrl + C命令能夠退出以前運行的Logstash。

 

 

 

 　　使用-e參數在命令行中指定配置是很經常使用的方式，不過若是須要配置更多設置則須要很長的內容。這種狀況，咱們首先建立一個簡單的配置文件，而且指定logstash使用這個配置文件。例如：在logstash安裝目錄下建立一個「基本配置」測試文件logstash-simple.conf。

　　Logstash使用-f參數替換命令行中的-e參數（既能夠寫到配置文件裏，爲了方便）

[hadoop@HadoopMaster logstash-2.4.1]$ pwd
/home/hadoop/app/logstash-2.4.1
[hadoop@HadoopMaster logstash-2.4.1]$ ll
total 160
drwxrwxr-x. 2 hadoop hadoop   4096 Mar 27 03:58 bin
-rw-rw-r--. 1 hadoop hadoop 102879 Nov 14 10:04 CHANGELOG.md
-rw-rw-r--. 1 hadoop hadoop   2249 Nov 14 10:04 CONTRIBUTORS
-rw-rw-r--. 1 hadoop hadoop   5084 Nov 14 10:07 Gemfile
-rw-rw-r--. 1 hadoop hadoop  23015 Nov 14 10:04 Gemfile.jruby-1.9.lock
drwxrwxr-x. 4 hadoop hadoop   4096 Mar 27 03:58 lib
-rw-rw-r--. 1 hadoop hadoop    589 Nov 14 10:04 LICENSE
-rw-rw-r--. 1 hadoop hadoop    149 Nov 14 10:04 NOTICE.TXT
drwxrwxr-x. 4 hadoop hadoop   4096 Mar 27 03:58 vendor
[hadoop@HadoopMaster logstash-2.4.1]$ vim logstash-simple.conf

 

 

input {
     stdin { } 
}
output {
    stdout { }
}

 

 

 

 

[hadoop@HadoopMaster logstash-2.4.1]$ pwd
/home/hadoop/app/logstash-2.4.1
[hadoop@HadoopMaster logstash-2.4.1]$ ll
total 164
drwxrwxr-x. 2 hadoop hadoop   4096 Mar 27 03:58 bin
-rw-rw-r--. 1 hadoop hadoop 102879 Nov 14 10:04 CHANGELOG.md
-rw-rw-r--. 1 hadoop hadoop   2249 Nov 14 10:04 CONTRIBUTORS
-rw-rw-r--. 1 hadoop hadoop   5084 Nov 14 10:07 Gemfile
-rw-rw-r--. 1 hadoop hadoop  23015 Nov 14 10:04 Gemfile.jruby-1.9.lock
drwxrwxr-x. 4 hadoop hadoop   4096 Mar 27 03:58 lib
-rw-rw-r--. 1 hadoop hadoop    589 Nov 14 10:04 LICENSE
-rw-rw-r--. 1 hadoop hadoop     46 Mar 27 05:30 logstash-simple.conf
-rw-rw-r--. 1 hadoop hadoop    149 Nov 14 10:04 NOTICE.TXT
drwxrwxr-x. 4 hadoop hadoop   4096 Mar 27 03:58 vendor
[hadoop@HadoopMaster logstash-2.4.1]$ bin/logstash -f logstash-simple.conf
Settings: Default pipeline workers: 1
Pipeline main started

2017-03-26T21:32:32.782Z HadoopMaster 
abcd
2017-03-26T21:32:36.848Z HadoopMaster abcd
^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>"main"}
^CSIGINT received. Terminating immediately.. {:level=>:fatal}
[hadoop@HadoopMaster logstash-2.4.1]$ 

 

 

 　　推薦用這個！！！

bin/logstash -f logstash-simple.conf --auto-reload

　　由於，在調試，每次都要重啓。加這個，不需每次去重啓Logstash，即本身會加載。