ElasticSearch+Kibana+Packetbeat

1、介紹
Packetbeat 是一個實時網絡數據包分析工具，經過嗅探應用服務器之間的網絡通信，來解碼應用層協議類型如HTTP、MySQL、redis等等，關聯請求與響應，並記錄每一個事務有意義的字段。
Kibana是一個分析與可視化平臺，設計出來用於和Elasticsearch一塊兒使用的。你能夠用kibana搜索、查看、交互存放在Elasticsearch索引裏的數據，使用各類不一樣的圖表、表格、地圖等kibana可以很輕易地展現高級數據分析與可視化。
ElasticSearch是一個開源的分佈式搜索引擎，具有高可靠性，支持很是多的企業級搜索用例。像Solr4同樣，是基於Lucene構建的。支持時間時間索引和全文檢索。

2、系統要求：Java環境

3、安裝步驟
安裝ElasticSearch
1.下載解壓
2../bin/elasticsearch
若是是用root帳號啓動，會報如下錯誤
java.lang.RuntimeException: can not run elasticsearch as root
3.添加普通用戶elsearch

[root@localhost bin]# groupadd elsearch
[root@localhost bin]# useradd elsearch -g elsearch 
[root@localhost local]# chown -R elsearch:elsearch elasticsearch-5.3.1/
[root@localhost logs]# su elsearch
[elsearch@localhost logs]$ cd /usr/local/elasticsearch-5.3.1/bin/cd elasticsearch/bin
[elsearch@localhost bin]$./elasticsearch -d

啓動後打印信息以下

[2017-04-29T02:05:37,773][INFO ][o.e.n.Node ] [] initializing ...
[2017-04-29T02:05:38,283][INFO ][o.e.e.NodeEnvironment ] [BdGOiwE] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [12.5gb], net total_space [16.9gb], spins? [unknown], types [rootfs]
[2017-04-29T02:05:38,284][INFO ][o.e.e.NodeEnvironment ] [BdGOiwE] heap size [503.6mb], compressed ordinary object pointers [true]
[2017-04-29T02:05:38,286][INFO ][o.e.n.Node ] node name [BdGOiwE] derived from node ID [BdGOiwEfSXa1393zTZkD5w]; set [node.name] to override
[2017-04-29T02:05:38,286][INFO ][o.e.n.Node ] version[5.3.1], pid[53678], build[5f9cf58/2017-04-17T15:52:53.846Z], OS[Linux/3.10.0-514.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_102/25.102-b14]
[2017-04-29T02:05:41,445][INFO ][o.e.p.PluginsService ] [BdGOiwE] loaded module [aggs-matrix-stats]
[2017-04-29T02:05:41,445][INFO ][o.e.p.PluginsService ] [BdGOiwE] loaded module [ingest-common]
[2017-04-29T02:05:41,445][INFO ][o.e.p.PluginsService ] [BdGOiwE] loaded module [lang-expression]
[2017-04-29T02:05:41,446][INFO ][o.e.p.PluginsService ] [BdGOiwE] loaded module [lang-groovy]
[2017-04-29T02:05:41,446][INFO ][o.e.p.PluginsService ] [BdGOiwE] loaded module [lang-mustache]
[2017-04-29T02:05:41,446][INFO ][o.e.p.PluginsService ] [BdGOiwE] loaded module [lang-painless]
[2017-04-29T02:05:41,446][INFO ][o.e.p.PluginsService ] [BdGOiwE] loaded module [percolator]
[2017-04-29T02:05:41,446][INFO ][o.e.p.PluginsService ] [BdGOiwE] loaded module [reindex]
[2017-04-29T02:05:41,446][INFO ][o.e.p.PluginsService ] [BdGOiwE] loaded module [transport-netty3]
[2017-04-29T02:05:41,446][INFO ][o.e.p.PluginsService ] [BdGOiwE] loaded module [transport-netty4]
[2017-04-29T02:05:41,446][INFO ][o.e.p.PluginsService ] [BdGOiwE] no plugins loaded
[2017-04-29T02:05:52,078][INFO ][o.e.n.Node ] initialized
[2017-04-29T02:05:52,094][INFO ][o.e.n.Node ] [BdGOiwE] starting ...
[2017-04-29T02:05:52,830][INFO ][o.e.t.TransportService ] [BdGOiwE] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-04-29T02:05:52,835][WARN ][o.e.b.BootstrapChecks ] [BdGOiwE] max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2017-04-29T02:05:52,835][WARN ][o.e.b.BootstrapChecks ] [BdGOiwE] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2017-04-29T02:05:56,068][INFO ][o.e.c.s.ClusterService ] [BdGOiwE] new_master {BdGOiwE}{BdGOiwEfSXa1393zTZkD5w}{SEihKgNKTqe1bHDo316wlw}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-04-29T02:05:56,177][INFO ][o.e.g.GatewayService ] [BdGOiwE] recovered [0] indices into cluster_state
[2017-04-29T02:05:56,188][INFO ][o.e.h.n.Netty4HttpServerTransport] [BdGOiwE] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-04-29T02:05:56,190][INFO ][o.e.n.Node ] [BdGOiwE] started

沒法啓動的錯誤，解決辦法不少，Google之，在此略過
瀏覽器打開http://ip:9200/ 顯示以下

安裝Kibana
1.下載解壓
2. ./bin/kibana
3. 瀏覽器打開 http://localhost:5601
若是打不開頁面，能夠參考https://segmentfault.com/q/1010000007488864

安裝packetbeat
1.下載解壓
2.加載Packetbeat索引模板

[root@localhost packetbeat-5.3.1-linux-x86_64]# curl -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/packetbeat' -d@packetbeat.template.json
{"acknowledged":true}

3.導入 Kibana 儀表盤示例

[root@localhost scripts]# ./import_dashboards -es http://localhost:9200
Create temporary directory /tmp/tmp244734216
Downloading https://artifacts.elastic.co/downloads/beats/beats-dashboards/beats-dashboards-5.3.1.zip
Unzip archive /tmp/tmp244734216
Importing Kibana from /tmp/tmp244734216/beats-dashboards-5.3.1/filebeat
Importing Kibana from /tmp/tmp244734216/beats-dashboards-5.3.1/heartbeat
Importing Kibana from /tmp/tmp244734216/beats-dashboards-5.3.1/metricbeat
......

4.測試

[root@localhost packetbeat-5.3.1-linux-x86_64]# curl -XGET 'http://localhost:9200/packet'eat-*/_search?pretty 
{
  "took" : 106,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 2576,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "packetbeat-2017.04.29",
        "_type" : "flow",
        "_id" : "AVu135SZFyYD-AeZcX68",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2017-04-28T18:43:00.016Z",
          "beat" : {
            "hostname" : "localhost.localdomain",
            "name" : "localhost.localdomain",
            "version" : "5.3.1"
            ......