Elastic Stack配置和使用

時間  2019-12-08
標籤 elastic stack 配置 使用 简体版
原文   原文鏈接

Elastic Stack是一個開源的解決方案，能夠收集各類類型，各類格式的源數據，同時提供數據搜索，分析和可視化的展現
＃ 通用搜索引擎
索引組件：獲取數據-->創建文檔-->文檔分析-->文檔索引（倒排索引），如Lucene
搜索組件：用戶搜索接口-->創建查詢（將用戶鍵入的信息轉換爲可處理的查詢對象）-->搜索查詢-->展示結果，如Solr，ElasticSearchhtml

各組件介紹

Lucene Core

Apache LuceneTM is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform.

Solr

SolrTM is a high performance search server built using Lucene Core, with XML/HTTP and JSON/Python/Ruby APIs, hit highlighting, faceted search, caching, replication, and a web admin interface.

ElasticSearch

Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected.

Elastic Stack組件

架構圖

以index爲界線，下半部分的主要功能完成索引，上半部分完成搜索。node

ElasticSearch

功能如上。

Logstash

Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite 「stash.」 (Ours is Elasticsearch, naturally.)

Beats

Filebeat：Log Files
Metricbeat：Metrics
Packetbeat：Network Data
Winlogbeat：Windows Event Logs
Heartbeat：Uptime Monitoring

Kibana

Kibana lets you visualize your Elasticsearch data and navigate the Elastic Stack, so you can do anything from learning why you're getting paged at 2:00 a.m. to understanding the impact rain might have on your quarterly numbers.
小結：對應於架構圖，Logstash做爲agent，和Beats組件同時能夠完成獲取內容，logstash比較重量級在消耗系統資源方面，因此實際中用Beats較多；Logstash做爲服務器端，能夠完成建立文檔；ElasticSearch負責索引，同時提供搜索功能；Kibana提供可視的圖形管理和展現界面。

ElasticSearch

配置文件：

/etc/elasticsearch/elasticsearch.yml
/etc/elasticsearch/jvm.options
/etc/elasticsearch/log4j2.properties
Unit File：elasticsearch.service

程序文件：

/usr/share/elasticsearch/bin/elasticsearch
/usr/share/elasticsearch/bin/elasticsearch-keystore：
/usr/share/elasticsearch/bin/elasticsearch-plugin：管理插件程序

搜索服務：9200/tcp

集羣服務：9300/tcp

集羣配置和使用

工做邏輯：全部節點選舉一個主節點，負責管理整個集羣的狀態(green/yellow/red)，以及各shards的分佈方式；
### ELS構成
    集羣：一個或多個節點的集合；
    節點：運行的單個els實例；
    索引：切成多個獨立的shard；（以Lucene的視角，每一個shard即爲一個獨立而完整的索引）
###　集羣配置：
    1. 到官網下載elasticsearch-5.6.10.rpm
    2. 準備三臺服務器
    3. yum install elasticsearch-5.6.10.rpm
    4. elasticsearch.yml配置文件：
        cluster.name: myels (三個節點上的cluster.name一致)
        node.name: node1 （各個不一樣的節點更改成自已的名字，分別爲node2，node3）
                    #node.attr.rack: r1 （機櫃感知配置，須要定義好服務器處於的機架）
        path.data: /data/els/data （ chown elasticsearch.elasticsearch）
        path.logs: /data/els/logs
        network.host: 0.0.0.0
        http.port: 9200
        discovery.zen.ping.unicast.hosts: ["node1", "node2", "node3"]
        discovery.zen.minimum_master_nodes: 2   
    5. #  curl -XGET 'http://node01:9200/'
        {
              "name" : "node01",
              "cluster_name" : "myels-evan",
              "cluster_uuid" : "w_N3c2aXQnWBEe1UFrIQ8A",
              "version" : {
                "number" : "5.6.10",
                "build_hash" : "b727a60",
                "build_date" : "2018-06-06T15:48:34.860Z",
                "build_snapshot" : false,
                "lucene_version" : "6.6.1"
          },
          "tagline" : "You Know, for Search"
            }
        # 檢查集羣狀態 curl -XGET 'http://node01:9200/_cluster/health?pretty=true'
            {
              "cluster_name" : "myels-evan",
              "status" : "green",
              "timed_out" : false,
              "number_of_nodes" : 3,
              "number_of_data_nodes" : 3,
              "active_primary_shards" : 0,
              "active_shards" : 0,
              "relocating_shards" : 0,
              "initializing_shards" : 0,
              "unassigned_shards" : 0,
              "delayed_unassigned_shards" : 0,
              "number_of_pending_tasks" : 0,
              "number_of_in_flight_fetch" : 0,
              "task_max_waiting_in_queue_millis" : 0,
              "active_shards_percent_as_number" : 100.0
            }
### 集羣使用：（RTESful接口風格）
    RESTful API: CRUD(Create, Read, Update, Delete)
    curl  -X<VERB> '<PROTOCOL>://<HOST>:<PORT>/<PATH>?<QUERY_STRING>' -d '<BODY>'
        <BODY>：json格式的請求主體；
                    <VERB>：GET，POST，PUT，DELETE
                    特殊PATH：/_cat, /_search, /_cluster （檢查狀態）
                    <PATH> /index_name/type/Document_ID/
                    curl  -XPUT 建立文檔
                    文檔語法：{"key1": "value1", "key2": value, ...}
    實例：
       # curl -XGET 'http://node02:9200/_cluster/stats?pretty=true' （檢查集羣的詳細信息）
       # curl -XPUT http://node02:9200/myindex （建立索引）
       # curl -XGET http://node02:9200/_cat/indices（檢查索引）
       # curl -XDELETE http://node02:9200/myindex （刪除索引）
       # curl -XGET http://node02:9200/_cat/shards
       # curl -XPUT http://node02:9200/myindex/students/1?pretty -d ‘{"name":"Liang Jindong","age":30,"major":"good english"}’ 建立文檔
       #  curl -XGET http://node02:9200/_search? （能夠在不一樣的路徑上搜索來定義搜索範圍）

Logstash日誌收集工具

Logstash能夠同時做爲agent和server來從指定的位置（如file，mysql, redis）抽取數據，並進行文檔化，而後發送給ElasticSearch，也能夠只做爲服務端，配合輕量化的filebeat抽取數據，在此應用中，logstash只做日誌文檔化，併發送給ElasticSearch。如下是幾種應用場景的架構圖：

配置文件有三部分組成，分別定義輸入，過濾，輸出，由不一樣類型的插件支持。（注意任何定義在conf.d下的文件都會做爲配置文件加載，不一樣於httpd下必須以.conf結尾才能做爲配置文件)

input {
        ...
    }
    
    filter{
        ...
    }
    
    output {
        ...
    }

Logstash安裝使用

1. 下載logstash rpm
2. 安裝JAVA SDK
3. rpm -ivh logstash-5.6.10.rpm
4. vim /etc/profile.d/logstash.sh
    export PATH=/usr/share/logstash/bin/logstash:$PATH

### Logstash配置示例1（標準輸入和輸出）
#vim /etc/logstash/conf.d/test.conf
input {
stdin{}
}mysql

output {
                    stdout {
                        codec => rubydebug
            }
            }
# logstash -f test.conf -t (測試語法)
# logstash -f test.conf  （以test.conf配置文件啓動）
# 生成新的文件test.conf，加上filter
    input {
    stdin{}
        }

    filter {
        grok {
                match => { "message" => "%{NUMBER:duration} %{IP:client}" }
        }
    }
    output {
            stdout {
                    codec => rubydebug
            }
    }
    #logstash -f test.conf (啓動)
    # 輸入32 1.2.3.4 
    # 輸出爲以下，模式中的NUMBER和IP用key duration和client把數字和IP進行拆分，輸出新的值。
        The stdin plugin is now waiting for input:
        {
          "@version" => "1",
             "host" => "kvm",
              "duration" => "32",
                "client" => "1.2.3.4",
                "@timestamp" => 2018-08-01T06:15:23.613Z,
               "message" => "32 1.2.3.4"

### Logstash配置示例2（web服務生成日誌，logstash加載並輸出，同時agent和server）web

# yum install httpd
# for i in {1..50}; do echo "Test page $i." > test$i.html; done (生成40個頁面)
# systemctl start httpd
# LogFormat "%{X-Forwarded-For}i  (更改httpd的logformat)
# while true; do client=$[$RANDOM%254+1]; curl -s --header "X-Forwarded-For: 172.20.0.$client" http://172.20.42.239/test$client.html;sleep 1;done （模擬客戶端訪問，生成httpd的access_log）
以文件爲輸入插件，同時加載Grok過濾器插件對日誌分段，每一段加有特定意義的key，才能顯示數據的價值。Grok內置支持120多種模式，去匹配如htpd, redis等，能夠查看/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns，每一個模式匹配都有惟一的名字對應，在logstash配置文件中直接引用內置模式的名稱。
# vim test2.conf
   input {
    file{
            start_position => end
            path => ["/var/log/httpd/access_log"]
        }
    }

    filter {
            grok {
                match => { "message" => "%{HTTPD_COMBINEDLOG}" }
        }
    }
    output {
            stdout {
                    codec => rubydebug
                }
    }
# logstash -f test3.conf
# 顯示結果，模式匹配切割正常，達到輸出內容結構化。
       "request" => "/test54.html",
      "agent" => "\"curl/7.29.0\"",
       "auth" => "-",
      "ident" => "-",
       "verb" => "GET",
    "message" => "172.20.0.54 - - [01/Aug/2018:02:38:48 -0400] \"GET /test54.html HTTP/1.1\" 200 14 \"-\" \"curl/7.29.0\"",
       "path" => "/var/log/httpd/access_log",
       "referrer" => "\"-\"",
     "@timestamp" => 2018-08-01T06:38:48.897Z,
       "response" => "200",
      "bytes" => "14",
       "clientip" => "172.20.0.54",
       "@version" => "1",
       "host" => "kvm",
    "httpversion" => "1.1",
      "timestamp" => "01/Aug/2018:02:38:48 -0400"
    }
插件date，能把上述的@timestamp字段以timestamp字段顯示，同時把timestamp字段刪除（插件date的默認功能）
# vim test3.conf
input {
    file{
            start_position => end
            path => ["/var/log/httpd/access_log"]
        }
    }

    filter {
            grok {
                    match => { "message" => "%{HTTPD_COMBINEDLOG}" }
                    remove_field => "message"
        }
            date {
                    match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
                    remove_field => "timestamp"
        }
    }
    output {
            stdout {
                    codec => rubydebug
                }
    }

# 輸出 
{
    "request" => "/test111.html",
      "agent" => "\"curl/7.29.0\"",
       "auth" => "-",
      "ident" => "-",
       "verb" => "GET",
       "path" => "/var/log/httpd/access_log",
       "referrer" => "\"-\"",
     "@timestamp" => 2018-08-01T06:47:35.000Z,
       "response" => "404",
      "bytes" => "210",
       "clientip" => "172.20.0.111",
       "@version" => "1",
       "host" => "kvm",
    "httpversion" => "1.1"
    }

Geoip插件，能把訪問的IP用圖形展現所處的詳細。
# 到https://dev.maxmind.com/geoip/geoip2/geolite2/下載簡單版本
# vim test4.conf
input {
    file{
            start_position => end
            path => ["/var/log/httpd/access_log"]
        }
    }

    filter {
            grok {
                    match => { "message" => "%{HTTPD_COMBINEDLOG}" }
                    remove_field => "message"
            }
    date {
            match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
            remove_field => "timestamp"
    }
    geoip {
            source => "clientip"
            target => "geoip"
            database => "/etc/logstash/GeoLite2-City_20180703/GeoLite2-City.mmdb"
                }
    }
   output {
   stdout {
            codec => rubydebug
            }
    }
# 輸出IP所在的國家，城市，經緯度
{
    "request" => "/test208.html",
      "agent" => "\"curl/7.29.0\"",
      "geoip" => {
                "ip" => "208.20.0.100",
          "latitude" => 37.751,
      "country_name" => "United States",
     "country_code2" => "US",
    "continent_code" => "NA",
     "country_code3" => "US",
          "location" => {
        "lon" => -97.822,
        "lat" => 37.751
    },

輸出插件ElasticSearch
# vim test5.conf
input {
    file{
            start_position => end
            path => ["/var/log/httpd/access_log"]
    }
    }

filter {
        grok {
            match => { "message" => "%{HTTPD_COMBINEDLOG}" }
            remove_field => "message"
    }
    date {
            match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
            remove_field => "timestamp"
    }
    geoip {
            source => "clientip"
            target => "geoip"
            database => "/etc/logstash/GeoLite2-City_20180703/GeoLite2-City.mmdb"
                }
    }
    output {
    elasticsearch {
           hosts => ["http://node1:9200/","http://node2:9200/","http://node3:9200/"]
           index => "logstash-%{+YYYY.MM.dd}"
           document_type => "apache_logs"
           }
    }
# logstash -f test5.conf 啓動
# 查看ElasticSearch，能夠看到以logstash爲索引的
    curl -XGET node02:9200/_cat/indices
    green open logstash-2018.08.01 LceRz3DlQkiBx8BlEDhs_g 5 1 611 0 1.1mb 746kb
# 查詢此index的具體信息
    curl -XGET node02:9200/logstash-2018.08.01/_search?q=clientip:109.20.0.100
    curl -XGET node02:9200/logstash-2018.08.01/_search?q=response:404

安裝Kibana圖表展現工具：
只要Kinba能訪問ElasticSearch中的任何一個結點， 就能夠圖形展現
# wget https://artifacts.elastic.co/downloads/kibana/kibana-5.6.8-x86_64.rpm
# rpm -ivh kibana-5.6.8-x86_64.rpm
# cd /etc/kibana/
# server.port: 5601; server.basePath: "/kibina"; server.host: "0.0.0.0"; elasticsearch.url: "http://node02:9200"
# systemctl start kibana
# http://172.20.42.240:5601/app/kibana 訪問網頁
# 選擇加載的index能夠用通配符匹配
# 設置搜索item如geoip.timezone:Asia
response:[200 TO 302] 範圍查詢
agent:curl 客戶端訪問類型
agent:curl OR chrome or firefox OR safari
agent:curr~ 模糊搜索
# 還能夠用Visualize
redis

Logstash配置示例3（web服務生成日誌，beats組件輕量級的收集日誌，logstash只作爲server，輸出到ElasticSearch）

filebeat須要從指定的位置讀入日誌，而後再發給logstash，filebeat沒法作日誌的文檔化
# vim filebeat.yml
     paths:
    - /var/log/httpd/access_log*
    output.elasticsearch:
      output.elasticsearch:
      hosts: ["node03:9200","node02:9200"]
# systemctl start filebeat
# curl node03:9200/_cat/indices
        yellow open filebeat-2018.08.01 M2TgTLhFTe2MtprkyR9lag 5 1    0 0   324b   324b
# 更改filebeat輸出到logstash
    output.logstash:
    hosts: ["172.20.42.239:5044"]

# 更改logstash的配置文件（input更改成beats，其他保持不變）
    input {
    beats {
            port => 5044
        }
    }
# curl node03:9200/_cat/indices
    yellow open logstash-2018.08.01 PbCtEn_UTZOfknX_5LnGaA 5 1    148 0 838.5kb 838.5kb

Logstash配置示例4（web服務生成日誌，beats組件輕量級的收集日誌，發送給redis消息隊列，logstash只作爲server去redis取數據，輸出到ElasticSearch）

# yum install redis
# vim /etc/redis.conf
    bind 0.0.0.0
    requirepass redhat
# systemctl start redis
# filebeat.full.yml中有redis的配置示例 ,複製到filebeat.yml，更改相關的配置
    output.redis:
    enabled: true
     hosts: ["172.20.42.222:6379"]
     port: 6379
     key: filebeat
     password: redhat
     db: 0
     datatype: list
# logstash從redis中加載日誌
    input {
        redis {
                data_type => "list"
                db => 0
                host => "172.20.42.239"
                key => "filebeat"
                port => 6379
                password => "redhat"
        }
    }
# 在ElasticSearch上 curl node03:9200/_cat/indices
    yellow open logstash-2018.08.02 dKULdxU_SzuxmJQToCMbug 5 1 2246 0  1.3mb  1.3mb
# 在redis服務器，redis-cli中查看，filebeat中消息隊列爲0，即消息隊列每增長一個，都會被logstash取走。
    127.0.0.1:6379> LLEN filebeat
    (integer) 0
相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程