filebeat導入Hadoop日誌到Elasticsearch中

    最近工做須要,用filebeat將Hadoop日誌導入到Elasticsearch中,在kibana中展現,記錄下。瀏覽器

    版本分別是elasticsearch:6.5.一、kibana:6.5.一、filebeat:6.4,版本要對應,開始我用的filebeat版本是7.0,致使在kibana中展現時有問題,後來用6.4版本就沒有這個問題了。bash

    首先要啓動elasticearch和kibana,我是在Docker中運行這倆個的,具體能夠看個人elasticsearchkibana博客。瀏覽器打開localhost:5601能夠看到kibana的界面。elasticsearch

    去filebeat官網下載6.4版本的壓縮包,以後解壓並進入目錄,以後修改filebeat.yml:oop

    List-1fetch

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    - /opt/software/tool/hadoop/hadoop/logs/*.log
    #- c:\programdata\elasticsearch\logs\*

    修改filebeat的input:ui

  1. enabled的值改成true
  2. paths的值添加hadoop的日誌目錄

    List-2this

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "localhost:5601"

    如上List-2中所示,修改Kibana模塊,將'host: "localhost:5601"'以前的註釋去掉。.net

    List-3日誌

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

    如List-3所示,修改Outputs模塊,將Elasticsearch output的hosts: ["localhost:9200"]的值改成咱們本身elasticsearch的IP與端口。code

    以後如List-4所示,沒有報錯就說明成功了。

    List-4

./filebeat setup --dashboards
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards

#啓動filebeat
./filebeat -e -c filebeat.yml

      來看kibana的界面,以下圖1中的Discover能夠看到日誌,圖2中的logs能夠看到隨着hadoop的日誌文件內容被修改,kibana中的日誌會相應的滾動顯示,圖1和圖2中的搜索框中能夠輸入值進行搜索。

                                                                                  圖1

                                                                                 圖2

相關文章
相關標籤/搜索