最近工做須要,用filebeat將Hadoop日誌導入到Elasticsearch中,在kibana中展現,記錄下。瀏覽器
版本分別是elasticsearch:6.5.一、kibana:6.5.一、filebeat:6.4,版本要對應,開始我用的filebeat版本是7.0,致使在kibana中展現時有問題,後來用6.4版本就沒有這個問題了。bash
首先要啓動elasticearch和kibana,我是在Docker中運行這倆個的,具體能夠看個人elasticsearch、kibana博客。瀏覽器打開localhost:5601能夠看到kibana的界面。elasticsearch
去filebeat官網下載6.4版本的壓縮包,以後解壓並進入目錄,以後修改filebeat.yml:oop
List-1fetch
#=========================== Filebeat inputs ============================= filebeat.inputs: # Each - is an input. Most options can be set at the input level, so # you can use different inputs for various configurations. # Below are the input specific configurations. - type: log # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: #- /var/log/*.log - /opt/software/tool/hadoop/hadoop/logs/*.log #- c:\programdata\elasticsearch\logs\*
修改filebeat的input:ui
List-2this
#============================== Kibana ===================================== # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API. # This requires a Kibana endpoint configuration. setup.kibana: # Kibana Host # Scheme and port can be left out and will be set to the default (http and 5601) # In case you specify and additional path, the scheme is required: http://localhost:5601/path # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601 host: "localhost:5601"
如上List-2中所示,修改Kibana模塊,將'host: "localhost:5601"'以前的註釋去掉。.net
List-3日誌
#================================ Outputs ===================================== # Configure what output to use when sending the data collected by the beat. #-------------------------- Elasticsearch output ------------------------------ output.elasticsearch: # Array of hosts to connect to. hosts: ["localhost:9200"] # Optional protocol and basic auth credentials. #protocol: "https" #username: "elastic" #password: "changeme"
如List-3所示,修改Outputs模塊,將Elasticsearch output的hosts: ["localhost:9200"]的值改成咱們本身elasticsearch的IP與端口。code
以後如List-4所示,沒有報錯就說明成功了。
List-4
./filebeat setup --dashboards Loading dashboards (Kibana must be running and reachable) Loaded dashboards #啓動filebeat ./filebeat -e -c filebeat.yml
來看kibana的界面,以下圖1中的Discover能夠看到日誌,圖2中的logs能夠看到隨着hadoop的日誌文件內容被修改,kibana中的日誌會相應的滾動顯示,圖1和圖2中的搜索框中能夠輸入值進行搜索。
圖1
圖2