ELK之filebeat-redis-logstash-es構架模式

　　下載filebeat的rpm包安裝filebeat

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.3.0-x86_64.rpm

　　安裝

filebeat-6.3.0-x86_64.rpm

　　配置文件/etc/filebeat/filebeat.yml 

　　寫一個配置文件

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
    - /var/log/messages
  
  exclude_lines: ['^DBG','^$']
  document_type: system-log-5611
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
output.file:
  path: "/tmp"
  name: "filebeat.txt"

　　默認不帶type這裏自定義type爲document_type: system-log-5611

　　排除空行exclude_lines: ['^DBG','^$']

　　這裏不寫入到elasticsearch而是先寫入到一個文件

　　啓動

systemctl start filebeat

　　PS：在/tmp下面生成了文件filebeat可是沒有txt（緣由未知）

　　

　　修改配置文件把輸出改爲redis

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
    - /var/log/messages
  tags: ["system-log-5611"]
  exclude_lines: ['^DBG','^$']
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
output.redis:
  hosts: ["192.168.56.11"]
  db: "3"
  port: "6379"
  password: "123456"
  key: "system-log-5611"

　　PS：tags才能生效 redis裏面的key不能輸出對應的key值（filebeat版本爲6.3）

　　redis必須設置密碼，不然啓動filebeat報錯，報錯日誌文件爲/var/log/filebeat/filebeat

　　重啓filebeat

systemctl restart filebeat

　 　使用echo的方式往/var/log/messages插入幾條數據而後使用客戶端鏈接redis查看

　　配置使用logstash取出redis裏面的數據

input{
    redis {
    host => "192.168.56.11"
    port => "6379"
    password => "123456"
    db => "3"
    data_type => "list"
    key => "system-log-5611"

}
}

output{
    if "system-log-5611" in [tags]  {
       elasticsearch {
            hosts => ["192.168.56.11:9200"]
            index => "system-log-5611-%{+YYYY.MM.dd}"
        }
       stdout{
           codec => rubydebug
       }
    }
}

　　啓動logstash輸出

　　同時elasticsearch也收到了