使用Docker構建ELK Docker集羣日誌收集系統

當咱們搭建好Docker集羣后就要解決如何收集日誌的問題 ELK就提供了一套完整的解決方案 本文主要介紹使用Docker搭建ELK 收集Docker集羣的日誌

ELK簡介

ELK由ElasticSearch、Logstash和Kiabana三個開源工具組成

Elasticsearch是個開源分佈式搜索引擎，它的特色有：分佈式，零配置，自動發現，索引自動分片，索引副本機制，restful風格接口，多數據源，自動搜索負載等。

Logstash是一個徹底開源的工具，他能夠對你的日誌進行收集、過濾，並將其存儲供之後使用

Kibana 也是一個開源和免費的工具，它Kibana能夠爲 Logstash 和 ElasticSearch 提供的日誌分析友好的 Web 界面，能夠幫助您彙總、分析和搜索重要數據日誌。

使用Docker搭建ELK平臺

首先咱們編輯一下 logstash的配置文件 logstash.conf

input { 
   udp {
    port => 5000
    type => json
  }
}
filter {
   json {
      source => "message"
   }
}
output {
	elasticsearch {
             hosts => "elasticsearch:9200"  #將logstash的輸出到 elasticsearch 這裏改爲大家本身的host 
	}
}

而後咱們還須要須要一下Kibana 的啓動方式

編寫啓動腳本 等待elasticserach 運行成功後啓動

#!/usr/bin/env bash

# Wait for the Elasticsearch container to be ready before starting Kibana.
echo "Stalling for Elasticsearch" 
while true; do
    nc -q 1 elasticsearch 9200 2>/dev/null && break
done

echo "Starting Kibana"
exec kibana

修改Dockerfile 生成自定義的Kibana鏡像

FROM kibana:latest

RUN apt-get update && apt-get install -y netcat

COPY entrypoint.sh /tmp/entrypoint.sh
RUN chmod +x /tmp/entrypoint.sh

RUN kibana plugin --install elastic/sense

CMD ["/tmp/entrypoint.sh"]

同時也能夠修改一下Kibana 的配置文件 選擇須要的插件

# Kibana is served by a back end server. This controls which port to use.
port: 5601

# The host to bind the server to.
host: "0.0.0.0"

# The Elasticsearch instance to use for all your queries.
elasticsearch_url: "http://elasticsearch:9200"

# preserve_elasticsearch_host true will send the hostname specified in `elasticsearch`. If you set it to false,
# then the host you use to connect to *this* Kibana instance will be sent.
elasticsearch_preserve_host: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations
# and dashboards. It will create a new index if it doesn't already exist.
kibana_index: ".kibana"

# If your Elasticsearch is protected with basic auth, this is the user credentials
# used by the Kibana server to perform maintence on the kibana_index at statup. Your Kibana
# users will still need to authenticate with Elasticsearch (which is proxied thorugh
# the Kibana server)
# kibana_elasticsearch_username: user
# kibana_elasticsearch_password: pass

# If your Elasticsearch requires client certificate and key
# kibana_elasticsearch_client_crt: /path/to/your/client.crt
# kibana_elasticsearch_client_key: /path/to/your/client.key

# If you need to provide a CA certificate for your Elasticsarech instance, put
# the path of the pem file here.
# ca: /path/to/your/CA.pem

# The default application to load.
default_app_id: "discover"

# Time in milliseconds to wait for elasticsearch to respond to pings, defaults to
# request_timeout setting
# ping_timeout: 1500

# Time in milliseconds to wait for responses from the back end or elasticsearch.
# This must be > 0
request_timeout: 300000

# Time in milliseconds for Elasticsearch to wait for responses from shards.
# Set to 0 to disable.
shard_timeout: 0

# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
# startup_timeout: 5000

# Set to false to have a complete disregard for the validity of the SSL
# certificate.
verify_ssl: true

# SSL for outgoing requests from the Kibana Server (PEM formatted)
# ssl_key_file: /path/to/your/server.key
# ssl_cert_file: /path/to/your/server.crt

# Set the path to where you would like the process id file to be created.
# pid_file: /var/run/kibana.pid

# If you would like to send the log output to a file you can set the path below.
# This will also turn off the STDOUT log output.
log_file: ./kibana.log
# Plugins that are included in the build, and no longer found in the plugins/ folder
bundled_plugin_ids:
 - plugins/dashboard/index
 - plugins/discover/index
 - plugins/doc/index
 - plugins/kibana/index
 - plugins/markdown_vis/index
 - plugins/metric_vis/index
 - plugins/settings/index
 - plugins/table_vis/index
 - plugins/vis_types/index
 - plugins/visualize/index

好了下面咱們編寫一下 Docker-compose.yml 方便構建

端口之類的能夠根據本身的需求修改 配置文件的路徑根據你的目錄修改一下 總體系統配置要求較高 請選擇配置好點的機器

elasticsearch:
  image: elasticsearch:latest
  command: elasticsearch -Des.network.host=0.0.0.0
  ports:
    - "9200:9200"
    - "9300:9300"
logstash:
  image: logstash:latest
  command: logstash  -f /etc/logstash/conf.d/logstash.conf
  volumes:
    - ./logstash/config:/etc/logstash/conf.d
  ports:
    - "5001:5000/udp"
  links:
    - elasticsearch
kibana:
  build: kibana/
  volumes:
    - ./kibana/config/:/opt/kibana/config/
  ports:
    - "5601:5601"
  links:
    - elasticsearch

#好了命令 就能夠直接啓動ELK了  
docker-compose up -d

訪問以前的設置的kibanna的5601端口就能夠看到是否啓動成功了

使用logspout收集Docker日誌

下一步咱們要使用logspout對Docker日誌進行收集 咱們根據咱們的需求修改一下logspout鏡像

編寫配置文件 modules.go

package main

import (
    _ "github.com/looplab/logspout-logstash"
    _ "github.com/gliderlabs/logspout/transports/udp"

)

編寫Dockerfile

FROM  gliderlabs/logspout:latest
COPY ./modules.go /src/modules.go

從新構建鏡像後 在各個節點運行便可

docker run -d --name="logspout"  --volume=/var/run/docker.sock:/var/run/docker.sock \
                 jayqqaa12/logspout  logstash://你的logstash地址

如今打開Kibana 就能夠看到收集到的 docker日誌了

注意Docker容器應該選擇以console輸出 這樣才能採集到

好了咱們的Docker集羣下的ELK 日誌收集系統就部署完成了

若是是大型集羣還須要添加logstash 和elasticsearch 集羣 這個咱們下回分解