docker-compose ELK+Filebeat查看docker及容器的日誌

我目前所在公司開發團隊比較小，爲集團下面的工廠開發了一套小的系統，跑在一臺CentOS服務器上，服務器搭建了docker環境，安裝了docker-compose，但在日誌處理方面，暫時沒有一個好的方法可以收集徹底的日誌，只能依賴進入至服務器後，以docker logs containerID的方法來進入查看，很是不方便，以前也有關注ELK的技術，但一直在開發系統功能，全力實現，今天得空，從新想起了ELK查看日誌的任務。

 

kibana文檔：https://www.elastic.co/guide/cn/kibana/current/index.html

關於elastic開始視頻：https://www.elastic.co/guide/cn/index.html

 

 

 

項目文件夾

其中docker-compose.yml

version: '3' services: filebeat: hostname: filebeat image: weschen/filebeat build: context: filebeat dockerfile: Dockerfile volumes: # needed to access all docker logs (read only) :
     - "/var/lib/docker/containers:/usr/share/dockerlogs/data:ro"
      # needed to access additional informations about containers
     - "/var/run/docker.sock:/var/run/docker.sock" links: - logstash kibana: image: docker.elastic.co/kibana/kibana:6.5.2 environment: - "LOGGING_QUIET=true" links: - elasticsearch ports: - 5601:5601 logstash: hostname: logstash image: weschen/logstash build: context: logstash dockerfile: Dockerfile ports: - 5044:5044 environment: LOG_LEVEL: error links: - elasticsearch elasticsearch: hostname: elasticsearch image: weschen/elasticsearch build: context: elasticsearch dockerfile: Dockerfile environment: - cluster.name=docker-elk-cluster - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms256m -Xmx256m" ulimits: memlock: soft: -1 hard: -1 ports: - 9200:9200

 

 

1.Elasticsearch

文件elasticsearch/Dockerfile

FROM docker.elastic.co/elasticsearch/elasticsearch:6.5.2 COPY --chown=elasticsearch:elasticsearch elasticsearch.yml /usr/share/elasticsearch/config/ CMD ["elasticsearch", "-Elogger.level=INFO"]

 

文件elasticsearch/elasticsearch.yml

cluster.name: ${cluster.name} network.host: 0.0.0.0

# minimum_master_nodes need to be explicitly set when bound on a public IP # set to 1 to allow single node clusters # Details: https://github.com/elastic/elasticsearch/pull/17288
discovery.zen.minimum_master_nodes: 1

 

 

2.Logstash

文件logstash/Dockerfile

FROM docker.elastic.co/logstash/logstash:6.5.2 RUN rm -f /usr/share/logstash/pipeline/logstash.conf COPY pipeline /usr/share/logstash/pipeline/

 

文件logstash/pipeline/logstash.conf

input { beats { port => 5044 host => "0.0.0.0" } } output { elasticsearch { hosts => elasticsearch manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" } stdout { codec => rubydebug } }

 

 

 

3.Filebeat

文件filebeat/Dockerfile

FROM docker.elastic.co/beats/filebeat:6.5.2

# Copy our custom configuration file
COPY filebeat.yml /usr/share/filebeat/filebeat.yml USER root # Create a directory to map volume with all docker log files
RUN mkdir /usr/share/filebeat/dockerlogs RUN chown -R root /usr/share/filebeat/ RUN chmod -R go-w /usr/share/filebeat/

 

文件filebeat/filebeat.yml

filebeat.inputs: - type: docker combine_partial: true containers: path: "/usr/share/dockerlogs/data" stream: "stdout" ids: - "*" exclude_files: ['\.gz$'] ignore_older: 10m processors: # decode the log field (sub JSON document) if JSON encoded, then maps it's fields to elasticsearch fields
- decode_json_fields: fields: ["log", "message"] target: ""
    # overwrite existing target elasticsearch fields while decoding json fields 
    overwrite_keys: true
- add_docker_metadata: host: "unix:///var/run/docker.sock" filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false

# setup filebeat to send output to logstash
output.logstash: hosts: ["logstash"] # Write Filebeat own logs only to file to avoid catching them with itself in docker log files
logging.level: error logging.to_files: false logging.to_syslog: false loggins.metrice.enabled: false logging.files: path: /var/log/filebeat name: filebeat keepfiles: 7 permissions: 0644 ssl.verification_mode: none

 

 

 

使用docker-compose up -d跑起來

 

 

 

 

 

在瀏覽器打開[Host-IP]:9200，可以打開如下界面，說明elasticsearch服務已經起來了

 

 

再在瀏覽器打開[Host-IP]:5601，是Kibana日誌查看平臺

 

進入至系統菜單【管理】中的【index-pattern】

 

 

首次使用Kibana須要先建立index-pattern，建立index-pattern操做以下，若是在Discover菜單中建立index-pattern時，會出現如下

 

建立了index-pattern後，查看Logs應該能夠查看到日誌

 

首頁查看日誌

 

 

 

源碼地址：https://github.com/ChenWes/docker-elk