docker-compose部署ELK

本章基於

http://www.javashuo.com/article/p-ubnecgbn-bc.html

在此基礎上將ELK系統docker-compose.yml化。

其docker-compose 須要注意

1.不要把 docker 當作數據容器來使用，數據必定要用 volumes 放在容器外面

2.不要把 docker-compose 文件暴露給別人， 由於上面有你的服務器信息

3.多用 docker-compose 的命令去操做， 不要用 docker 手動命令&docker-compose 去同時操做

4.寫一個腳本類的東西，自動備份docker 映射出來的數據。

5.不要把全部服務都放在一個 docker 容器裏面

 

準備環境：

管理節點10.191.51.44

數據節點 10.191.51.45/46/47

 

具體文件：

es docker-compose.yml

version: '2' services: elasticsearch: container_name: ES environment : - ES_JAVA_OPTS=-Xms4G -Xmx4G image: 10.191.51.5/elk/elasticsearch:6.5.4 volumes: - ./config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml - ./data:/usr/share/elasticsearch/data ports: - "9200:9200"
      - "9300:9300"

View Code

管理節點 elasticsearch.yml

cluster.name: elasticsearch-cluster node.name: es-node1 network.bind_host: 0.0.0.0 network.publish_host: 10.191.51.44 http.port: 9200 transport.tcp.port: 9300 http.cors.enabled: true http.cors.allow-origin: "*" node.master: false node.data: false node.ingest: true discovery.zen.ping.unicast.hosts: ["10.191.51.44:9300","10.191.51.45:9300","10.191.51.46:9300","10.191.51.47:9300"] discovery.zen.minimum_master_nodes: 2

View Code

數據節點 elasticsearch.yml

cluster.name: elasticsearch-cluster node.name: es-node2 network.bind_host: 0.0.0.0 network.publish_host: 10.191.51.45 http.port: 9200 transport.tcp.port: 9300 http.cors.enabled: true http.cors.allow-origin: "*" node.master: true node.data: true discovery.zen.ping.unicast.hosts: ["10.191.51.44:9300","10.191.51.45:9300","10.191.51.46:9300","10.191.51.47:9300"] discovery.zen.minimum_master_nodes: 2

View Code

 

kafka docker-compose.yml （其中environment須要配置）

version: '2' services: kafka: container_name: kafka0 environment: - KAFKA_BROKER_ID=0
      - KAFKA_ZOOKEEPER_CONNECT=10.191.51.44:2181
      - KAFKA_DEFAULT_REPLICATION_FACTOR=3
      - KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092
      - KAFKA_ADVERTISED_HOST_NAME=kafka1 - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://10.191.51.45:9092
      - KAFKA_delete_topic_enable=true image: 10.191.51.5/elk/wurstmeister/kafka:2.1.1 ports: - "9092:9092"

View Code

 

logstash docker-compose.yml

version: '2' services: logstash: container_name: logstash image: 10.191.51.5/elk/logstash:6.5.4 volumes: - ./config/:/usr/share/logstash/config/
      - ./pipeline/:/usr/share/logstash/pipeline/ ports: - "5044:5044"
      - "9600:9600"

View Code

pipeline/logstash.conf 

 kafka{ bootstrap_servers => ["10.191.51.45:9092,10.191.51.46:9092,10.191.51.47:9092"] client_id => "logstash-garnet" group_id => "logstash-garnet" consumer_threads => 8 decorate_events => true topics => ["garnet_garnetAll_log"] type => "garnet_all_log" } } filter{ if[type]=="garnet_all_log"{ mutate{ gsub => ["message", "@timestamp", "sampling_time"] } json{ source=>"message" } grok{ match=>{ "message"=>[ "%{TIMESTAMP_ISO8601:log_time}\s+\[(?<thread_name>[a-zA-Z0-9\-\s]*)\]\s+%{LOGLEVEL:log_level}\s+\[(?<class_name>[a-zA-Z0-9.]*)\]\s+%{GREEDYDATA:msg_info}parameters=%{GREEDYDATA:msg_json_info}", "%{TIMESTAMP_ISO8601:log_time}\s+\[(?<thread_name>[a-zA-Z0-9\-\s]*)\]\s+%{LOGLEVEL:log_level}\s+\[(?<class_name>[a-zA-Z0-9.]*)\]\s+%{GREEDYDATA:msg_info}" ] } } if [msg_json_info]{ json{ source=>"msg_json_info" } } } } output { if[type] == "garnet_all_log"{ elasticsearch{ hosts => ["10.191.51.45:9200","10.191.51.46:9200","10.191.51.47:9200"] index => "garnet_all-%{+YYYY.MM.dd}" } } }

View Code

config/pipeline.yml

- pipeline.id: pipeline_1 pipeline.batch.size: 200 pipeline.batch.delay: 1 path.config: /usr/share/logstash/pipeline

View Code

config/logstash.yml

log.level: warn xpack.license.self_generated.type: basic xpack.monitoring.enabled: true xpack.monitoring.elasticsearch.url: "http://10.191.51.44:9200"

View Code

config/jvm.options 修改

-Xms4g -Xmx4g

View Code