利用docker-compose搭建ELK5.0

時間 2020-08-02

標籤利用 docker compose 搭建 elk5.0 elk 欄目 Docker 简体版

原文原文鏈接

1、搭建環境

172.19.2.51：elasticsearch+kibana+logstash+kopf
172.19.2.50：elasticsearch+nginx+filebeat
172.19.2.49：elasticsearch

其中nginx的訪問日誌爲咱們要採集的內容，用filebeat傳輸，因此nginx和filebeat都沒有在docker中運行node

其餘全部組件都在docker中運行，版本爲5nginx

2、172.19.2.51安裝elk組件

一、安裝docker-compose

curl -L https://github.com/docker/compose/releases/download/1.3.1/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
vim /etc/profile
export PATH="$PATH:/usr/local/bin"
source /etc/profile
echo $PATH

二、調整單進程的虛擬內存數，若是不調啓動容器會報錯

sysctl -a | grep vm.max_map_count
sysctl -w vm.max_map_count=262144

三、建立配置文件目錄和文件

建立elasticsearch數據存儲目錄git

mkdir -pv /root/elk/elasticsearch

建立elasticsearch配置文件目錄github

mkdir -pv /root/elk/es

建立kibana配置文件目錄docker

mkdir -pv /root/elk/kibana

建立logstash配置文件目錄json

mkdir -pv /root/elk/logstash

建立elasticsearch配置文件vim

vim /root/elk/es/elasticsearch.yml
network.bind_host: 0.0.0.0
network.host: 172.19.2.51
cluster.name: es-cluster
node.name: "es-node1"
node.master: true
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.unicast.hosts:
   -  172.19.2.51
   -  172.19.2.50
   -  172.19.2.49

建立kibana配置文件markdown

vim /root/elk/kibana/kibana.yml
port: 5601
host: "0.0.0.0"
elasticsearch_url: "http://172.19.2.50:9100"
elasticsearch_preserve_host: true
kibana_index: ".kibana"
default_app_id: "discover"
request_timeout: 300000
shard_timeout: 0
verify_ssl: true
bundled_plugin_ids:
 - plugins/dashboard/index
 - plugins/discover/index
 - plugins/doc/index
 - plugins/kibana/index
 - plugins/markdown_vis/index
 - plugins/metric_vis/index
 - plugins/settings/index
 - plugins/table_vis/index
 - plugins/vis_types/index
 - plugins/visualize/index

建立logstash配置文件app

vim /root/elk/logstash/logstash.conf
input {
  beats {
        port => 20000
        codec => "json"
    }
}

output {
  elasticsearch {
    hosts => "172.19.2.50:9100"
    index => "nginx" }
}

建立docker-compose配置文件curl

vim /root/elk/docker-compose.yml
elasticsearch:
  image: elasticsearch:5
  command: elasticsearch
  environment:
    - "ES_JAVA_OPTS=-Xmx1g -Xms1g"
  volumes:
    - ./elasticsearch:/usr/share/elasticsearch/data
    - ./es/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
  ports:
    - "9200:9200"
    - "9300:9300"

logstash:
  image: logstash:latest
  command: logstash -w 4 -f /etc/logstash/conf.d/logstash.conf
  environment:
    - LS_HEAP_SIZE=2048m
  volumes:
    - ./logstash/logstash.conf:/etc/logstash/conf.d/logstash.conf
  ports:
    - "20000:20000"

kibana:
  image: kibana:latest
  volumes:
    - ./kibana/kibana.yml:/etc/kibana/kibana.yml
  ports:
    - "5601:5601"

kopf:
  image: lmenezes/elasticsearch-kopf
  ports:
    - "80:80"
  environment:
    - KOPF_SERVER_NAME=kopf
    - KOPF_ES_SERVERS=172.19.2.50:9100

四、啓動docker-compose

cd /root/elk
docker-compose up
docker-compose ps

3、172.19.2.51安裝elasticsearch和nginx+filebeat

一、安裝docker-compose

curl -L https://github.com/docker/compose/releases/download/1.3.1/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
vim /etc/profile
export PATH="$PATH:/usr/local/bin"
source /etc/profile
echo $PATH

二、調整單進程的虛擬內存數

sysctl -a | grep vm.max_map_count
sysctl -w vm.max_map_count=262144

三、建立配置文件目錄和文件

建立elasticsearch數據存儲目錄

mkdir -pv /root/elk/elasticsearch

建立elasticsearch配置文件目錄

mkdir -pv /root/elk/es

建立elasticsearch配置文件

vim /root/elk/es/elasticsearch.yml
network.bind_host: 0.0.0.0
network.host: 172.19.2.50
cluster.name: es-cluster
node.name: "es-node2"
node.master: true
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.unicast.hosts:
   -  172.19.2.51
   -  172.19.2.50 
   -  172.19.2.49

建立docker-compose配置文件

vim /root/elk/docker-compose.yml
elasticsearch:
  image: elasticsearch:5
  command: elasticsearch
  environment:
    - "ES_JAVA_OPTS=-Xmx1g -Xms1g"
  volumes:
    - ./elasticsearch:/usr/share/elasticsearch/data
    - ./es/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
  ports:
    - "9200:9200"
    - "9300:9300"

修改nginx配置文件（此nginx用來返帶elasticsearch集羣的9200端口至9100，即es集羣的3臺主機的9200端口都經過172.19.2.50:9200訪問，同時咱們採集此nginx的80端口訪問日誌）

vim /etc/nginx/nginx.conf
user  nginx;
worker_processes  1;
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
events {
    worker_connections  1024;
}
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    log_format  logstash_json  '{ "@timestamp": "$time_local", '
                               '"@fields": { '
                               '"remote_addr": "$remote_addr", '
                               '"remote_user": "$remote_user", '
                               '"body_bytes_sent": "$body_bytes_sent", '
                               '"request_time": "$request_time", '
                               '"status": "$status", '
                               '"request": "$request", '
                               '"request_method": "$request_method", '
                               '"http_referrer": "$http_referer", '
                               '"body_bytes_sent":"$body_bytes_sent", '
                               '"http_x_forwarded_for": "$http_x_forwarded_for", '
                               '"http_user_agent": "$http_user_agent" } }';
    access_log  /var/log/nginx/access.log  logstash_json;
    sendfile        on;
    keepalive_timeout  65;

    upstream els {
        server 172.19.2.49:9200 weight=1 max_fails=2 fail_timeout=1;
        server 172.19.2.50:9200 weight=1 max_fails=2 fail_timeout=1;
        server 172.19.2.51:9200 weight=1 max_fails=2 fail_timeout=1;
        }

    server {
        listen       9100;
        access_log  /var/log/nginx/accessels.log  logstash_json;

        location / {
            proxy_pass   http://els/;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            }
        }

    include /etc/nginx/conf.d/*.conf;
}

四、安裝和配置filebeat

cd /root/
curl -L -O https://download.elastic.co/beats/filebeat/filebeat-1.3.0-x86_64.rpm
rpm -vi filebeat-1.3.0-x86_64.rpm
vim /etc/filebeat/filebeat.yml
filebeat:
  prospectors:
    -
      paths:
        - /var/log/nginx/access.log
      input_type: log
      multiline:
        negate: true
        match: after
      tail_files: false
  registry_file: /var/lib/filebeat/registry
output:
  logstash:
    hosts: ["172.19.2.51:20000"]
    worker: 4
shipper:
logging:
  files:
    rotateeverybytes: 10485760 # = 10MB

五、啓動docker-compose，啓動nginx，啓動filebeat

cd /root/elk
docker-compose up
service nginx start
service filebeat start

4、172.19.2.49安裝elasticsearch節點

一、安裝docker-compose

curl -L https://github.com/docker/compose/releases/download/1.3.1/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
vim /etc/profile
export PATH="$PATH:/usr/local/bin"
source /etc/profile
echo $PATH

二、調整單進程的虛擬內存數

sysctl -a | grep vm.max_map_count
sysctl -w vm.max_map_count=262144

三、建立配置文件目錄和文件

建立elasticsearch數據存儲目錄

mkdir -pv /root/elk/elasticsearch

建立elasticsearch配置文件目錄

mkdir -pv /root/elk/es

建立elasticsearch配置文件

vim /root/elk/es/elasticsearch.yml
network.bind_host: 0.0.0.0
network.host: 172.19.2.49
cluster.name: es-cluster
node.name: "es-node3"
node.master: true
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.unicast.hosts:
   -  172.19.2.51
   -  172.19.2.50 
   -  172.19.2.49

建立docker-compose配置文件

vim /root/elk/docker-compose.yml
elasticsearch:
  image: elasticsearch:5
  command: elasticsearch
  environment:
    - "ES_JAVA_OPTS=-Xmx1g -Xms1g"
  volumes:
    - ./elasticsearch:/usr/share/elasticsearch/data
    - ./es/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
  ports:
    - "9200:9200"
    - "9300:9300"