ELK集羣模式部署

架構拓撲圖爲：

 

準備工做：

下載資源包：

Elasticsearch： wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz  # 這一步用docker啓動，能夠不用下載。

Kibana: wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.4-linux-x86_64.tar.gz

Logstash:wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.4.tar.gz

Filebeat:wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-linux-x86_64.tar.gz

安裝：

根據拓撲圖把對應的服務安裝在對應的服務器。

tar xvf kibana-6.2.3-linux-x86_64.tar.gz -C /usr/local/

tar xvf logstash-6.2.4.tar.gz -C /usr/local/

tar -xvf filebeat-6.2.4-linux-x86_64.tar.gz -C /usr/local/

部署啓動:

Elasticsearch-master(192.168.11.1)：

需自行提早安裝好docker 和 docker-compose。 

vim docker-compose.yml

version: '2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.2.3
    container_name: elasticsearch
    environment:
      - cluster.name=es-cluster
      - bootstrap.memory_lock=true
      - "network.publish_host=192.168.11.1"
      - "ES_JAVA_OPTS=-Xms4096m -Xmx4096m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /data/elasticsearch_data:/usr/share/elasticsearch/data
    ports:
      - "192.168.11.1:9200:9200"
      - "192.168.11.1:9300:9300"

mkdir -p /data/elasticsearch_data && chmod 775 /data/elasticsearch_data && docker-compose up -d 

 

Elasticsearch-node2.data(192.168.11.2)：

需自行提早安裝好docker 和 docker-compose。 

vim docker-compose.yml

version: '2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.2.3
    container_name: elasticsearch
    environment:
      - cluster.name=es-cluster
      - bootstrap.memory_lock=true
      - "network.publish_host=192.168.11.2"
      - "discovery.zen.ping.unicast.hosts=192.168.11.1"
      - "ES_JAVA_OPTS=-Xms4096m -Xmx4096m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /data/elasticsearch_data:/usr/share/elasticsearch/data
    ports:
      - "192.168.11.2:9200:9200"
      - "192.168.11.2:9300:9300"

mkdir -p /data/elasticsearch_data && chmod 775 /data/elasticsearch_data && docker-compose up -d 

Elasticsearch-node3.data(192.168.11.3)：

需自行提早安裝好docker 和 docker-compose。 

vim docker-compose.yml

version: '2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.2.3
    container_name: elasticsearch
    environment:
      - cluster.name=es-cluster
      - bootstrap.memory_lock=true
      - "network.publish_host=192.168.11.3"
      - "discovery.zen.ping.unicast.hosts=192.168.11.1"
      - "ES_JAVA_OPTS=-Xms4096m -Xmx4096m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /data/elasticsearch_data:/usr/share/elasticsearch/data
    ports:
      - "192.168.11.3:9200:9200"
      - "192.168.11.3:9300:9300"

mkdir -p /data/elasticsearch_data && chmod 775 /data/elasticsearch_data && docker-compose up -d 

查看集羣節點：curl -XGET 'http://192.168.11.1:9200/_cat/nodes?pretty'

 

 kibana(192.168.11.4)：

cd /usr/local/kibana-6.2.3-linux-x86_64

cp ./config/kibana.yml ./config/kibana.yml.bak

echo '' > ./config/kibana.yml

vim ./config/kibana.yml

server.port: 5601
server.host: "192.168.11.4"
elasticsearch.url: "http://192.168.11.1:9200"

啓動：./bin/kibana

擴展包（可裝可不裝）：

kibana 報警擴展: 

安裝 ./bin/kibana-plugin install https://github.com/sirensolutions/sentinl/releases/download/tag-6.2.3-2/sentinl-v6.2.3.zip

重啓 kibana

logstash(192.168.11.5)

cd /usr/local/logstash-6.2.4

vim nginx.conf

input {
beats {
port => 5066
host => "192.168.11.5"
codec => "json"
}
}
filter {

mutate {

gsub => ["message", "\\x", "\\\x"]

}

json {

source => "message"

}

}
output {
    elasticsearch {
        action => "index"
        hosts => "192.168.11.1:9200"
        index => "nginx-json-%{+YYYY.MM.dd}"
   }
}

mkdir -p /data/logstash-data/nginx

啓動：./bin/logstash -f ./nginx.conf --path.data=/data/logstash-data/nginx &

擴展

grok調試地址：http://grokdebug.herokuapp.com/

如需調試請修改output成如下，方便debug

output {
        stdout {
        codec    => rubydebug
    }
}

 

filebeat(192.168.11.6)

cd /usr/local/filebeat-6.2.4-linux-x86_64

vim nginx.yml

filebeat.prospectors:
- input_type: log
  document_type: jsonlog
  paths:
    - /var/log/nginx/access.log
output:
  logstash:
    hosts: ["192.168.11.5:5066"]

啓動：./filebeat -c ./nginx.yml &

nginx 日誌的輸出格式爲：

log_format json '{"@timestamp":"$time_iso8601",'
               '"remote_addr":"$remote_addr",'
               '"request":"$request",'
               '"status":$status,'
               '"body_bytes":$body_bytes_sent,'
               '"user_agent":"$http_user_agent",'
               '"resp_time":"$upstream_response_time",'
               '"req_time":$request_time,'
               '"host":"$host",'
               '"@version":"1",'
               '"http_x_forwarded_for":"$http_x_forwarded_for",'
               '"upstream_addr":"$upstream_addr",'
               '"req_body":"$request_body"}';

參考地址：

https://www.elastic.co/guide/index.html 

http://docs.docker.com/compose/

https://www.elastic.co/guide/en/logstash/current/index.html

https://www.elastic.co/guide/en/elasticsearch/reference/6.2/docker.html

https://www.elastic.co/guide/en/kibana/current/install.html