ELK 原理&部署過程

時間 2019-12-06

標籤 elk 原理部署過程简体版

原文原文鏈接

一圖勝千言

基礎架構

工做原理

Logstash工做原理

Logstash工做流程

ELK總體部署圖

ELK 安裝配置簡化過程

1 基本配置
    vim /etc/hosts
    192.168.2.61         master-node
    192.168.2.62       data-node1
    192.168.2.63       data-node2
    wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.0.0.rpm
    rpm -ivh elasticsearch-6.0.0.rpm

    elasticsearch.yml
    jvm.options
    log4j2.properties

    vim /etc/elasticsearch/elasticsearch.yml 
    cluster.name: master-node  # 集羣中的名稱
    node.name: master  # 該節點名稱
    node.master: true  # 意思是該節點爲主節點
    node.data: false  # 表示這不是數據節點
    network.host: 0.0.0.0  # 監聽所有ip，在實際環境中應設置爲一個安全的ip
    http.port: 9200  # es服務的端口號
    discovery.zen.ping.unicast.hosts: ["192.168.2.61", "192.168.2.62", "192.168.2.63"] # 配置自動發現

    scp /etc/elasticsearch/elasticsearch.yml data-node1:/tmp/
    scp /etc/elasticsearch/elasticsearch.yml data-node2:/tmp/

    cp /tmp/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml
    systemctl start elasticsearch.service
    curl '192.168.2.61:9200/_cluster/health?pretty'
    curl '192.168.2.61:9200/_cluster/state?pretty'
2 kibana配置
    wget https://artifacts.elastic.co/downloads/kibana/kibana-6.0.0-x86_64.rpm
    rpm -ivh kibana-6.0.0-x86_64.rpm

    vim /etc/kibana/kibana.yml
    server.port: 5601  # 配置kibana的端口
    server.host: 192.168.2.61  # 配置監聽ip
    # 配置es服務器的ip，若是是集羣則配置該集羣中主節點的ip
    elasticsearch.url: "http://192.168.2.61:9200" 
    # 配置kibana的日誌文件路徑，否則默認是messages裏記錄日誌 
    logging.dest: /var/log/kibana.log  

    touch /var/log/kibana.log; chmod 777 /var/log/kibana.log
    systemctl start kibana
    http://192.168.2.61:5601/ 
3 logstash配置
    192.168.2.62
    wget https://artifacts.elastic.co/downloads/logstash/logstash-6.0.0.rpm
    rpm -ivh logstash-6.0.0.rpm

    vim /etc/logstash/conf.d/syslog.conf 
    input {  # 定義日誌源
                        syslog {
                                            type => "system-syslog"  # 定義類型
                                            port => 10514    # 定義監聽端口
                        }
    }
    output {  # 定義日誌輸出
                        stdout {
                                            codec => rubydebug  # 將日誌輸出到當前的終端上顯示
                        }
    }

    cd /usr/share/logstash/bin
    檢查配置
    ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit

    配置kibana服務器的ip以及配置的監聽端口
    vim /etc/rsyslog.conf
    #### RULES ####

    *.* @@192.168.2.62:10514
    systemctl restart rsyslog

    指定配置文件，啓動logstash
    cd /usr/share/logstash/bin
    ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf
    
logstash收集nginx日誌
vim /etc/logstash/conf.d/nginx.conf 

input {
  file {  # 指定一個文件做爲輸入源
    path => "/var/log/nginx/access.log"  # 指定文件的路徑
    start_position => "beginning"  # 指定什麼時候開始收集
    type => "nginx"  # 定義日誌類型，可自定義
  }
}
filter {  # 配置過濾器
    grok {
        match => { "message" => "%{IPORHOST:http_host} %{IPORHOST:clientip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:response} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time:float}"}  # 定義日誌的輸出格式
    }
    geoip {
        source => "clientip"
    }
}
output {
    stdout { codec => rubydebug }
    elasticsearch {
        hosts => ["192.168.2.61:9200"]
        index => "nginx-test-%{+YYYY.MM.dd}"
  }
}

cd /usr/share/logstash/bin
./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/nginx.conf --config.test_and_exit

cd /etc/nginx/http_virtual_host.d
vim elk.conf
server {
      listen 80;
      server_name elk.test.com;

      location / {
          proxy_pass      http://192.168.2.61:5601;
          proxy_set_header Host   $host;
          proxy_set_header X-Real-IP      $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      }

      access_log  /tmp/elk_access.log main2;
}

vim 
log_format main2 '$http_host $remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$upstream_addr" $request_time';
nginx -t
nginx -s reload
配置hosts  192.168.2.62 elk.eichong.com

ls /var/log/nginx/access.log
wc -l !$
重啓logstash服務，生成日誌的索引
systemctl restart logstash

重啓完成後，在es服務器上檢查是否有nginx-test開頭的索引生成
curl '192.168.2.61:9200/_cat/indices?v'

nginx-test索引已經生成了，那麼這時就能夠到kibana上配置該索引
managent  index patterns   create index patterns
discover
http://192.168.2.61:5601/status 查看狀態

主要配置文件

001 elasticsearch
cat /etc/elasticsearch/elasticsearch.yml |grep ^[^#]

cluster.name: my-elk
node.name: master
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["172.19.1.216", "172.19.1.217"]

002 kibana
cat /etc/kibana/kibana.yml |grep ^[^#]

server.port: 5601
server.host: "172.19.1.216"
elasticsearch.url: "http://172.19.1.216:9200"
logging.dest: /var/log/kibana.log  # 文件需建立並受權

003 logstash

漢化

https://github.com/anbai-inc/Kibana_Hanization

其餘優秀博客

https://www.cnblogs.com/kevingrace/p/5919021.html
http://blog.51cto.com/zero01/2079879

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。

ELK 原理&部署過程

一圖勝千言

ELK 安裝配置簡化過程

最新版本yum安裝

主要配置文件

漢化

其餘優秀博客