ELK5.6.4+Redis+Filebeat+Nginx(CentOS7.4)

下載地址:

Elasticsearhc:  https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.4.tar.gz
Logstash:  https://artifacts.elastic.co/downloads/logstash/logstash-5.6.4.tar.gz
Kibana:   https://artifacts.elastic.co/downloads/kibana/kibana-5.6.4-x86_64.rpm
Filebeat:  https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.4-x86_64.rpm
Redis:    v3.2.10
Nginx

主機

Elasticsearch/kibana/Nginx: 192.168.1.106
Redis: 192.168.1.107
Logstash 192.168.1.108
Filebeat: 192.168.1.109

1、環境準備

1.jdk安裝 Elastic須要Java8環境(logstash和elastic主機安裝)
2.修改文件描述符和單個用戶進程數量
3.cat /etc/security/limits.conf
*    soft    nofile      65536
*    hard    nofile      65536
*    soft    nproc       65536
*    hard    nproc       65536
4.cat /etc/security/limits.d/20-nproc.conf
*          soft    nproc     65536
root       soft    nproc     unlimited
#修改進程能夠擁有的虛擬內存區域數量
5.echo 「vm.max_map_count=655360」 >> /etc/sysctl.conf
6.sysctl -p

Elasticsearch安裝

1.useradd op
2.mkdir /opt/op && mkdir /data
3.chown -R op:op /opt/op && chown -R op:op /data
4.cd /opt/op
5.su - op
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.4.tar.gz
6.tar xf elasticsearch-5.6.4.tar.gz
7.mv elasticsearch-5.6.4 elasticsearch
8.cd elasticsearch
9.vim conf/elasticsearch.yml
#綁定監聽IP
network.host: 192.168.1.106
#設置對外提供http服務的端口,默認是9200
http.port: 9002
#設置集羣名字
cluster.name: buka_es_test
#節點名字
node.name: node1
#數據存儲位置
path.data: /data
#日誌存儲目錄
path.logs: /data/logs
#鎖定內存
bootstrap.mlockall: true
10. 啓動
bin/elasticsearch -d

kibana安裝

1.wget https://artifacts.elastic.co/downloads/kibana/kibana-5.6.4-x86_64.rpm
2.rpm -ivh kibana-5.6.4-x86_64.rpm
3.vim /etc/kibana/kibana.yml
server.port: 5601
server.host: "192.168.1.106"
elasticsearch.url: "http://192.168.1.106:9002"
4.啓動
/etc/init.d/kibana start
systemctl enable kibana

Logstash安裝

1.wget https://artifacts.elastic.co/downloads/logstash/logstash-5.6.4.tar.gz
2.tar xf logstash-5.6.4.tar.gz
3.cd logstash-5.6.4
4.cat config/server.conf
input {
       redis {
               host => "192.168.1.108"
               port => "6379"
               data_type => 'list'
               key => "srs"   
       }
}
filter {
  if "srs" in [type]{
        grok {
            match => ["message","\[%{TIMESTAMP_ISO8601:logtime}\]\[%{WORD:level}\]\[%{INT:id1}\]\[%{INT:id2}\] source url=%{WORD:app}\/%{WORD:root}\/%{DATA:stream_id}\, ip=%{IP:source_ip}\,"]
        }
            date {
                match => [ "logtime", "yyyy-MM-dd HH:mm:ss,SSS","yyyy-MM-dd HH:mm:ss","yyyy-MM-dd HH:mm:ss:SSS" ]
                target => "@timestamp"
            }
  }
}

output {
   if "srs" in [type]{ 
      elasticsearch {
         hosts => "192.168.1.106:9002"
         index => "logstash-srs-%{+YYYY.MM.dd}"
      }
   }
}
5.啓動
nohup ./bin/logstash -f config/server.conf &

redis安裝

1.yum -y install redis
2.vim /etc/redis.conf
  bind 127.0.0.1 192.168.1.108
3.systemctl start redis && systemctl enable redis

Filebeat安裝

1.wget  https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.0.0-x86_64.rpm
2.rpm -ivh filebeat-6.0.0-x86_64.rpm
3.vim /etc/filebeat/filebeat.yml
filebeat.prospectors:
- type: log
  paths:
    - /opt/srs/logs/srs.log
  document_type: srs
output.redis:
  hosts: ["192.168.1.108"]
  port: 6379
  key: "rtc"
4.systemctl start filebeat && systemctl enable filebeat

Nginx反向代理kibana並增長認證功能

1. yum -y install nginx httpd
2. 使用命令生成passfile 文件： htpasswd -c -d /etc/nginx/pass_file  admin
    #輸入以上命令，回車輸入密碼，再次回車，輸入確認密碼
3. nginx.conf 配置文件        
location / {
            auth_basic "Restricted";
            auth_basic_user_file /etc/nginx/pass_file;
            proxy_pass http://192.168.1.106:5601;
            proxy_set_header Host $host:5601;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
4. systemctl start nginx && systemctl enable nginx

訪問kibana

http://192.168.1.106