Elasticsearch+Kibana+Logstash安裝

安裝環境：

[root@node-1 src]# cat /etc/redhat-release 
CentOS Linux release 7.5.1804 (Core) 

安裝以前關閉防火牆 firewalld 和 selinux：

[root@node-1 logs]# systemctl stop firewalld
[root@node-1 logs]# setenforce 0

安裝流程：

Kibana->Elasticsearch->Logstash

1、安裝運行所需的Java環境，Elasticsearch、Logstash依賴於java環境,使用官方的二進制包解壓安裝，先下載java linux 64位tar.gz包，java 1.8的下載連接： 

http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

配置JAVA環境：

[root@node-1 src]# cd /usr/local/src
[root@node-1 src]# tar xf jdk-8u191-linux-x64.tar.gz 
[root@node-1 src]# mv jdk1.8.0_191 /usr/local

#用全路徑驗證java是否安裝成功
/usr/local/jdk1.8.0_191/bin/java -version

#配置java環境變量
vim /etc/profile加入
export JAVA_HOME=/usr/local/jdk1.8.0_191/
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$CLASSPATH

#環境變量生效
source /etc/profile

#java版本查看
[root@node-1 ~]# java -version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)

安裝Kibana：

#kibana下載地址(kibana主要用來展示數據，它自己不存儲數據)
https://artifacts.elastic.co/downloads/kibana/kibana-6.2.3-linux-x86_64.tar.gz

#準備工做，添加elk用戶，用elk用戶來啓動elk
useradd elk
usermod -s /sbin/nologin elk    #不讓elk用戶來登陸系統
#解壓安裝kibana:
tar -zxf kibana-6.2.3-linux-x86_64.tar.gz
mv kibana-6.2.3-linux-x86_64 /usr/local/kibana-6.2.3

#kibana配置文件
vim /usr/local/kibana-6.2.3/config/kibana.yml修改：
server.port: 5601
server.host: "0.0.0.0"(監聽在全部網卡，有風險)
#elasticsearch.url: "http://localhost:9200" (默認是鏈接elasticsearch的9200端口)
#elasticsearch.username: "user" (配置鏈接elasticsearch的用戶名和密碼)
#elasticsearch.password: "pass"

#把kibana目錄改成elk用戶
chown -R elk:elk /usr/local/kibana-6.2.3/

#新增啓動腳本vim /usr/local/kibana-6.2.3/bin/start.sh
nohup /usr/local/kibana-6.2.3/bin/kibana >>/tmp/kibana.log 2>>/tmp/kibana.log &

chmod a+x /usr/local/kibana-6.2.3/bin/start.sh

#用普通用戶啓動
su -s /bin/bash elk '/usr/local/kibana-6.2.3/bin/start.sh'

訪問kibana，若有防火牆須要放開tcp 5601端口

 Nginx限制訪問kibana：

默認的kibana是沒有任何的權限控制，先把kibana改到監聽127.0.0.1，藉助nginx來限制訪問

1:藉助nginx來限制訪問，控制源ip的訪問
worker_processes  1;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    sendfile        on;
    keepalive_timeout  65;
    server {
       listen       5609;
       access_log  /usr/local/nginx/logs/kibana_access.log main;
       error_log /usr/local/nginx/logs/kibana_error.log error;
       location / {
           allow 127.0.0.1;
           deny all;
           proxy_pass http://127.0.0.1:5601;
       }
    }
}

能夠在日誌裏面找到源ip地址：tail -f /usr/local/nginx/logs/kibana_access.log

2: 若是ip常常變化，就會很麻煩。nginx支持簡單的用戶名密碼認證。
location / {
            auth_basic "elk auth";
            auth_basic_user_file /usr/local/nginx/conf/htpasswd;
            proxy_pass http://127.0.0.1:5601;
        }

printf "elk:$(openssl passwd -1 elkpass)\n" >/usr/local/nginx/conf/htpasswd

3: nginx源碼編譯安裝腳本
if [ -d "/usr/local/nginx/" ];then
    echo "nginx is install"
    exit 1
else
    echo "nginx in not install"
fi

for softpack in wget tar gcc gcc-c++ make pcre pcre-devel zlib zlib-devel openssl openssl-devel;do
soft_result=`rpm -qa $softpack`
if [ -z "$soft_result" ];then
echo "${softpack} is not exist,install it"
yum -y install ${softpack}
else
echo "${softpack} is exist"
fi
done

cd /usr/local/src
wget 'http://nginx.org/download/nginx-1.12.2.tar.gz'
tar -zxvf nginx-1.12.2.tar.gz
cd nginx-1.12.2
./configure --prefix=/usr/local/nginx --with-http_ssl_module --with-stream  --with-stream_ssl_module
make
make install
exit 0

ln -sf /usr/local/nginx/sbin/nginx /usr/local/bin/

elasticsearch安裝配置：

elasticsearch未安裝以前，kibana網頁上報錯，提示找不到elasticsearch。

1: elasticsearch的下載地址(elasticsearch主要用來存儲數據，供kibana調取並進行展示)
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.3.tar.gz

解壓安裝：
cd /usr/local/src/
tar -zxf elasticsearch-6.2.3.tar.gz
mv elasticsearch-6.2.3 /usr/local/

2: elasticsearch配置
vim /usr/local/elasticsearch-6.2.3/config/elasticsearch.yml 修改：
path.data: /usr/local/elasticsearch-6.2.3/data
path.logs: /usr/local/elasticsearch-6.2.3/logs
network.host: 127.0.0.1
http.port: 9200
bootstrap.memory_lock: false
bootstrap.system_call_filter: false

3: 把elasticsearch目錄的用戶和屬主都更新爲elk
chown -R elk:elk /usr/local/elasticsearch-6.2.3/

4: 更改jvm的內存限制(看我的配置)
vim /usr/local/elasticsearch-6.2.3/config/jvm.options
-Xms100M
-Xmx100M

5: 編輯elasticsearch啓動腳本，使用-d進行後臺啓動。elasticsearch
vim /usr/local/elasticsearch-6.2.3/bin/start.sh
/usr/local/elasticsearch-6.2.3/bin/elasticsearch -d

chmod a+x /usr/local/elasticsearch-6.2.3/bin/start.sh

6: 啓動elasticsearch
su -s /bin/bash elk '/usr/local/elasticsearch-6.2.3/bin/start.sh'
觀察日誌
觀察kibana網頁，看下還會不會報elasticsearch的錯誤

7: elasticsearch若是監聽在非127.0.0.1，須要配置內核參數等
network.host: 0.0.0.0

vim /etc/security/limits.conf(處理max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536])
* soft nofile 65536
* hard nofile 65536

vim /etc/security/limits.d/20-nproc.conf(處理max number of threads [3885] for user [elk] is too low, increase to at least [4096])
*          soft    nproc     10240
*          hard    nproc     10240

sysctl.conf添加(處理max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144])
vm.max_map_count = 262144    #須要運行sysctl -p生效

Logstash安裝配置：

1: logstash的下載地址(用來讀取日誌，正則分析日誌，發送給elasticsearch數據庫)
https://artifacts.elastic.co/downloads/logstash/logstash-6.2.3.tar.gz

解壓安裝：
tar -zxf logstash-6.2.3.tar.gz
mv logstash-6.2.3 /usr/local/
ll -h /usr/local/logstash-6.2.3

2: 更改logstash jvm配置vim /usr/local/logstash-6.2.3/config/jvm.options 
-Xms150M
-Xmx150M

3: logstash配置 vim /usr/local/logstash-6.2.3/config/logstash.conf
input {
  file {
    path => "/usr/local/nginx/logs/kibana_access.log"
  }
}
output {
  elasticsearch {
    hosts => ["http://127.0.0.1:9200"]
  }
}

4: logstash的啓動腳本：
vim /usr/local/logstash-6.2.3/bin/start.sh
nohup /usr/local/logstash-6.2.3/bin/logstash -f /usr/local/logstash-6.2.3/config/logstash.conf >>/tmp/logstash.log 2>>/tmp/logstash.log &

chmod a+x  /usr/local/logstash-6.2.3/bin/start.sh

5: 啓動logstash
/usr/local/logstash-6.2.3/bin/start.sh

logstash的啓動時間會有點慢，等啓動事後查看kibana的界面，會有能夠建立索引的地方。