ELK6.3版本安裝部署

1、Elasticsearch 安裝

1、部署系統以及環境準備
cat /etc/redhat-release 
CentOS Linux release 7.4.1708 (Core) 
uname -r
3.10.0-693.el7.x86_64

#Firewalld and selinux
systemctl stop firewalld
systemctl disable firewalld
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0


sed -i 's/localhost.localdomain/ELK.localdomain/' /etc/hostname
hostnamectl set-hostname  ELK.localdomain


安裝jdk8以上版本
mkdir /application/
tar xf jdk-8u151-linux-x64.tar.gz -C /application/
ln -s /application/jdk1.8.0_151 /application/jdk
sed -i.ori '$a export JAVA_HOME=/application/jdk\nexport PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH\nexport CLASSPATH=.$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$JAVA_HOME/lib/tools.jar' /etc/profile
source /etc/profile

java -version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)


二、部署elasticsearch6.3.2
採用壓縮包解壓縮安裝方式啓動服務
建立另外獨立帳戶專供es使用
cd /usr/local
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.tar.gz
tar zxvf elasticsearch-6.3.2.tar.gz -C /application/
groupadd ela 建立ela組
useradd -g ela ela 建立ela用戶，而且加入ela組
passwd ela 爲ela用戶設定登陸密碼

ln -s /application/elasticsearch-6.3.2/ /application/elasticsearch
chown -R ela.ela /application/elasticsearch

配置文件
grep -n '^[a-Z]' /application/elasticsearch/config/elasticsearch.yml 
17:cluster.name: elk
23:node.name: node-1
33:path.data: /home/elkdata
37:path.logs: /var/log/elasticsearch
42:bootstrap.memory_lock: true
54:network.host: 0.0.0.0
58:http.port: 9200
88:http.cors.enabled: true 
89:http.cors.allow-origin: "*"


mkdir /home/elkdata/ -p
chown -R ela.ela /home/elkdata/
mkdir /var/log/elasticsearch -p
chown -R ela.ela /var/log/elasticsearch/



vim /etc/security/limits.conf #增長如下行
* soft nofile 65536
* hard nofile 131072
ela soft memlock unlimited
ela hard memlock unlimited

讓設置生效重啓系統或者另外開一個終端


vim /etc/sysctl.conf
#增長一行
vm.max_map_count = 655360 
sysctl -p

檢查文件權限是不是ela用戶
ls -l /application/elasticsearch-6.3.2/
ls -l /application/elasticsearch/
ls -l /application/elasticsearch/config/jvm.options 


切換到ela用戶，-d後臺運行
[ela@elk elasticsearch-6.3.2]$ ./bin/elasticsearch -d 

[ela@elk elasticsearch-6.3.2]$ netstat -ntpl|grep 9200
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6       0      0 :::9200                 :::*                    LISTEN      147438/java   

2、Kibana安裝

#下載
cd /usr/local/src
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-linux-x86_64.tar.gz
#解壓
tar xzvf kibana-6.3.2-linux-x86_64.tar.gz

ln -s /application/kibana-6.3.2-linux-x86_64 /application/kibana


vim /application/kibana/config/kibana.yml
#端口
server.port: 5601
 
#服務器IP
server.host: "10.10.114.4"
 
#elasticsearch服務器
elasticsearch.url: "http://10.10.114.4:9200"

啓動
nohup bin/kibana &


查看端口占用命令
netstat -apn |grep 5601
#殺掉進程
kill -9 進程號

瀏覽器訪問：http://ip:5601

kibana-6.3.0版本之後新增index索引不支持經常使用的正則匹配了，只能使用【*】

3、Logstash安裝

另一臺10.10.114.2安裝Logstash
mkdir /application/
tar xf jdk-8u151-linux-x64.tar.gz -C /application/
ln -s /application/jdk1.8.0_151 /application/jdk
sed -i.ori '$a export JAVA_HOME=/application/jdk\nexport PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH\nexport CLASSPATH=.$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$JAVA_HOME/lib/tools.jar' /etc/profile
source /etc/profile

tar xf logstash-6.3.2.tar.gz -C /application/
ln -s /application/logstash-6.3.2 /application/logstash

#收集IIS日誌爲例
cd /application/logstash/
mkdir conf && cd conf 
cat IIS.conf
input {
    beats {
        port => "5044"
        codec => json
    }
}

filter {
        if [message] =~ "^#" { drop {} }
        grok {
            match =>{"message" => "%{TIMESTAMP_ISO8601:timestamp} %{IPORHOST:s_ip} %{WORD:request_method} %{NOTSPACE:uripath} %{NOTSPACE:uri-query} %{NUMBER:port} - %{IPORHOST:c_ip} %{NOTSPACE:agent} %{NOTSPACE:referer} %{NUMBER:status} %{NUMBER:sc_bytes} %{NUMBER:cs_bytes} %{NUMBER:time}" }
        }
        date {
            match => ["timestamp","YYYY-MM-dd HH:mm:ss"]
        }
        mutate {
                 remove_field => ["message","beat","_id","host","@version","_score","tags"]
        }
}

output {
    elasticsearch {
        hosts => ["10.10.114.4:9200"]
        index => "logstash-iis-%{+YYYY.MM.dd}"
        }

    stdout {
        codec => rubydebug
        }
}


首先來測試配置文件是否可用,使用-f標誌指定配置文件。

/application/logstash/bin/logstash -f conf/IIS.conf --config.test_and_exit

--config.test_and_exit,會測試你配置文件的正確性,並給出錯誤信息.

若是可用,會輸出 OK ,以後能夠經過下面的命令來啓動logstash

/application/logstash/bin/logstash -f conf/IIS.conf &

查看端口
netstat -ntpl|grep 5044

 客戶端安裝Filebeat

windows下直接解壓縮後filebeat-6.3.2-windows-x86_64

修改filebeat.yml文件

    
filebeat.inputs:

- type: log
  encoding: GBK  #編碼格式
  enabled: true

  paths:
    - D:\ApplicationLogs\*\*\* #收集日誌的路徑


output.logstash:
  hosts: ["10.10.114.2:5044"] #直接寫到logstash

能夠在cmd下測試執行、能夠查看到傳輸的日誌文件

/路徑/filebeat.exe -c filebeat.yml -e

 

head插件安裝

Elasticsearch6.x版本不能使用命令直接安裝head插件
修改配置文件/etc/elasticsearch/elasticsearch.yml增長參數
# 增長參數，使head插件能夠訪問es 
http.cors.enabled: true 
http.cors.allow-origin: "*"

下載head插件
cd /usr/local/src
wget  https://github.com/mobz/elasticsearch-head/archive/master.zip
unzip master.zip 
mv elasticsearch-head-master/ /application/
　
安裝node
wget https://npm.taobao.org/mirrors/node/latest-v4.x/node-v4.4.7-linux-x64.tar.gz
tar -zxvf node-v4.4.7-linux-x64.tar.gz
　　
修改環境變量/etc/profile添加

export NODE_HOME=/application/node-v4.4.7-linux-x64
export PATH=$PATH:$NODE_HOME/bin
export NODE_PATH=$NODE_HOME/lib/node_modules
　　
設置生效
source /etc/profile
　　
安裝grunt
cd /application/elasticsearch-head-master
npm install -g grunt-cli

檢查是否安裝成功
[root@elk elasticsearch]# grunt -version
grunt-cli v1.3.1

 修改head插件源碼/application/elasticsearch-head-master/Gruntfile.js

hostname是新增的，不要忘記原有的true後面加,符號

修改鏈接地址/application/elasticsearch-head-master/_site/app.js 

下載運行head必要的文件（放置在文件夾/tmp下）

cd /tmp
wget https://github.com/Medium/phantomjs/releases/download/v2.1.1/phantomjs-2.1.1-linux-x86_64.tar.bz2
yum -y install bzip2

運行head

cd /application/elasticsearch-head-master
npm install

後臺啓動

grunt server &

web頁面驗證

能夠查看到node1節點

 最簡單的作法就是直接在谷歌瀏覽器添加應用程序