1、Elasticsearch 安裝
1、部署系統以及環境準備 cat /etc/redhat-release CentOS Linux release 7.4.1708 (Core) uname -r 3.10.0-693.el7.x86_64 #Firewalld and selinux systemctl stop firewalld systemctl disable firewalld sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config setenforce 0 sed -i 's/localhost.localdomain/ELK.localdomain/' /etc/hostname hostnamectl set-hostname ELK.localdomain 安裝jdk8以上版本 mkdir /application/ tar xf jdk-8u151-linux-x64.tar.gz -C /application/ ln -s /application/jdk1.8.0_151 /application/jdk sed -i.ori '$a export JAVA_HOME=/application/jdk\nexport PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH\nexport CLASSPATH=.$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$JAVA_HOME/lib/tools.jar' /etc/profile source /etc/profile java -version java version "1.8.0_151" Java(TM) SE Runtime Environment (build 1.8.0_151-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode) 二、部署elasticsearch6.3.2 採用壓縮包解壓縮安裝方式啓動服務 建立另外獨立帳戶專供es使用 cd /usr/local wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.tar.gz tar zxvf elasticsearch-6.3.2.tar.gz -C /application/ groupadd ela 建立ela組 useradd -g ela ela 建立ela用戶，而且加入ela組 passwd ela 爲ela用戶設定登陸密碼 ln -s /application/elasticsearch-6.3.2/ /application/elasticsearch chown -R ela.ela /application/elasticsearch 配置文件 grep -n '^[a-Z]' /application/elasticsearch/config/elasticsearch.yml 17:cluster.name: elk 23:node.name: node-1 33:path.data: /home/elkdata 37:path.logs: /var/log/elasticsearch 42:bootstrap.memory_lock: true 54:network.host: 0.0.0.0 58:http.port: 9200 88:http.cors.enabled: true 89:http.cors.allow-origin: "*" mkdir /home/elkdata/ -p chown -R ela.ela /home/elkdata/ mkdir /var/log/elasticsearch -p chown -R ela.ela /var/log/elasticsearch/ vim /etc/security/limits.conf #增長如下行 * soft nofile 65536 * hard nofile 131072 ela soft memlock unlimited ela hard memlock unlimited 讓設置生效重啓系統或者另外開一個終端 vim /etc/sysctl.conf #增長一行 vm.max_map_count = 655360 sysctl -p 檢查文件權限是不是ela用戶 ls -l /application/elasticsearch-6.3.2/ ls -l /application/elasticsearch/ ls -l /application/elasticsearch/config/jvm.options 切換到ela用戶，-d後臺運行 [ela@elk elasticsearch-6.3.2]$ ./bin/elasticsearch -d [ela@elk elasticsearch-6.3.2]$ netstat -ntpl|grep 9200 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp6 0 0 :::9200 :::* LISTEN 147438/java
2、Kibana安裝
#下載 cd /usr/local/src wget https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-linux-x86_64.tar.gz #解壓 tar xzvf kibana-6.3.2-linux-x86_64.tar.gz ln -s /application/kibana-6.3.2-linux-x86_64 /application/kibana vim /application/kibana/config/kibana.yml #端口 server.port: 5601 #服務器IP server.host: "10.10.114.4" #elasticsearch服務器 elasticsearch.url: "http://10.10.114.4:9200" 啓動 nohup bin/kibana & 查看端口占用命令 netstat -apn |grep 5601 #殺掉進程 kill -9 進程號
瀏覽器訪問：http://ip:5601java
kibana-6.3.0版本之後新增index索引不支持經常使用的正則匹配了，只能使用【*】node
3、Logstash安裝
另一臺10.10.114.2安裝Logstash mkdir /application/ tar xf jdk-8u151-linux-x64.tar.gz -C /application/ ln -s /application/jdk1.8.0_151 /application/jdk sed -i.ori '$a export JAVA_HOME=/application/jdk\nexport PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH\nexport CLASSPATH=.$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$JAVA_HOME/lib/tools.jar' /etc/profile source /etc/profile tar xf logstash-6.3.2.tar.gz -C /application/ ln -s /application/logstash-6.3.2 /application/logstash #收集IIS日誌爲例 cd /application/logstash/ mkdir conf && cd conf cat IIS.conf input { beats { port => "5044" codec => json } } filter { if [message] =~ "^#" { drop {} } grok { match =>{"message" => "%{TIMESTAMP_ISO8601:timestamp} %{IPORHOST:s_ip} %{WORD:request_method} %{NOTSPACE:uripath} %{NOTSPACE:uri-query} %{NUMBER:port} - %{IPORHOST:c_ip} %{NOTSPACE:agent} %{NOTSPACE:referer} %{NUMBER:status} %{NUMBER:sc_bytes} %{NUMBER:cs_bytes} %{NUMBER:time}" } } date { match => ["timestamp","YYYY-MM-dd HH:mm:ss"] } mutate { remove_field => ["message","beat","_id","host","@version","_score","tags"] } } output { elasticsearch { hosts => ["10.10.114.4:9200"] index => "logstash-iis-%{+YYYY.MM.dd}" } stdout { codec => rubydebug } } 首先來測試配置文件是否可用,使用-f標誌指定配置文件。 /application/logstash/bin/logstash -f conf/IIS.conf --config.test_and_exit --config.test_and_exit,會測試你配置文件的正確性,並給出錯誤信息. 若是可用,會輸出 OK ,以後能夠經過下面的命令來啓動logstash /application/logstash/bin/logstash -f conf/IIS.conf & 查看端口 netstat -ntpl|grep 5044
客戶端安裝Filebeat
windows下直接解壓縮後filebeat-6.3.2-windows-x86_64linux
修改filebeat.yml文件git
filebeat.inputs: - type: log encoding: GBK #編碼格式 enabled: true paths: - D:\ApplicationLogs\*\*\* #收集日誌的路徑 output.logstash: hosts: ["10.10.114.2:5044"] #直接寫到logstash
能夠在cmd下測試執行、能夠查看到傳輸的日誌文件github
/路徑/filebeat.exe -c filebeat.yml -e
head插件安裝
Elasticsearch6.x版本不能使用命令直接安裝head插件 修改配置文件/etc/elasticsearch/elasticsearch.yml增長參數 # 增長參數，使head插件能夠訪問es http.cors.enabled: true http.cors.allow-origin: "*" 下載head插件 cd /usr/local/src wget https://github.com/mobz/elasticsearch-head/archive/master.zip unzip master.zip mv elasticsearch-head-master/ /application/ 安裝node wget https://npm.taobao.org/mirrors/node/latest-v4.x/node-v4.4.7-linux-x64.tar.gz tar -zxvf node-v4.4.7-linux-x64.tar.gz 修改環境變量/etc/profile添加 export NODE_HOME=/application/node-v4.4.7-linux-x64 export PATH=$PATH:$NODE_HOME/bin export NODE_PATH=$NODE_HOME/lib/node_modules 設置生效 source /etc/profile 安裝grunt cd /application/elasticsearch-head-master npm install -g grunt-cli 檢查是否安裝成功 [root@elk elasticsearch]# grunt -version grunt-cli v1.3.1
修改head插件源碼/application/elasticsearch-head-master/Gruntfile.jsweb
hostname是新增的，不要忘記原有的true後面加,符號npm
修改鏈接地址/application/elasticsearch-head-master/_site/app.js json
下載運行head必要的文件（放置在文件夾/tmp下）bootstrap
cd /tmp wget https://github.com/Medium/phantomjs/releases/download/v2.1.1/phantomjs-2.1.1-linux-x86_64.tar.bz2 yum -y install bzip2
運行headvim
cd /application/elasticsearch-head-master npm install
後臺啓動
grunt server &
web頁面驗證
能夠查看到node1節點
最簡單的作法就是直接在谷歌瀏覽器添加應用程序