提升安全性
集中存放日誌
缺陷:對日誌的分析困難前端
Elasticsearch:存儲,索引池
Logstash:日誌收集器
Kibana:數據可視化java
1,將日誌進行集中化管理
2,將日誌格式化(Logstash)並輸出到Elasticsearch
3,對格式化後的數據進行索引和存儲(Elasticsearch)
4,前端數據的展現(Kibana)node
提供了一個分佈式多用戶能力的全文搜索引擎linux
接近實時
集羣
節點
索引:索引(庫)-->類型(表)-->文檔(記錄)
分片和副本c++
一款強大的數據處理工具,能夠實現數據傳輸、格式處理、格式化輸出
數據輸入、數據加工(如過濾,改寫等)以及數據輸出web
Shipper
Indexer
Broker
Search and Storage
Web Interfaceexpress
一個針對Elasticsearch的開源分析及可視化平臺
搜索、查看存儲在Elasticsearch索引中的數據
經過各類圖表進行高級數據分析及展現apache
Elasticsearch無縫之集成
整合數據,複雜數據分析
讓更多團隊成員受益
接口靈活,分享更容易
配置簡單,可視化多數據源
簡單數據導出npm
[root@node1 ~]# vim /etc/hosts ##配置解析名 192.168.52.133 node1 192.168.52.134 node2 [root@node1 ~]# systemctl stop firewalld.service ##關閉防火牆 [root@node1 ~]# setenforce 0 ##關閉加強型安全功能 [root@node1 ~]# java -version ##查看是否支持Java [root@node1 ~]# mount.cifs //192.168.100.100/tools /mnt/tools/ ##掛載 Password for root@//192.168.100.100/tools: [root@node1 ~]# cd /mnt/tools/elk/ [root@node1 elk]# rpm -ivh elasticsearch-5.5.0.rpm ##安裝 警告:elasticsearch-5.5.0.rpm: 頭V4 RSA/SHA512 Signature, 密鑰 ID d88e42b4: NOKEY 準備中... ################################# [100%] Creating elasticsearch group... OK Creating elasticsearch user... OK 正在升級/安裝... 1:elasticsearch-0:5.5.0-1 ################################# [100%] ### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd sudo systemctl daemon-reload sudo systemctl enable elasticsearch.service ### You can start elasticsearch service by executing sudo systemctl start elasticsearch.service [root@node1 elk]# systemctl daemon-reload ##重載守護進程 [root@node1 elk]# systemctl enable elasticsearch.service ##開機自啓 Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service. [root@node1 elk]# cd /etc/elasticsearch/ [root@node1 elasticsearch]# cp elasticsearch.yml elasticsearch.yml.bak ##備份 [root@node1 elasticsearch]# vim elasticsearch.yml ##修改配置文件 cluster.name: my-elk-cluster ##集羣名 node.name: node1 ##節點名,第二個節點爲node2 path.data: /data/elk_data ##數據存放位置 path.logs: /var/log/elasticsearch/ ##日誌存放位置 bootstrap.memory_lock: false ##不在啓動時鎖定內存 network.host: 0.0.0.0 ##提供服務綁定的IP地址,爲全部地址 http.port: 9200 ##端口號爲9200 discovery.zen.ping.unicast.hosts: ["node1", "node2"] ##集羣發現經過單播實現 [root@node1 elasticsearch]# grep -v "^#" /etc/elasticsearch/elasticsearch.yml ##檢查配置是否正確 cluster.name: my-elk-cluster node.name: node1 path.data: /data/elk_data path.logs: /var/log/elasticsearch/ bootstrap.memory_lock: false network.host: 0.0.0.0 http.port: 9200 discovery.zen.ping.unicast.hosts: ["node1", "node2"] [root@node1 elasticsearch]# mkdir -p /data/elk_data ##建立數據存放點 [root@node1 elasticsearch]# chown elasticsearch.elasticsearch /data/elk_data/ ##給權限 [root@node1 elasticsearch]# systemctl start elasticsearch.service ##開啓服務 [root@node1 elasticsearch]# netstat -ntap | grep 9200 ##查看開啓狀況 tcp6 0 0 :::9200 :::* LISTEN 83358/java [root@node1 elasticsearch]#
查看node1節點信息json
查看node2節點信息
node1健康檢查
node2健康檢查
node1狀態
node2狀態
[root@node1 elasticsearch]# yum install gcc gcc-c++ make -y ##安裝編譯工具 [root@node1 elasticsearch]# cd /mnt/tools/elk/ [root@node1 elk]# tar xf node-v8.2.1.tar.gz -C /opt/ ##解壓插件 [root@node1 elk]# cd /opt/node-v8.2.1/ [root@node1 node-v8.2.1]# ./configure ##配置 [root@node1 node-v8.2.1]# make && make install ##編譯安裝
[root@node1 node-v8.2.1]# cd /mnt/tools/elk/ [root@node1 elk]# tar xf phantomjs-2.1.1-linux-x86_64.tar.bz2 -C /usr/local/src/ ##解壓到/usr/local/src下 [root@node1 elk]# cd /usr/local/src/phantomjs-2.1.1-linux-x86_64/bin/ [root@node1 bin]# cp phantomjs /usr/local/bin/ ##編譯系統識別
[root@node1 bin]# cd /mnt/tools/elk/ [root@node1 elk]# tar xf elasticsearch-head.tar.gz -C /usr/local/src/ ##解壓 [root@node1 elk]# cd /usr/local/src/elasticsearch-head/ [root@node1 elasticsearch-head]# npm install ##安裝 npm WARN elasticsearch-head@0.0.0 license should be a valid SPDX license expression npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.11 (node_modules/fsevents): npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.11: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"}) added 71 packages in 7.262s [root@node1 elasticsearch-head]#
[root@node1 elasticsearch-head]# cd ~ [root@node1 ~]# vim /etc/elasticsearch/elasticsearch.yml #末行插入 http.cors.enabled: true ##開啓跨域訪問支持,默認爲false http.cors.allow-origin: "*" ##跨域訪問容許的域名地址 [root@node1 ~]# systemctl restart elasticsearch.service ##重啓 [root@node1 ~]# cd /usr/local/src/elasticsearch-head/ [root@node1 elasticsearch-head]# npm run start & ##後臺運行數據可視化服務 [1] 83664 [root@node1 elasticsearch-head]# > elasticsearch-head@0.0.0 start /usr/local/src/elasticsearch-head > grunt server Running "connect:server" (connect) task Waiting forever... Started connect web server on http://localhost:9100 [root@node1 elasticsearch-head]# [root@node1 elasticsearch-head]# netstat -ntap | grep 9200 tcp6 0 0 :::9200 :::* LISTEN 83358/java [root@node1 elasticsearch-head]# netstat -ntap | grep 9100 tcp 0 0 0.0.0.0:9100 0.0.0.0:* LISTEN 83674/grunt [root@node1 elasticsearch-head]#
node1
node2
[root@node2 ~]# curl -XPUT 'localhost:9200/index-demo/test/1?pretty&pretty' -H 'content-Type: application/json' -d '{"user":"zhangsan","mesg":"hello world"}' ##建立索引信息 { "_index" : "index-demo", "_type" : "test", "_id" : "1", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 2, "failed" : 0 }, "created" : true } [root@node1 ~]#
[root@apache ~]# systemctl stop firewalld.service [root@apache ~]# setenforce 0 [root@apache ~]# yum install httpd -y ##安裝服務 [root@apache ~]# systemctl start httpd.service ##啓動服務 [root@apache ~]# java -version [root@apache ~]# mount.cifs //192.168.100.100/tools /mnt/tools/ ##掛載 Password for root@//192.168.100.100/tools: [root@apache ~]# cd /mnt/tools/elk/ [root@apache elk]# rpm -ivh logstash-5.5.1.rpm ##安裝logstash 警告:logstash-5.5.1.rpm: 頭V4 RSA/SHA512 Signature, 密鑰 ID d88e42b4: NOKEY 準備中... ################################# [100%] 正在升級/安裝... 1:logstash-1:5.5.1-1 ################################# [100%] Using provided startup.options file: /etc/logstash/startup.options OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N Successfully created system startup script for Logstash [root@apache elk]# systemctl start logstash.service ##開啓服務 [root@apache elk]# systemctl enable logstash.service ##開機自啓 Created symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service. [root@apache elk]# ln -s /usr/share/logstash/bin/logstash /usr/local/bin/ ##便於系統識別 [root@apache elk]#
[root@apache elk]# chmod o+r /var/log/messages ##給其餘用戶讀權限 [root@apache elk]# vim /etc/logstash/conf.d/system.conf ##建立文件 input { file{ path => "/var/log/messages" ##輸出目錄 type => "system" start_position => "beginning" } } output { elasticsearch { #輸入地址指向node1節點 hosts => ["192.168.13.129:9200"] index => "system-%{+YYYY.MM.dd}" } } [root@apache elk]# systemctl restart logstash.service ##重啓服務 ##也能夠用數據瀏覽查看詳細信息
[root@node1 ~]# cd /mnt/tools/elk/ [root@node1 elk]# rpm -ivh kibana-5.5.1-x86_64.rpm ##安裝 警告:kibana-5.5.1-x86_64.rpm: 頭V4 RSA/SHA512 Signature, 密鑰 ID d88e42b4: NOKEY 準備中... ################################# [100%] 正在升級/安裝... 1:kibana-5.5.1-1 ################################# [100%] [root@node1 elk]# cd /etc/kibana/ [root@node1 kibana]# cp kibana.yml kibana.yml.bak ##備份 [root@node1 kibana]# vim kibana.yml ##修改配置文件 server.port: 5601 ##端口號 server.host: "0.0.0.0" ##監放任意網段 elasticsearch.url: "http://192.168.13.129:9200" ##本機節點地址 kibana.index: ".kibana" ##索引名稱 [root@node1 kibana]# systemctl start kibana.service ##開啓服務 [root@node1 kibana]# systemctl enable kibana.service ##開機自啓 Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /etc/systemd/system/kibana.service. [root@node1 elk]# [root@node1 elk]# netstat -ntap | grep 5601 ##查看端口 tcp 0 0 127.0.0.1:5601 0.0.0.0:* LISTEN 84837/node [root@node1 elk]#
[root@apache elk]# vim /etc/logstash/conf.d/apache_log.conf ##建立配置文件 input { file{ path => "/etc/httpd/logs/access_log" ##輸入信息 type => "access" start_position => "beginning" } file{ path => "/etc/httpd/logs/error_log" type => "error" start_position => "beginning" } } output { if [type] == "access" { ##根據條件判斷輸出信息 elasticsearch { hosts => ["192.168.13.129:9200"] index => "apache_access-%{+YYYY.MM.dd}" } } if [type] == "error" { elasticsearch { hosts => ["192.168.13.129:9200"] index => "apache_error-%{+YYYY.MM.dd}" } } } [root@apache elk]# logstash -f /etc/logstash/conf.d/apache_log.conf ##根據配置文件配置logstach
只有error日誌
瀏覽器訪問Apache服務
生成access日誌
##選擇management>Index Patterns>create index patterns ##建立apache兩個日誌的信息
在kibana建立access訪問日誌
在kibana建立error訪問日誌
查看access日誌統計狀況
查看error日誌統計狀況