一：需求及基礎：html
場景：java
一、開發人員不能登陸線上服務器查看詳細日誌node
二、各個系統都有日誌，日誌數據分散難以查找python
三、日誌數據量大，查詢速度慢，或者數據不夠實時mysql
四、一個調用會涉及到多個系統，難以在這些協調中快速定位數據linux
Elastic Search + LogStash + Kibana = ELK Stackandroid
logstash1----| （redis實現鬆耦合功能）nginx
logstash2----|----->broker redis----->indexer logstash---->search storage<--------Web Logstashgit
logstash3----|github
ELS的概念：
1、索引：數據會放在多個索引中，索引能夠理解爲database，索引裏面存放的基本單位是文檔，LES會把索引分片，便於橫向擴展，分別能夠作備份，多個分片讀比較快，備份分片在主的掛掉以後能夠自動將本身提高爲主分片（實現橫向擴展和冗餘）
2、文檔類型：和redis同樣，key是有類型的
3、節點：一個ELS的實例是一個節點
4、集羣：多節點的集合組成集羣，相似於zookeeper會選舉出主節點，客戶端不須要關注主節點，鏈接任何一個均可以，數據會自動同步，所以應用不須要關注那個是主節點。前提是要把
配置文件：
[root@elk-server1 config]# vim elasticsearch.yml
cluster.name: hfelk-server #集羣的名稱，名稱相同就是一個集羣
node.name: Server1 #集羣狀況下，當前node的名字，每一個node應該不同
node.master: true #當前節點是否能夠被選舉爲master節點，能夠不選舉master作保存數據
node.data: true #當前節點是否存儲數據，也能夠不存儲數據，只作master
bootstrap.mlockall: true #鎖住內存，不作swap，提升效率
http.port: 9200 #客戶端訪問端口
transport.tcp.port: 9300 #集羣訪問端口：
index.number_of_shards: 5 #默認每一個項目5個分片
index.number_of_replicas: 1 #每一個主分片一個副本分片，即5個主分片就有5個副本
二：安裝及配置：
官網下載地址:
https://www.elastic.co/downloads
官方文檔：
https://www.elastic.co/guide/index.html
一、安裝：
安裝java環境，1.8.20或以上的版本
配置yum源或使用源碼安裝
二、啓動：
/usr/local/elasticsearch/bin/elasticsearch -d #後臺進程方式啓動
/etc/init.d/elasticsearch restart
三、設置啓動腳本：
下載：elasticsearch-servicewrapper-master.zip
[root@elk-server1 tianqi]# mv elasticsearch-servicewrapper-master/service/ /usr/local/elasticsearch/bin/
幫助信息：
[root@elk-server1 tianqi]# /usr/local/elasticsearch/bin/service/elasticsearch
Usage: /usr/local/elasticsearch/bin/service/elasticsearch [ console | start | stop | restart | condrestart | status | install | remove | dump ]
Commands:
console Launch in the current console.
start Start in the background as a daemon process.
stop Stop if running as a daemon or in another console.
restart Stop if running and then start.
condrestart Restart only if already running.
status Query the current status.
install Install to start automatically when system boots.
remove Uninstall.
dump Request a Java thread dump if running.
四、安裝啓動腳本：
[root@elk-server1 tianqi]# /usr/local/elasticsearch/bin/service/elasticsearch install #安裝腳本
Detected RHEL or Fedora:
Installing the Elasticsearch daemon..
[root@elk-server1 tianqi]# ls /etc/init.d/elasticsearch #驗證是否安裝完成
/etc/init.d/elasticsearch
[root@elk-server1 tianqi]# chkconfig --list | grep ela #自動設置爲開機啓動
elasticsearch 0:off 1:off 2:on 3:on 4:on 5:on 6:off
五、啓動elasticsearch服務：
[root@elk-server1 tianqi]# /etc/init.d/elasticsearch start
Starting Elasticsearch...
Waiting for Elasticsearch......
running: PID:14183
[root@elk-server1 tianqi]# /etc/init.d/elasticsearch status
Elasticsearch is running: PID:14183, Wrapper:STARTED, Java:STARTED
六、java的配置文件：
[root@elk-server1 service]# ls /usr/local/elasticsearch/bin/service/elasticsearch.conf
9200：訪問的都端口
9300：服務器之間通訊的端口
七、測試：
[root@elk-server1 elasticsearch]# curl -i -XGET http://192.168.0.251:9200
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 335
{
"status" : 200,
"name" : "Server1",
"cluster_name" : "HFELK-Server1",
"version" : {
"number" : "1.7.0",
"build_hash" : "929b9739cae115e73c346cb5f9a6f24ba735a743",
"build_timestamp" : "2015-07-16T14:31:07Z",
"build_snapshot" : false,
"lucene_version" : "4.10.4"
},
"tagline" : "You Know, for Search"
}
三:ES 概念和集羣：
一、基於http的RESTful API
以jsop返回查詢結果：
[root@elk-server1 config]# curl -XGET 'http://192.168.0.251:9200/_count?pretty' -d '
> {
> "query":{
> "match_all":{}
> }
> }
>
> '
{
"count" : 1,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
}
}
curl -i：
[root@elk-server1 config]# curl -i -XGET 'http://192.168.0.251:9200/_count?pretty' -d '
{
"query":{
"match_all":{}
}
}
'
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 95
{
"count" : 1,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
}
}
二、安裝ELS監控管理插件：
[root@elk-server1 service]# /usr/local/elasticsearch/bin/plugin -i elasticsearch/marvel/latest/
-> Installing elasticsearch/marvel/latest/...
Trying http://download.elasticsearch.org/elasticsearch/marvel/marvel-latest.zip...
Downloading ......................................................................................................................................................................................................................................................DONE
Installed elasticsearch/marvel/latest/ into /usr/local/elasticsearch/plugins/marvel
三、web訪問：http://xx.chinacloudapp.cn:9200/_plugin/marvel/
選選免費試用：
四、進入測試界面：
五、界面效果：
提交內容：
六、提交的代碼以下：
POST /index-demo/test
{
"user":"jack",
"message":"hello word"
}
}
七、 查看和刪除指定文檔內容：
GET /index-demo/test/AVP0y8ANAZWiuuxBK3mq/_source
DELETE /index-demo/test/AVP0y8ANAZWiuuxBK3mq/_source
八、搜索文檔：
GET /index-demo/test/_search?q=hello
{
"took": 97,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0.15342641,
"hits": [
{
"_index": "index-demo",
"_type": "test",
"_id": "AVP0y8ANAZWiuuxBK3mq",
"_score": 0.15342641,
"_source": {
"user": "jack",
"message": "hello word"
}
}
]
}
}
四：elasticsearch集羣管理程序之head：
一、安裝集羣的管理插件head：
集羣更換了虛擬機環境，因此主機名不同，安裝的時候要多安裝幾回，有的時候會由於網絡問題沒法一次 安裝完成。
[root@node6 local]# /usr/local/elasticsearch/bin/plugin -i mobz/elasticsearch-head/
-> Installing mobz/elasticsearch-head/...
Trying https://github.com/mobz/elasticsearch-head/archive/master.zip...
Downloading ....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE
Installed mobz/elasticsearch-head/ into /usr/local/elasticsearch/plugins/head
二、打開web管理端,查看是否已經有本機被管理了：
三、在另一臺主機上配置好java和elasticsearch，配置文件只要吧node節點的名字改了就行，其餘的不須要改，而後配置腳本啓動服務，再把elasticsearch啓動便可，會自動在集羣顯示，推薦三臺或以上的節點，由於其中一臺主機掛掉不影響整個服務的訪問
綠色表明分片都正常運行，20個分片都正常，表示集羣很是健康
黃色表示全部主分片都正常，可是副本分片有丟失，意味着ELS能夠正常工做，可是有必定的風險，性能也不是最好的
紅色表明有主分片丟失，此部分數據就沒法使用了
四、成功的集羣效果：
五、以上是在監控程序marvel的界面建立了兩個項目：
打開鏈接：http://192.168.10.206:9200/_plugin/marvel/sense/index.html
POST /index-demo/hi
{
"user":"tom1",
"message":"hello word"
}
}
POST /index-hello/hi
{
"user":"tom1",
"message":"hello word"
}
}
##############################################################################################
一：文檔
官方文檔地址：1.x版本和2.x版本
https://www.elastic.co/guide/en/elasticsearch/guide/index.html
硬件要求：
一、內存，官方推薦64G，可是本身使用32G或16G也能夠
二、CPU,核心越多越好
三、硬盤，越快越好，不建議使用NAS網絡存儲，官方建議使用RAID 0
四、網絡，千兆或以上
五、JVM，建議使用比較新的版本，能夠是openJDK或oracle的Java JDK
六、文件描述符，便可以打開的最大文件數，必定要改大
七、cluster 同一個集羣要同樣，集羣被的各node name不能相同
八、組播和單播設置
九、JDK 內存設置不要超過32G，能夠關閉Swap分區
十、鎖住內存設置
動態改變配置：
PUT /_cluster/settings
{
"persistent" : {
"discovery.zen.minimum_master_nodes" : 2
},
"transient" : {
"indices.store.throttle.max_bytes_per_sec" : "50mb"
}
}
滾動升級或維護：
一、儘可能不寫新的數據，就不產生新的索引
二、關閉自動分片：
PUT /_cluster/settings
{
"transient" : {
"cluster.routing.allocation.enable" : "none"
}
}
三、關閉當前節點：
中止後會在集羣當中找其餘的幾點並提高爲主節點，並將被中止的節點的分片分配給其餘節點，並將數據同步到其餘節點。
POST /_cluster/nodes/_local/_shutdown
四、執行升級或維護操做
五、重啓節點，會自動加入到集羣
六、開啓當前節點的分片功能
注意：即分片同步須要一段時間，須要等待集羣狀態轉換爲綠色即集羣可用狀態。
Shard rebalancing may take some time. Wait until the cluster has returned to status green before continuing.
七、對集羣當中的其餘節點作2-6步驟的操做
數據備份--->快照備份：curl
一、建立備份目錄，節點之間須要共享一個目錄，共享的文件系統須要每一個節點均可以訪問，而且每一個節點可掛載的路徑要一致
支持的共享系統：
Shared filesystem, such as a NAS
Amazon S3
HDFS (Hadoop Distributed File System)
Azure Cloud
執行掛載目錄的命令：
PUT _snapshot/my_backup
{
"type": "fs",
"settings": {
"location": "/mount/backups/my_backup"
}
}
二、建立快照：
POST _snapshot/my_backup/
{
"type": "fs",
"settings": {
"location": "/mount/backups/my_backup",
"max_snapshot_bytes_per_sec" : "50mb",
"max_restore_bytes_per_sec" : "50mb"
}
}
ELK的工做原理：
使用多播進行機器發現同一個集羣內的節點，並彙總各個節點的返回組成一個集羣，主節點要讀取各個節點的狀態，在關鍵的時候進行數據的恢復，主節點會堅持各個節點的狀態，並決定每一個分片的位置，經過ping的request檢測各失效的節點
三：安裝logstash：
官網下載地址：
https://www.elastic.co/downloads/logstash
一、安裝：
tar xvf logstash-1.5.3.zip
mv logstash-1.5.3 /usr/local/logstash
二、測試：
[root@node6 ~]# /usr/local/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }'
test
Logstash startup completed
2016-04-09T18:14:47.891Z node6.a.com test
三、使用ruby進行更詳細的輸出：
須要主機能解析本身的主機名，能夠在hosts文件解析：
[root@node6 ~]# /usr/local/logstash/bin/logstash -e 'input { stdin{} } output { stdout{codec => rubydebug}}'
asd
Logstash startup completed
{
"message" => "asd",
"@version" => "1",
"@timestamp" => "2016-04-09T18:13:51.250Z",
"host" => "node6.a.com"
}
四、經過logstas將輸出交給elasticsearch：
啓動：
[root@node6 ~]# /usr/local/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { host => "192.168.10.206" protocol => "http"} }'
'[DEPRECATED] use `require 'concurrent'` instead of `require 'concurrent_ruby'`
Logstash startup completed
五、經過logstash直接輸出到屏幕：
#配置文件以下:
input {
stdin{
}
}
output {
stdout { codec => rubydebug }
}
測試標準輸出：
[root@elk-server2 conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/03.conf
"message" => "{\"@timestamp\":\"2016-05-14T11:24:45+08:00\",\"host\":\"192.168.0.22\",\"clientip\":\"36.104.21.88\",\"size\":650,\"responsetime\":0.000,\"upstreamtime\":\"-\",\"upstreamhost\":\"-\",\"http_host\":\"webapi.weather.com.cn\",\"url\":\"/data/\",\"domain\":\"webapi.weather.com.cn\",\"xff\":\"-\",\"referer\":\"-\",\"status\":\"200\"}",
"@version" => "1",
"@timestamp" => "2016-05-14T03:25:04.068Z",
"host" => "elk-server2"
六、輸入一個hello word！ 以進行測試：
[root@node6 ~]# /usr/local/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { host => "192.168.10.206" protocol => "http"} }'
'[DEPRECATED] use `require 'concurrent'` instead of `require 'concurrent_ruby'`
Logstash startup completed
hello word!
七、查看集羣管理軟件head的狀態：
八、進行基本查詢：
九、配置文件格式：
input {
file {
path => "/var/log/messages"
type => "syslog"
}
file {
path => "/var/log/apache/access.log"
type => "apache"
}
}
十、數組類型--->多個文件：
path => [ "/var/log/messages", "/var/log/*.log" ] #經過*匹配多個文件
path => "/data/mysql/mysql.log"
十一、bool類型：
十二、字節設置：
my_bytes => "1113" # 1113 bytes
my_bytes => "10MiB" # 10485760 bytes
my_bytes => "100kib" # 102400 bytes
my_bytes => "180 mb" # 180000000 bytes
1三、codec：
1四、Hash：
match => {
"field1" => "value1"
"field2" => "value2"
...
}
1五、Number--->數字：
1六、Path--->密碼：
my_password => "password"
1七、Path---->路徑：
my_path => "/tmp/logstash"
1八、string-->字符串：
四：logstash的input使用語法：
一、input,默認不支持目錄的遞歸，即目錄中還有文件是不支持直接讀取的，可是可使用*/*進行匹配。
二、exclude--->排除文件，
三、sincedb_path，記錄讀取的時候位置，默認是一個隱藏文件
四、sincedb_write_interval，記錄sincedb_path文件的寫間隔，默認是15秒
五、start_position，從這個文件的什麼位置開始讀，默認是end，能夠改爲beginning
六、stat_interval，多久檢測一次此文件的更新狀態
五：logstash的output使用及插件：
一、能夠輸出到文件、rendis等
二、gzip，是否壓縮，默認爲false，壓縮是安裝數據流一點點增量壓縮的
三、message_format，消息的格式
六：logstash --> file -->elasticsearch:
經過logstash輸出到文件在輸入到elasticsearch：
一、啓動腳本：
vim /etc/init.d/logstash
[root@node6 tmp]# cat /etc/init.d/logstash
#!/bin/sh
# Init script for logstash
# Maintained by Elasticsearch
# Generated by pleaserun.
# Implemented based on LSB Core 3.1:
# * Sections: 20.2, 20.3
#
### BEGIN INIT INFO
# Provides: logstash
# Required-Start: $remote_fs $syslog
# Required-Stop: $remote_fs $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description:
# Description: Starts Logstash as a daemon.
### END INIT INFO
PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/logstash/bin/
export PATH
if [ `id -u` -ne 0 ]; then
echo "You need root privileges to run this script"
exit 1
fi
name=logstash
pidfile="/var/run/$name.pid"
export JAVA_HOME=/opt/jdk1.8.0_45
export JRE_HOME=/opt/jdk1.8.0_45/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$PATH
LS_USER=root
LS_GROUP=root
LS_HOME=/usr/local/logstash
LS_HEAP_SIZE="512m"
LS_LOG_DIR=/usr/local/logstash
LS_LOG_FILE="${LS_LOG_DIR}/$name.log"
LS_CONF_DIR=/etc/logstash.conf
LS_OPEN_FILES=16384
LS_NICE=19
LS_OPTS=""
[ -r /etc/default/$name ] && . /etc/default/$name
[ -r /etc/sysconfig/$name ] && . /etc/sysconfig/$name
program=/usr/local/logstash/bin/logstash
args="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS}"
quiet() {
"$@" > /dev/null 2>&1
return $?
}
start() {
LS_JAVA_OPTS="${LS_JAVA_OPTS} -Djava.io.tmpdir=${LS_HOME}"
HOME=${LS_HOME}
export PATH HOME LS_HEAP_SIZE LS_JAVA_OPTS LS_USE_GC_LOGGING
# chown doesn't grab the suplimental groups when setting the user:group - so we have to do it for it.
# Boy, I hope we're root here.
SGROUPS=$(id -Gn "$LS_USER" | tr " " "," | sed 's/,$//'; echo '')
if [ ! -z $SGROUPS ]
then
EXTRA_GROUPS="--groups $SGROUPS"
fi
# set ulimit as (root, presumably) first, before we drop privileges
ulimit -n ${LS_OPEN_FILES}
# Run the program!
nice -n ${LS_NICE} chroot --userspec $LS_USER:$LS_GROUP $EXTRA_GROUPS / sh -c "
cd $LS_HOME
ulimit -n ${LS_OPEN_FILES}
exec \"$program\" $args
" > "${LS_LOG_DIR}/$name.stdout" 2> "${LS_LOG_DIR}/$name.err" &
# Generate the pidfile from here. If we instead made the forked process
# generate it there will be a race condition between the pidfile writing
# and a process possibly asking for status.
echo $! > $pidfile
echo "$name started."
return 0
}
stop() {
# Try a few times to kill TERM the program
if status ; then
pid=`cat "$pidfile"`
echo "Killing $name (pid $pid) with SIGTERM"
kill -TERM $pid
# Wait for it to exit.
for i in 1 2 3 4 5 ; do
echo "Waiting $name (pid $pid) to die..."
status || break
sleep 1
done
if status ; then
if [ "$KILL_ON_STOP_TIMEOUT" -eq 1 ] ; then
echo "Timeout reached. Killing $name (pid $pid) with SIGKILL. This may result in data loss."
kill -KILL $pid
echo "$name killed with SIGKILL."
else
echo "$name stop failed; still running."
fi
else
echo "$name stopped."
fi
fi
}
status() {
if [ -f "$pidfile" ] ; then
pid=`cat "$pidfile"`
if kill -0 $pid > /dev/null 2> /dev/null ; then
# process by this pid is running.
# It may not be our pid, but that's what you get with just pidfiles.
# TODO(sissel): Check if this process seems to be the same as the one we
# expect. It'd be nice to use flock here, but flock uses fork, not exec,
# so it makes it quite awkward to use in this case.
return 0
else
return 2 # program is dead but pid file exists
fi
else
return 3 # program is not running
fi
}
force_stop() {
if status ; then
stop
status && kill -KILL `cat "$pidfile"`
fi
}
configtest() {
# Check if a config file exists
if [ ! "$(ls -A ${LS_CONF_DIR}/* 2> /dev/null)" ]; then
echo "There aren't any configuration files in ${LS_CONF_DIR}"
return 1
fi
HOME=${LS_HOME}
export PATH HOME JAVA_OPTS LS_HEAP_SIZE LS_JAVA_OPTS LS_USE_GC_LOGGING
test_args="-f ${LS_CONF_DIR} --configtest ${LS_OPTS}"
$program ${test_args}
[ $? -eq 0 ] && return 0
# Program not configured
return 6
}
case "$1" in
start)
status
code=$?
if [ $code -eq 0 ]; then
echo "$name is already running"
else
start
code=$?
fi
exit $code
;;
stop) stop ;;
force-stop) force_stop ;;
status)
status
code=$?
if [ $code -eq 0 ] ; then
echo "$name is running"
else
echo "$name is not running"
fi
exit $code
;;
restart)
quiet configtest
RET=$?
if [ ${RET} -ne 0 ]; then
echo "Configuration error. Not restarting. Re-run with configtest parameter for details"
exit ${RET}
fi
stop && start
;;
configtest)
configtest
exit $?
;;
*)
echo "Usage: $SCRIPTNAME {start|stop|force-stop|status|restart|configtest}" >&2
exit 3
;;
esac
exit $?
二、設置開機啓動：
chmod a+x /etc/init.d/logstash
chkconfig --add logstash
三、編輯配置文件：
[root@node6 tmp]# vim /etc/logstash.conf
input {
file {
path => "/var/log/messages"
}
}
output {
file {
path => "/tmp/log-%{+YYYY-MM-dd}messages.gz"
gzip => true
}
}
四、測試：
[root@node6 tmp]# echo "Apr 12 14:03:53 HTC-Server2 snmpd[1573]: Connection from UDP: [60.195.252.107]:31001->[192.168.0.116]" >> /var/log/messages
測試結果：
[root@node6 tmp]# ls /tmp/
hsperfdata_root log-2016-04-09.messages.gz
五、把輸出直接傳輸到elasticsearch：
[root@node6 ~]# vim /etc/logstash.conf
input {
file {
path => "/var/log/messages"
}
}
output {
file {
path => "/tmp/log-%{+YYYY-MM-dd}.messages.gz"
gzip => true
}
elasticsearch {
host => ["192.168.10.206"]
protocol => "http"
index => "system-message-%{+YYYY.MM.dd}"
}
}
六、在集羣管理平臺查看結果：
七：將logstash輸出給redis：
一、基本語法：
db：使用的數據庫，默認爲0，也可使用其餘的，可是redis主從不支持其餘數據庫同步。
host: redis服務器的地址
key：key的名稱
password：redis服務器的redis鏈接密碼
port：redis端口，默認6379
data_type：數據類型，支持string和list，咱們使用list
二、重啓logstash
三、向logstash監控的文件寫入數據
[root@node6 yum.repos.d]# echo "Apr 12 14:03:53 HTC-Server2 snmpd[1573]: Connection from UDP: [60.195.252.107]:31001->[192.168.0.116]" >> /var/log/messages
[root@node6 yum.repos.d]# echo "Apr 12 14:03:53 HTC-Server2 snmpd[1573]: Connection from UDP: [60.195.252.107]:31001->[192.168.0.116]" >> /var/log/messages
[root@node6 yum.repos.d]# echo "Apr 12 14:03:53 HTC-Server2 snmpd[1573]: Connection from UDP: [60.195.252.107]:31001->[192.168.0.116]" >> /var/log/messages
[root@node6 yum.repos.d]# echo "Apr 12 14:03:53 HTC-Server2 snmpd[1573]: Connection from UDP: [60.195.252.107]:31001->[192.168.0.116]" >> /var/log/messages
[root@node6 yum.repos.d]# echo "Apr 12 14:03:53 HTC-Server2 snmpd[1573]: Connection from UDP: [60.195.252.107]:31001->[192.168.0.116]" >> /var/log/messages
[root@node6 yum.repos.d]# echo "Apr 12 14:03:53 HTC-Server2 snmpd[1573]: Connection from UDP: [60.195.252.107]:31001->[192.168.0.116]" >> /var/log/messages
四、到redis查看結果：
[root@node5 ~]# redis-cli
127.0.0.1:6379> KEYS *
1) "system-message-jack" #已經生成數據
127.0.0.1:6379> LLEN system-message-jack #查看key的長度
(integer) 681
127.0.0.1:6379> LINDEX system-message-jack -1 #查看最後一行數據
"{\"message\":\"Apr 12 14:03:53 HTC-Server2 snmpd[1573]: Connection from UDP: [60.195.252.107]:31001->[192.168.0.116]\",\"@version\":\"1\",\"@timestamp\":\"2016-04-12T08:37:51.025Z\",\"host\":\"node6.a.com\",\"path\":\"/var/log/messages\"}"
五、在從新找一臺機器安裝logstash，步驟參考以前的步驟：
六、另一臺logstash的配置文件：
input { #讀取redis的數據
redis {
data_type => "list"
key => "system-message-jack"
host => "192.168.10.205"
port => "6379"
db => "0"
}
}
output { #將讀取到的reids的數據寫入到elasticsearch
elasticsearch {
host => ["192.168.10.206"]
protocol => "http"
index => "redis-message-%{+YYYY.MM.dd}"
}
}
七、向message文件寫入數據，寫入的數據會讀取到redis，reids的數據則會被傳輸給
[root@node6 yum.repos.d]# echo "Apr 12 14:03:53 HTC-Server2 snmpd[1573]: Connection from UDP: [60.195.252.107]:31001->[192.168.0.116]" >> /var/log/messages
[root@node6 yum.repos.d]# echo "Apr 12 14:03:53 HTC-Server2 snmpd[1573]: Connection from UDP: [60.195.252.107]:31001->[192.168.0.116]" >> /var/log/messages
[root@node6 yum.repos.d]# echo "Apr 12 14:03:53 HTC-Server2 snmpd[1573]: Connection from UDP: [60.195.252.107]:31001->[192.168.0.116]" >> /var/log/messages
[root@node6 yum.repos.d]# echo "Apr 12 14:03:53 HTC-Server2 snmpd[1573]: Connection from UDP: [60.195.252.107]:31001->[192.168.0.116]" >> /var/log/messages
八、在集羣管理平臺查看：
九、 查看索引：
八：分析的日誌類型：
1、系統日誌：/var/log下的全部的內容，google每個文件的內容
2、經過ELS分析某一個訪問記錄
3、錯誤日誌，收集後反饋給開發
4、系統運行日誌
5、其餘類型的日誌
九：日誌的字段劃分：
一、grok模塊：經過正則表達式，比較複雜，並且當數據大的時候會佔用CPU
二、json，簡單易用
三、將nginx的日誌設置爲json模式：
安裝nginx：能夠編譯或yum安裝，省略
四、日誌配置部分：
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
# log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log /var/log/nginx/access.log main;
log_format logstash_json '{"@timestamp":"$time_iso8601",' #定義日誌格式logstash_json
'"host":"$server_addr",'
'"clientip":"$remote_addr",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"url":"$uri",'
'"domain":"$host",'
'"xff":"$http_x_forwarded_for",'
'"referer":"$http_referer",'
'"agent":"$http_user_agent",'
'"status":"$status"}';
sendfile on;
serevr配置：
#
server {
listen 9009;
server_name localhost;
#charset koi8-r;
access_log /var/log/nginx/json.access.log logstash_json; #日誌文件保存路徑及使用上面定義的日誌格式logstash_json
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
error_page 404 /404.html;
location = /404.html {
root /usr/share/nginx/html;
}
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
五、使用ab訪問nginx測試日誌：
[root@node5 nginx]# ab -n1000 -c10 http://192.168.10.205:9009/ #一共1000個請求，每次併發10個，即100次請求完成
六、查看日誌是否有內容：
[root@node5 nginx]# tail /var/log/nginx/json.access.log
{"@timestamp":"2016-04-12T18:21:31+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"}
{"@timestamp":"2016-04-12T18:21:31+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"}
{"@timestamp":"2016-04-12T18:21:31+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"}
{"@timestamp":"2016-04-12T18:21:31+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"}
{"@timestamp":"2016-04-12T18:21:31+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.001,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"}
{"@timestamp":"2016-04-12T18:21:31+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"}
{"@timestamp":"2016-04-12T18:21:31+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"}
{"@timestamp":"2016-04-12T18:21:31+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"}
{"@timestamp":"2016-04-12T18:21:31+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"}
{"@timestamp":"2016-04-12T18:21:31+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"}
七、配置logstash手機nginx的json日誌並寫入到redis：
[root@node5 logstash]# cat /etc/logstash.conf
input {
# redis {
# data_type => "list"
# key => "system-message-jack"
# host => "192.168.10.205"
# port => "6379"
# db => "0"
# }
file {
path => "/var/log/nginx/json.access.log" #讀取指定的json格式的日誌
codec => "json" #指定json格式
}
}
output {
# elasticsearch {
# host => ["192.168.10.206"]
# protocol => "http"
# index => "redis-message-%{+YYYY.MM.dd}"
# }
redis {
data_type => "list"
key => "nginx-json-log" #nginx的json格式日誌的key名稱
host => "192.168.10.205" #寫入到redis服務器
port => "6379"
db => "1" #使用redis的數據庫1
}
}
八、重啓logstash服務，並使用ab從新訪問web地址，以產生新的日誌寫入redis：
九、在redis查詢是否有當前key的日誌：
[root@node5 nginx]# redis-cli
127.0.0.1:6379> SELECT 1
OK
127.0.0.1:6379[1]> KEYS *
1) "nginx-json-log" #已有日誌
127.0.0.1:6379[1]> LLEN nginx-json-log #日誌長度
(integer) 1000
十、配置logstash當前redis日誌並輸出至elasticsearch：
[root@node5 nginx]# grep "#" -v /etc/logstash.conf
input {
redis {
data_type => "list"
key => "nginx-json-log"
host => "192.168.10.205"
port => "6379"
db => "1"
}
}
output {
elasticsearch {
host => ["192.168.10.206"]
protocol => "http"
index => "nginx-json-log-%{+YYYY.MM.dd}"
}
}
十一、重啓logstash並使用ab批量訪問web，再查看elasticsearch集羣管理界面是否有nginx的json定義的nginx-json-log的key出現：
################################################################################################
一：kibana安裝：
kibana主要是搜索elasticsearch的數據，並進行數據可視化的展示，新版使用nodejs。
一、下載地址：
https://www.elastic.co/downloads/kibana
二、解壓安裝：
[root@node6 local]# tar xvf kibana-4.1.1-linux-x64.tar.gz
[root@node6 local]# mv kibana-4.1.1-linux-x64 kibana
[root@node6 ~]# cd /usr/local/kibana/
[root@node6 kibana]# ls
bin config LICENSE.txt node plugins README.txt src
三、編輯配置文件：
[root@node6 kibana]# cd config/
[root@node6 config]# ls
kibana.yml
[root@node6 config]# vim kibana.yml
elasticsearch_url: "http://192.168.10.206:9200"
四、直接啓動：
[root@node6 kibana]# bin/kibana
{"name":"Kibana","hostname":"node6.a.com","pid":3942,"level":30,"msg":"No existing kibana index found","time":"2016-04-12T12:20:50.069Z","v":0}
{"name":"Kibana","hostname":"node6.a.com","pid":3942,"level":30,"msg":"Listening on 0.0.0.0:5601","time":"2016-04-12T12:20:50.096Z","v":0}
五、驗證啓動:
[root@node6 ~]# ps -ef | grep kibana
root 3942 3745 3 20:20 pts/2 00:00:01 bin/../node/bin/node bin/../src/bin/kibana.js
root 3968 3947 0 20:21 pts/3 00:00:00 grep kibana
[root@node6 ~]# ss -tnl | grep 5601
LISTEN 0 128 *:5601 *:*
六、後臺啓動：
[root@node6 kibana]# nohup bin/kibana &
[1] 3975
七、訪問測試：默認監聽端口5601
http://192.168.10.206:560１
八、配置索引：索引的名稱要和logstash的output生成的索引能進行匹配才能夠
九、查看數據：默認顯示最新的500個文檔
十、數據精確搜索：
十一、搜索高級語法：
status:404 OR status:500 #搜索狀態是404或者是500之一的
status:301 AND status:200 #搜索便是301和200同時匹配的
status：[200 TO 300] ：搜索指定範圍的
十二、保存經常使用的搜索語法：
二：其餘的經常使用模塊：
一、系統日誌收集---> syslog：配置syslog結果寫入到elasticsearch，指定端口514，主機就是要收集日誌的服務器IP地址，便可使用
二、訪問日誌：nginx轉換成json格式
三、錯誤日誌：使用codec插件：
https://www.elastic.co/guide/en/logstash/1.5/codec-plugins.html
input {
stdin {
codec => multiline { #多行日誌，好比java的日誌
pattern => "^\s" #pattern => ".*\t.*" #找到換行符，會把多行認爲是一行，即會把當前行和上一行合成一行，直到有換行符結束
what => "previous"
}
}
}
四、運行日誌 codec => json，若是不是json要使用grok進行匹配，相對比較麻煩，若是丟日誌就看logstash.log，另外檢查日誌是否有效的json格式：
json效驗地址：http://www.bejson.com/
五、kibana的時區和時間問題:kibana會自動根據瀏覽器將時間加8小時，經過logstash寫入會自動解決，若是經過python腳本等寫入會產生時間問題
六、在地圖顯示IP具體來源地址：
https://www.elastic.co/guide/en/logstash/1.5/filter-plugins.html
七、條件判斷：
input {
file {
type => "apache"
path => "/var/log/apache.log"
}
file {
type => "tomcat"
path => "/var/log/tomcat.log"
}
}
filter {
if [type] == "apache" { #假如索引爲apache，就執行如下操做
redis {
data_type => "list"
key => "system-message-jack"
host => "192.168.10.205"
port => "6379"
db => "0"
}
if [type] == "tomcat" { #假如索引爲tomcat，就執行一次操做
redis {
data_type => "list"
key => "system-message-tomcat"
host => "192.168.10.205"
port => "6379"
db => "1" #寫不一樣的數據庫
}
}
nginx 最好設置buffer大小，64k
kibana要添加elastsearch的key
搜索的語法：直接搜索鍵值 a:b AND ALL NOT進行匹配。範圍 [200-299]
6.測試logstash配置文件語法是否正確：
6.1:配置正確的檢查結果：
[root@elk-server2 conf.d]# /etc/init.d/logstash configtest
Configuration OK
6.2：語法錯誤的顯示結果：
[root@elk-server2 tianqi]# /etc/init.d/logstash configtest
The given configuration is invalid. Reason: Expected one of #, {, } at line 17, column 53 (byte 355) after output {
if [type] == "nginx3" {
elasticsearch {
hosts => ["192.168.0.251:9200"]
index => "logstash-newsmart-nginx3-" {:level=>:fatal} #會指明語法錯誤的具體地方
三：tomcat日誌：
一、tomcat日誌默認不是json格式的，可是logstash分析的時候就沒有key和valus了，因此咱們能夠將tomcat日誌的格式定義爲json的格式：
directory="logs" prefix="localhost_access_log." suffix=".log"
pattern="{"client":"%h", "client user":"%l", "authenticated":"%u", "access time":"%t", "method":"%r", "status":"%s", "send bytes":"%b", "Query?string":"%q", "partner":"%{Referer}i", "Agent version":"%{User-Agent}i"}"/>
二、取到的日誌結果爲：
{"client":"180.95.129.206", "client user":"-", "authenticated":"-", "access time":"[20/Apr/2016:03:47:40 +0000]", "method":"GET /image/android_logo.png HTTP/1.1", "status":"200", "send bytes":"1915", "Query string":"", "partner":"http://mobile.weathercn.com/index.do?id=101160101&partner=1000001003", "Agent version":"Mozilla/5.0 (Linux; U; Android 5.1.1; zh-cn; NX510J Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko)Version/4.0 Chrome/37.0.0.0 MQQBrowser/6.6 Mobile Safari/537.36"}
三、在線驗證是否合法的json格式：
地址：http://www.bejson.com/，將完整的一行日誌複製到驗證框，而後點驗證便可：結果以下
四：nginx 日誌格式處理：
一、編輯nginx.conf配置文件，自定義一個日誌格式：
[root@node5 ~]# vim /etc/nginx/nginx.conf
二、添加內容以下：
log_format logstash_json '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
'"clientip":"$remote_addr",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"url":"$uri",'
'"domain":"$host",'
'"xff":"$http_x_forwarded_for",'
'"referer":"$http_referer",'
'"agent":"$http_user_agent",'
'"status":"$status"}';
三、編輯主機配置：
[root@node5 ~]# grep -v "#" /etc/nginx/conf.d/locathost.conf | grep -v "^$"
server {
listen 9009; #監聽的端口
server_name www.a.com; #主機名
access_log /var/log/nginx/json.access.log logstash_json; #定義日誌路徑爲/var/log/nginx/json.access.log,並引用在主配置文件nginx.conf中定義的json日誌格式
include /etc/nginx/default.d/*.conf;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
error_page 404 /404.html;
location = /404.html {
root /usr/share/nginx/html;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
四、重啓nginx，查看日誌格式是json格式了：
[root@node5 ~]# tail /var/log/nginx/json.access.log
{"@timestamp":"2016-04-12T22:15:19+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"}
{"@timestamp":"2016-04-12T22:15:19+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"}
{"@timestamp":"2016-04-12T22:15:19+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"}
{"@timestamp":"2016-04-12T22:15:19+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"}
{"@timestamp":"2016-04-12T22:15:19+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.001,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"}
{"@timestamp":"2016-04-12T22:15:19+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"}
五、在線效驗日誌格式是否正確：
效驗地址：http://www.bejson.com/
五：畫圖功能
在地圖顯示IP的訪問次數統計：
一、在elasticsearch服務器用戶家目錄下載一個Filebeat 模板：
cd ~
curl -O https://gist.githubusercontent.com/thisismitch/3429023e8438cc25b86c/raw/d8c479e2a1adcea8b1fe86570e42abab0f10f364/filebeat-index-template.json #這是一個模板文件
二、加載模板：
[root@elk-server1 ~]# curl -XPUT 'http://192.168.0.251:9200/_template/filebeat?pretty' -d@filebeat-index-template.json #是elasticsearch監聽的IP地址
{
"acknowledged" : true #必定要返回true才表示成功
}
三、下載GeoIP 數據庫文件：
[root@elk-server1 ~]# cd /etc/logstash/
[root@elk-server1 logstash]# curl -O "http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz"
[root@elk-server1 logstash]# gunzip GeoLiteCity.dat.gz
[root@elk-server1 logstash]# ls
conf.d GeoLiteCity.dat #確認文件存在
四、配置logstash使用GeoIP：
[root@elk-server1 logstash]# vim /etc/logstash/conf.d/11-mobile-tomcat-access.conf #logstash的文件配置要以.conf結尾
input {
redis {
data_type => "list"
key => "mobile-tomcat-access-log"
host => "192.168.0.251"
port => "6379"
db => "0"
codec => "json"
}
}
#input部分爲從redis讀取客戶端logstash分析提交後的訪問日誌
filter {
if [type] == "mobile-tomcat" {
geoip {
source => "client" #client 是客戶端logstash收集日誌時定義的公網IP的key名稱，必定要和實際名稱一致，由於要經過此名稱獲取到其對於的ip地址
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
}
}
}
output {
if [type] == "mobile-tomcat" {
elasticsearch {
hosts => ["192.168.0.251"]
manage_template => true
index => "logstash-mobile-tomcat-access-log-%{+YYYY.MM.dd}" #index的名稱必定要是logstash開頭的，不然會在使用地圖的時候出現geoIP type沒法找找到的相似錯誤
flush_size => 2000
idle_flush_time => 10
}
}
}
五、在kibana界面添加新的索引，而後visualize---->Tile map---->From a new search---->Select a index patterm--->選擇以前的index---->Geo coordinates，而後點綠色的運行按鈕便可：