分佈式實時日誌處理平臺ELK

   

這三樣東西分別做用是:日誌收集、索引與搜索、可視化展示java

 

l  logstashlinux

    這張架構圖能夠看出logstash只是collect和index的地方,運行時傳入一個.conf文件,配置分三部分:input ,filter,output。nginx

l  redisgit

    redis在這裏是做爲日誌收集與索引之間解耦做用github

l  elasticsearchredis

    核心組件,用來搜索。主要特色:real-time,distributed,Highly Available,document oriented,schema free,RESTfulapache

 

kibanajson

    可視化日誌組件,讓數據交互變得更容易後端

 

部署

須要的組件

 

 

 

Logstash

logstash 10 分鐘教程 :http://logstash.net/docs/1.4.2/tutorials/10-minute-walkthrough/tomcat

 

下載最新logstash版本並解壓

 

 

 

編輯logstash.conf配置文件

 

logstash用戶說明文檔:http://logstash.net/docs/1.4.2/

log4j server配置實例:log4j.conf

input {

  log4j {

    data_timeout => 5

# mode => "server"

# port => 4560

  }

}

 

filter {

  json {

    source => "message"

    remove_field => ["message","class","file","host","method","path","priority","thread","type","logger_name"]

  }

}

 

output{

    #stdout {     codec => json   }

    redis {

        host => "redis.internal.173"

        port => 6379

        data_type => "list"

        key => "soalog"

    }

}

 

 

 

 

logstash輸出elasticsearch配置實例:soalog-es.conf

input {

  redis {

    host => "redis.internal.173"

    port => "6379"

    key => "soalog"

    data_type => "list"

  }

}

 

filter {

  json {

    source => "message"

    remove_field => ["message","type"]

  }

}

 

output {

  elasticsearch {

    #host => "es1.internal.173,es2.internal.173,es3.internal.173"

    cluster => "soaes"

    index => "soa_logs-%{+YYYY.MM.dd}"

  }

}

 

 

這裏filter配置source => message,是把message裏json串解析出來,做爲索引字段,而後配置remove_field 把不須要字段刪除 

 

啓動

./logstash -f soalog-es.conf --verbose -l ../soalog-es.log &

./logstash -f log4j.conf --verbose -l ../log4j.log &

 

 

 

Elastcisearch

 

下載最新版本elasticsearch並解壓

 

bin/elasticsearch -d 後端運行

 

驗證

 

elasticsearch集羣配置:

編輯 config/elasticsearch.yml 

#指定你的集羣名稱,默認是elasticsearch,在使用客戶端鏈接集羣模式會用到

cluster.name: soaes

#指定數據存儲目錄,能夠多個磁盤 /path/to/data1,/path/to/data2

path.data: /mnt/hadoop/esdata

#指定日誌存儲目錄

path.logs: /mnt/hadoop/eslogs

#集羣主節點列表,執行發現新節點

discovery.zen.ping.unicast.hosts: ["hadoop74", "hadoop75"]

 

配置es模板 ,能夠指定字段是否索引,以及存儲類型

在config目錄下建立templates目錄

增長模板文件template-soalogs.json

{

  "template-soalogs" : {

    "template" : "soa_logs*",

    "settings" : {

      "index.number_of_shards" : 5,

      "number_of_replicas" : 1,

      "index" : {

        "store" : {

          "compress" : {

            "stored" : true,

            "tv": true

          }

        }

      }

    },

    "mappings" : {

 "logs" : {

 "properties" : {

 "providerNode" : {

 "index" : "not_analyzed",

 "type" : "string"

 },

 "serviceMethod" : {

 "index" : "not_analyzed",

 "type" : "string"

 },

 "appId" : {

 "index" : "not_analyzed",

 "type" : "string"

 },

 "status" : {

 "type" : "long"

 },

 "srcAppId" : {

 "index" : "not_analyzed",

 "type" : "string"

 },

 "remark" : {

 "type" : "string"

 },

 "serviceVersion" : {

 "index" : "not_analyzed",

 "type" : "string"

 },

 "srcServiceVersion" : {

 "index" : "not_analyzed",

 "type" : "string"

 },

 "logSide" : {

 "type" : "long"

 },

 "invokeTime" : {

 "type" : "long"

 },

 "@version" : {

 "type" : "string"

 },

 "@timestamp" : {

 "format" : "dateOptionalTime",

 "type" : "date"

 },

 "srcServiceInterface" : {

 "index" : "not_analyzed",

 "type" : "string"

 },

 "serviceInterface" : {

 "index" : "not_analyzed",

 "type" : "string"

 },

 "retryCount" : {

 "type" : "long"

 },

 "traceId" : {

 "index" : "not_analyzed",

 "type" : "string"

 },

 "processTime" : {

 "type" : "long"

 },

 "consumerNode" : {

 "index" : "not_analyzed",

 "type" : "string"

 },

 "rpcId" : {

 "index" : "not_analyzed",

 "type" : "string"

 },

 "srcServiceMethod" : {

 "index" : "not_analyzed",

 "type" : "string"

 }

 }

 }

 }

  }

}

 

 

kibana

進入elasticsearch目錄

bin/plugin -install elasticsearch/kibana 
驗證:http://localhost:9200/_plugin/kibana 

kibana須要配置查詢索引規則

 

 

這裏index是soa_logs,按天分索引格式須要指定爲YYYY-MM-DD

 

 

 

logstash時差8小時問題

 

logstash在按天天輸出到elasticsearch時,由於時區使用utc,形成天天8:00才建立當天索引,而8:00之前數據則輸出到昨天的索引

修改logstash/lib/logstash/event.rb 能夠解決這個問題

第226行

.withZone(org.joda.time.DateTimeZone::UTC)

修改成

.withZone(org.joda.time.DateTimeZone.getDefault())

 

 

log4j.properties配置

#remote logging

log4j.additivity.logstash=false

log4j.logger.logstash=INFO,logstash

log4j.appender.logstash = org.apache.log4j.net.SocketAppender

log4j.appender.logstash.RemoteHost = localhost

log4j.appender.logstash.Port = 4560

log4j.appender.logstash.LocationInfo = false

 

 

java日誌輸出

    private static final org.slf4j.Logger logstash = org.slf4j.LoggerFactory.getLogger("logstash");

     logstash.info(JSONObject.toJSONString(rpcLog));

 

 

KOPF

 elasticsearch集羣監控

bin/plugin -install lmenezes/elasticsearch-kopf

http://localhost:9200/_plugin/kopf

 

 

 


logstash接入tomcat日誌示例:

 logstash代理端配置tomcat.conf

input {

  file {

    type=> "usap"

    path=> ["/opt/17173/apache-tomcat-7.0.50-8090/logs/catalina.out","/opt/17173/apache-tomcat-7.0.50-8088/logs/catalina.out","/opt/17173/apache-tomcat-7.0.50-8086/logs/catalina.out","/opt/

17173/apache-tomcat-7.0.50-8085/logs/catalina.out","/opt/17173/apache-tomcat-6.0.37-usap-image/logs/catalina.out"]

    codec=> multiline {

     pattern => "(^.+Exception:.+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"

     what=> "previous"

    }

  }

}

filter {

  grok {

    #match => { "message" => "%{COMBINEDAPACHELOG}" }

    match => [ "message", "%{TOMCATLOG}", "message", "%{CATALINALOG}" ]

    remove_field => ["message"]

  }

}

output {

# stdout{ codec => rubydebug }

  redis {host => "redis.internal.173" data_type => "list" key=> "usap" }

}

 

 

修改logstash/patterns/grok-patterns 

增長tomcat日誌過濾正則

#tomcat log

JAVACLASS (?:[a-zA-Z0-9-]+\:)+[A-Za-z0-9$]+

JAVALOGMESSAGE (.*)

THREAD [A-Za-z0-9\-\[\]]+

# MMM dd, yyyy HH:mm:ss eg: Jan 9, 2014 7:13:13 AM

CATALINA_DATESTAMP %{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM)

# yyyy-MM-dd HH:mm:ss,SSS ZZZ eg: 2014-01-09 17:32:25,527 -0800

TOMCAT_DATESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) %{ISO8601_TIMEZONE}

LOG_TIME %{HOUR}:?%{MINUTE}(?::?%{SECOND})

CATALINALOG %{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:class} %{JAVALOGMESSAGE:logmessage}

# 11:27:51,786 [http-bio-8088-exec-4] DEBUG JsonRpcServer:504 - Invoking method: getHistory

#TOMCATLOG %{LOG_TIME:timestamp} %{THREAD:thread} %{LOGLEVEL:level} %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage}

TOMCATLOG %{TOMCAT_DATESTAMP:timestamp} %{LOGLEVEL:level} %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage}

 

啓動 tomcat 日誌代理:

./logstash -f tomcat.conf --verbose -l ../tomcat.log & 

 

tomcat日誌存入es 

配置tomcat-es.conf 

input {

redis {

       host => 'redis.internal.173'

       data_type => 'list'

       port => "6379"

       key => 'usap'

       #type => 'redis-input'

       #codec => json

        }

       }

output {

# stdout { codec => rubydebug }

        elasticsearch {

         #host => "es1.internal.173,es2.internal.173,es3.internal.173" 

         cluster => "soaes"

         index => "usap-%{+YYYY.MM.dd}"

        }

}

 

啓動tomcat日誌存儲

./logstash -f tomcat-es.conf --verbose -l ../tomcat-es.log & 

 

 

logstash接入nginx\syslog日誌示例

logstash代理端配置nginx.conf 

input {

 file{

  type => "linux-syslog"

  path => [ "/var/log/*.log", "/var/log/messages"]

 }

 file {

  type => "nginx-access"

  path => "/usr/local/nginx/logs/access.log"

 }

 file {

  type => "nginx-error"

  path => "/usr/local/nginx/logs/error.log"

 }

}

output {

# stdout{ codec => rubydebug }

  redis {host => "redis.internal.173" data_type => "list" key=> "nginx" }

}

 

啓動nginx日誌代理

./logstash -f nginx.conf --verbose -l ../nginx.log & 

 

nginx日誌存入es

配置nginx-es.conf

input {

redis {

       host => 'redis.internal.173'

       data_type => 'list'

       port => "6379"

       key => 'nginx'

       #type => 'redis-input'

       #codec => json

        }

       }

filter {

 grok {

  type => "linux-syslog"

  pattern => "%{SYSLOGLINE}"

 }

 grok {

  type => "nginx-access"

  pattern => "%{IPORHOST:source_ip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] %{IPORHOST:host} %{QS:request} %{INT:status} %{INT:body_bytes_sent} %{QS:http_refere

r} %{QS:http_user_agent}"

 }

}

output {

# stdout { codec => rubydebug }

        elasticsearch {

         #host => "es1.internal.173,es2.internal.173,es3.internal.173"

         cluster => "soaes"

         index => "nginx-%{+YYYY.MM.dd}"

        }

        }

 

啓動nginx日誌存儲

./logstash -f nginx-es.conf --verbose -l ../nginx-es.log & 

相關文章
相關標籤/搜索