Lotgstash日誌切割示例

logstash的功能有一點是把 各類軟件生成的各類格式的日誌 轉換成一個方便檢索篩選的格式，本文演示了一個最簡單的例子。

一  轉換的效果

 實例： rabbitmq-server 日誌：

=INFO REPORT==== 16-Jan-2017::09:27:09 ===
Mirrored queue 'heat-engine-listener.e9e416bb-6733-4981-bf00-bd64c104ccad' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2266.0>

轉換後的格式爲：

{

"year" => "2017",
"mounthday" => "16",
"logdata" => "Mirrored queue 'heat-engine-listener.e9e416bb-6733-4981-bf00-bd64c104ccad' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2266.0>",
"message" => "=INFO REPORT==== 16-Jan-2017::09:27:09 ===\nMirrored queue 'heat-engine-listener.e9e416bb-6733-4981-bf00-bd64c104ccad' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2266.0>",
"type" => "rabbit",
"tags" => [
[0] "multiline"
],
"path" => "/var/log/rabbitmq/rabbit@server-31.log",
"@timestamp" => 2017-01-16T01:27:09.718Z,
"loglevel" => "INFO",
"@version" => "1",
"host" => "server-31",
"time" => "09:27:09",
"mounth" => "Jan"
}

轉換後的內容傳入elasticsearch中，用戶就能夠按照時間、日誌等級、主機等對彙總的日誌進行篩選檢索

二 轉換的過程

仍是以剛纔那條日誌爲例

=INFO REPORT==== 16-Jan-2017::09:27:09 ===
Mirrored queue 'reply_963a14cce15f48e786240aad41817847' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2262.0>

=INFO REPORT==== 16-Jan-2017::09:27:09 ===
Mirrored queue 'heat-engine-listener.e9e416bb-6733-4981-bf00-bd64c104ccad' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2266.0>

=INFO REPORT==== 16-Jan-2017::09:27:09 ===
Mirrored queue 'q-agent-notifier-network-update' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2270.0>

日誌是多行，先後各有一行空行，日誌行以=開頭，

一、多行合併

首先是合併多行，

安裝多行插件：

/usr/share/logstash/bin/logstash-plugin install  logstash-filter-multiline

在配置文件中配置多行合併

codec => multiline {

pattern => "^="
what => "previous"
negate => true
}

最終日誌轉換爲 =INFO REPORT==== 16-Jan-2017::09:27:09 ===\nMirrored queue 'heat-engine-listener.e9e416bb-6733-4981-bf00-bd64c104ccad' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2266.0>

二、分析日誌的格式和規律

結合全部的rabbitmq的日誌總結規律爲

=「日誌級別」 REPORT====  "日期"::「時間」 ===\n「日誌內容」

注意不要忘記中間的空格

三、正則匹配

logstash內置了不少常規正則，參見

https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns

本文都是採用內置的正則

=INFO REPORT==== 16-Jan-2017::09:27:09 ===\nMirrored queue 'heat-engine-listener.e9e416bb-6733-4981-bf00-bd64c104ccad' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2266.0> 

我最終的匹配的表達式爲

^=%{LOGLEVEL:loglevel} REPORT=+ %{MONTHDAY:mounthday}-%{MONTH:mounth}-%{YEAR:year}::%{TIME:time} ===\n%{GREEDYDATA:logdata}$

%{LOGLEVEL:loglevel}表示這是一個變量，裏面的內容要匹配logstash內置的LOGLEVE正則，而且裏面的內容和loglevel這個key造成一對kv值："loglevel":"INFO"

其餘一直類推

logstash提供了一個測試表達式的網址http://grokdebug.herokuapp.com/