Logstash筆記（三）----Filter插件及grok的正則表達式來解析日誌
（一）簡介：html
豐富的過濾器插件的存在是 logstash 威力如此強大的重要因素。名爲過濾器，其實提供的不僅僅是過濾的功能，它們擴展了進入過濾器的原始數據，進行復雜的邏輯處理，甚至能夠無中生有的添加新的 logstash 事件到後續的流程中去！linux
Grok 是 Logstash 最重要的插件。你能夠在 grok 裏預約義好命名正則表達式，在稍後(grok參數或者其餘正則表達式裏)引用它。git
大多數Linux使用人員都有過用正則表達式來查詢機器中相關文件或文件裏內容的經歷，在Grok裏，咱們也是使用正則表達式來識別日誌裏的相關數據塊。github
有兩種方式來使用正則表達式：正則表達式
直接寫正則來匹配apache
用Grok表達式映射正則來匹配vim
重要提示：Grok表達式很像C語言裏的宏定義windows
（二）grok語法tomcat
grok表達式的打印複製格式的完整語法是下面這樣的：ruby
%{PATTERN_NAME:capture_name:data_type}
小貼士：data_type 目前只支持兩個值：int 和 float。
在線gork正則的地址：http://grokdebug.herokuapp.com/
Logstash基礎正則地址：https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns
也能夠在你的安裝路徑下查找grok-patterns內置的正則表達式：
[root@localhost patterns]# cat /usr/local/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-2.0.2/patterns/grok-patterns
USERNAME [a-zA-Z0-9._-]+
USER %{USERNAME}
EMAILLOCALPART [a-zA-Z][a-zA-Z0-9_.+-=:]+
EMAILADDRESS %{EMAILLOCALPART}@%{HOSTNAME}
HTTPDUSER %{EMAILADDRESS}|%{USER}
INT (?:[+-]?(?:[0-9]+))
BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))
NUMBER (?:%{BASE10NUM})
BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))
BASE16FLOAT \b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b
POSINT \b(?:[1-9][0-9]*)\b
NONNEGINT \b(?:[0-9]+)\b
WORD \b\w+\b
NOTSPACE \S+
SPACE \s*
DATA .*?
GREEDYDATA .*
QUOTEDSTRING (?>(?<!\\)(?>"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``))
UUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}
# Networking
MAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})
CISCOMAC (?:(?:[A-Fa-f0-9]{4}\.){2}[A-Fa-f0-9]{4})
WINDOWSMAC (?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})
COMMONMAC (?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})
IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?
IPV4 (?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9])
IP (?:%{IPV6}|%{IPV4})
HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)
IPORHOST (?:%{IP}|%{HOSTNAME})
HOSTPORT %{IPORHOST}:%{POSINT}
# paths
PATH (?:%{UNIXPATH}|%{WINPATH})
UNIXPATH (/([\w_%!$@:.,~-]+|\\.)*)+
TTY (?:/dev/(pts|tty([pq])?)(\w+)?/?(?:[0-9]+))
WINPATH (?>[A-Za-z]+:|\\)(?:\\[^\\?*]*)+
URIPROTO [A-Za-z]+(\+[A-Za-z+]+)?
URIHOST %{IPORHOST}(?::%{POSINT:port})?
# uripath comes loosely from RFC1738, but mostly from what Firefox
# doesn't turn into %XX
URIPATH (?:/[A-Za-z0-9$.+!*'(){},~:;=@#%_\-]*)+
#URIPARAM \?(?:[A-Za-z0-9]+(?:=(?:[^&]*))?(?:&(?:[A-Za-z0-9]+(?:=(?:[^&]*))?)?)*)?
URIPARAM \?[A-Za-z0-9$.+!*'|(){},~@#%&/=:;_?\-\[\]<>]*
URIPATHPARAM %{URIPATH}(?:%{URIPARAM})?
URI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?
# Months: January, Feb, 3, 03, 12, December
MONTH \b(?:Jan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?)\b
MONTHNUM (?:0?[1-9]|1[0-2])
MONTHNUM2 (?:0[1-9]|1[0-2])
MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
# Days: Monday, Tue, Thu, etc...
DAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)
# Years?
YEAR (?>\d\d){1,2}
HOUR (?:2[0123]|[01]?[0-9])
MINUTE (?:[0-5][0-9])
# '60' is a leap second in most time standards and thus is valid.
SECOND (?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)
TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it)
DATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}
DATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}
ISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))
ISO8601_SECOND (?:%{SECOND}|60)
TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?
DATE %{DATE_US}|%{DATE_EU}
DATESTAMP %{DATE}[- ]%{TIME}
TZ (?:[PMCE][SD]T|UTC)
DATESTAMP_RFC822 %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}
DATESTAMP_RFC2822 %{DAY}, %{MONTHDAY} %{MONTH} %{YEAR} %{TIME} %{ISO8601_TIMEZONE}
DATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}
DATESTAMP_EVENTLOG %{YEAR}%{MONTHNUM2}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND}
HTTPDERROR_DATE %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}
# Syslog Dates: Month Day HH:MM:SS
SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}
PROG [\x21-\x5a\x5c\x5e-\x7e]+
SYSLOGPROG %{PROG:program}(?:\[%{POSINT:pid}\])?
SYSLOGHOST %{IPORHOST}
SYSLOGFACILITY <%{NONNEGINT:facility}.%{NONNEGINT:priority}>
HTTPDATE %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}
# Shortcuts
QS %{QUOTEDSTRING}
# Log formats
SYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:
COMMONAPACHELOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
HTTPD20_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:errormsg}
HTTPD24_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}:tid %{NUMBER:tid}\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_errormessage}:)?( \[client %{IPORHOST:client}:%{POSINT:clientport}\])? %{DATA:errorcode}: %{GREEDYDATA:message}
HTTPD_ERRORLOG %{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}
# Log Levels
LOGLEVEL ([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)
[root@localhost patterns]#
（三）經常使用的表達式說明：
（1），USERNAME或USER。用戶名或用戶，由數字.大小寫字母.下劃線及特殊字符（._-）組成的字符串
USERNAME [a-zA-Z0-9._-]+
USER %{USERNAME}
第一行，用普通的正則表達式來定義一個 grok 表達式；第二行，經過打印賦值格式，用前面定義好的 grok 表達式來定義另外一個 grok 表達式。
eg:123,Alice,liqb 等等。
（2），EMAILLOCALPART。電子郵件用戶名部分，首位由大小寫字母組成，其餘部分是由大小寫字母.數字及特殊字符(_.+-=:）組成的字符串(備註：注意，國內的QQ純數字郵箱帳號是沒法匹配的，須要修改正則)。
EMAILLOCALPART [a-zA-Z][a-zA-Z0-9_.+-=:]+
eg: lqb,Grace_li,abc-wang
（3）,EMAILADDRESS。電子郵件地址：定義了電子郵件用戶名：%{EMAILLOCALPART}，定義了主機名：%{HOSTNAME}
EMAILADDRESS %{EMAILLOCALPART}@%{HOSTNAME}
eg:alice@yahoo.cn,alice@126.com,abc-123@qq.com
（4），HTTPDUSER。定義了apache服務器的用戶，能夠是EMAILADDRESS 或 USER
HTTPDUSER %{EMAILADDRESS} | %{USER}
(5) , INT :整數，包括0和正負整數
INT (?:[+-]?(?:[0-9]+))
eg:0 -123 123 2345
(6),BASE10NUM 或NUMBER :十進制數字，包括整數和小數。
BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))) eg:11 33 3.14
(7),BASE16NUM:十六進制數字，整數
BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+)) eg:0x0045fa2d、-0x3F8709
（8）,BASE16FLOAT:十六進制數字，整數和小數
BASE16FLOAT \b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b
(9),WORD:字符串，包括數字和大小寫字母。
WORD \b\w+\b eg:String . 34128 Ilove YOU
\b:匹配一個單詞邊界，也就是指單詞和空格間的位置。例如， 'er\b' 能夠匹配"never" 中的 'er'，但不能匹配 "verb" 中的 'er'。
\s:匹配包括下劃線的任何單詞字符。等價於'[A-Za-z0-9_]'。
(10),NOTSPACE:不帶任何空格的字符串
NOTSPACE \S+
\S:匹配任何非空白字符。等價於 [^ \f\n\r\t\v]。
(11),SPACE，空格字符串
SPACE \s*
\s:匹配任何空白字符，包括空格、製表符、換頁符等等。等價於 [ \f\n\r\t\v]。
(12),QUOTEDSTRING 或QS:帶引號的字符串。
QUOTEDSTRING (?>(?<!\\)(?>"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``)) eg:"this is an apple" "hello world"
(13),UUID:標準的uuid
UUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}
eg:b808962d-8f1a-4347-8389-5516d10e875a
(14),MAC:MAC 地址，能夠是思科設備裏得mac地址，也能夠是windows裏得mac地址
MAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})
eg:00:15:5D:6E:28:13
（15）,IP :IP地址，能夠是IPv4或IPv6地址
IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?
IPV4 (?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9])
IP (?:%{IPV6}|%{IPV4})
（16）,HOSTNAME：主機名稱
HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)
（17）IPORHOST：ip或者主機名稱
IPORHOST (?:%{IP}|%{HOSTNAME})
（18）,HOSTPORT :主機名(IP)+端口
HOSTPORT %{IPORHOST}:%{POSINT}
eg:192.168.180.21:3306
(19）,PATH:UNIX系統或windows系統裏得路徑模式
PATH (?:%{UNIXPATH}|%{WINPATH})
eg: /usr/local/tomcat/bin/startuo.sh D:\upload_crm\startup.bat
（20）,URIPROTO:URL協議
URIPROTO [A-Za-z]+(\+[A-Za-z+]+)? eg:http ftp tcp udp
(21),URIHOST:URL主機
URIHOST %{IPORHOST}(?::%{POSINT:port})?
eg:www.baidu.com 121.242.156.210:24444
（22）,URIPATH:URL路徑
URIPATH (?:/[A-Za-z0-9$.+!*'(){},~:;=@#%_\-]*)+
eg:http://udn.yyuap.com/doc/logstash-best-practice-cn/input/stdin.html
(23),URIPARAM :URL裏得GET參數
URIPARAM \?[A-Za-z0-9$.+!*'|(){},~@#%&/=:;_?\-\[\]<>]*
eg:?a=1&b=2&c=3
(24）URIPATHPARAM：URL路徑+GET參數
URIPATHPARAM %{URIPATH}(?:%{URIPARAM})?
（25）URI：完整的URL路徑
URI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?
日期表達式：
（26）MONTH:月份名稱
MONTH \b(?:Jan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?)\b eg:jan january Nov November
(27）,MONTHNUM:月份數字
MONTHNUM (?:0?[1-9]|1[0-2]) eg:03 3 12
(28),MONTHDAY:日期數字
MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) eg: 03 9 31
(29）,DAY :星期幾名稱
DAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?) eg MON MONDAY Tue Tuesday
(30),YEAR:年份數字
YEAR (?>\d\d){1,2}
eg 2012 2017
(31),小時數字
HOUR (?:2[0123]|[01]?[0-9])
（32）,MINUTE :分鐘數字
MINUTE (?:[0-5][0-9])
（33）,SECOND：秒數字
SECOND (?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)
（34）,TIME：時間
TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
eg 01:00:45
（35），DATE_US：美國日期格式
DATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}
eg:10/12/1986
(36),DATE_EU:歐洲日期格式
DATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}
eg:15-10-1986
(37),ISO8601_TIMEZONE:ISO8601時間格式
ISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))
eg：+15:19 -15:19
（38）,TIMESTAMP_ISO8601:ISO8601時間戳格式
TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?
(39),DATE ：日期，美國日期或者歐洲日期
DATE %{DATE_US}|%{DATE_EU}
（40）DATESTAMP完整日期+時間
DATESTAMP %{DATE}[- ]%{TIME}
（41）,http默認日期格式
HTTPDATE %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}
（42）,LOGLEVEL :日誌等級
LOGLEVEL ([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)
（四）.grok正則捕獲
(?#...) 否 註釋，拋棄
(?:...) 是 只集羣，不捕獲的圓括弧
命名分組格式爲(?<grp name>)
命名分組的匹配的結果存在在變量%+變量中，取命名分組值，$+{grp name}.
數字 [0-9] \d
\d+
空白 [\t\n\r\f] \s
詞 [a-zA-Z_0-9] \w
[root@localhost test]# vim grok.conf
input {stdin {}}
filter {
grok {
match =>{
"message" =>"\s+(?<request_time>\d+(?:\.\d+)?)\s+"
}
}
}
output {
stdout {
codec => rubydebug
}
}
[root@localhost logstash]# /usr/local/logstash/bin/logstash -f test/grok.conf
Settings: Default pipeline workers: 1
Logstash startup completed
begin 123.45678 end
{
"message" => " begin 123.45678 end",
"@version" => "1",
"@timestamp" => "2017-05-26T02:30:22.884Z",
"host" => "localhost.localdomain",
"request_time" => "123.45678"
}
（2）使用grok正則語法
[root@localhost test]# vim grok_match.conf
input {stdin {}}
filter {
grok {
match =>{
"message" =>"\s+(?<request_time>\d+(?:\.\d+)?)\s+"
}
}
}
filter {
grok {
match => {
"message" => "%{WORD} %{NUMBER:request_time:float} %{WORD}"
}
}
}
output {
stdout {
codec => rubydebug
}
}
[root@localhost logstash]# /usr/local/logstash/bin/logstash -f test/grok_match.
conf
begin 123.4321 end
{
"message" => "begin 123.4321 end",
"@version" => "1",
"@timestamp" => "2017-05-26T02:41:26.719Z",
"host" => "localhost.localdomain",
"request_time" => [
[0] "123.4321",
[1] 123.4321
]
}
begin 2231 lqb
{
"message" => "begin 2231 lqb",
"@version" => "1",
"@timestamp" => "2017-05-26T02:39:33.826Z",
"host" => "localhost.localdomain",
"request_time" => [
[0] "2231",
[1] 2231.0
]
}
相關文章
- 1. logstash使用grok正則解析日誌
- 2. 日誌分析logstash插件-grok詳解
- 3. 關於Logstash中grok插件的正則表達式例子
- 4. 【Logstash】Filter plugins Grok filter plugin正則匹配
- 5. logstash實戰filter插件之grok（收集apache日誌）
- 6. logstash-grok解析nginx+php日誌
- 7. Logstash grok解析IIS 10.0 日誌實例
- 8. logstash grok正則表達式測試工具GrokConstructor
- 9. logstash 正則表達式
- 10. logstash filter grok過濾日誌文件的配置方法
- 更多相關文章...
- • Scala 正則表達式 - Scala教程
- • PHP 正則表達式(PCRE) - PHP參考手冊
- • Tomcat學習筆記（史上最全tomcat學習筆記）
- • 互聯網組織的未來：剖析GitHub員工的任性之源
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
最新文章
- 1. 在windows下的虛擬機中,安裝華爲電腦的deepin操作系統
- 2. 強烈推薦款下載不限速解析神器
- 3. 【區塊鏈技術】孫宇晨：區塊鏈技術帶來金融服務的信任變革
- 4. 搜索引起的鏈接分析-計算網頁的重要性
- 5. TiDB x 微衆銀行 | 耗時降低 58%，分佈式架構助力實現普惠金融
- 6. 《數字孿生體技術白皮書》重磅發佈（附完整版下載）
- 7. 雙十一「避坑」指南：區塊鏈電子合同爲電商交易保駕護航！
- 8. 區塊鏈產業，怎樣「鏈」住未來？
- 9. OpenglRipper使用教程
- 10. springcloud請求一次好用一次不好用zuul Name or service not known
歡迎關注本站公眾號,獲取更多信息
相關文章
- 1. logstash使用grok正則解析日誌
- 2. 日誌分析logstash插件-grok詳解
- 3. 關於Logstash中grok插件的正則表達式例子
- 4. 【Logstash】Filter plugins Grok filter plugin正則匹配
- 5. logstash實戰filter插件之grok（收集apache日誌）
- 6. logstash-grok解析nginx+php日誌
- 7. Logstash grok解析IIS 10.0 日誌實例
- 8. logstash grok正則表達式測試工具GrokConstructor
- 9. logstash 正則表達式
- 10. logstash filter grok過濾日誌文件的配置方法