在某些狀況下，有些日誌文本文件類json，但它的是單引號，具體格式以下，咱們須要根據下列日誌數據，獲取正確的字段和字段類型
{'usdCnyRate': '6.728', 'futureIndex': '463.36', 'timestamp': '1532933162361'} {'usdCnyRate': '6.728', 'futureIndex': '463.378', 'timestamp': '1532933222335'} {'usdCnyRate': '6.728', 'futureIndex': '463.38', 'timestamp': '1532933348347'} {'usdCnyRate': '6.728', 'futureIndex': '463.252', 'timestamp': '1532933366866'} {'usdCnyRate': '6.728', 'futureIndex': '463.31', 'timestamp': '1532933372350'} {'usdCnyRate': '6.728', 'futureIndex': '463.046', 'timestamp': '1532933426899'} {'usdCnyRate': '6.728', 'futureIndex': '462.806', 'timestamp': '1532933432346'} {'usdCnyRate': '6.728', 'futureIndex': '462.956', 'timestamp': '1532933438353'} {'usdCnyRate': '6.728', 'futureIndex': '462.954', 'timestamp': '1532933456796'} {'usdCnyRate': '6.728', 'futureIndex': '462.856', 'timestamp': '1532933492411'} {'usdCnyRate': '6.728', 'futureIndex': '462.776', 'timestamp': '1532933564378'} {'usdCnyRate': '6.728', 'futureIndex': '462.628', 'timestamp': '1532933576849'} {'usdCnyRate': '6.728', 'futureIndex': '462.612', 'timestamp': '1532933588338'} {'usdCnyRate': '6.728', 'futureIndex': '462.718', 'timestamp': '1532933636808'}
此時咱們若是當json直接用logstash Json filter plugin來解析會以下報錯
[WARN ] 2018-07-31 10:20:12.708 [Ruby-0-Thread-5@[main]>worker1: :1] json - Error parsing json {:source=>"message", :raw=>"{'usdCnyRate': '6.728', 'futureIndex': '462.134', 'timestamp': '1532933714371'}", :exception=>#<LogStash::Json::ParserError: Unexpected character (''' (code 39)): was expecting double-quote to start field name at [Source: (byte[])"{'usdCnyRate': '6.728', 'futureIndex': '462.134', 'timestamp': '1532933714371'}"; line: 1, column: 3]>}
此處我認爲簡單的作法是替換單引號爲雙引號，替換過程應用了logstash mutate gsub
必定要看清楚我10-12行的寫法，做用爲替換字符串，14-15行爲解析json。咱們還須要將usdCnyRate和futureIndex轉爲float類型（18-21行），將timestamp轉爲時間類型，並從新定義一個logdate來存儲（23-25行）此處用到
logstash date filter plugin
input{
file {
path => "/usr/share/logstash/wb.cond/test.log" start_position => "beginning" sincedb_path => "/dev/null" } } filter{ mutate { gsub =>[ "message", "'", '"' ] } json { source => "message" } mutate { convert => { "usdCnyRate" => "float" "futureIndex" => "float" } } date { match => [ "timestamp", "UNIX_MS" ] target => "logdate" } } output{ stdout{ codec=>rubydebug } }
利用上述配置文件，咱們能正確解析出日誌文件的字段和類型
{
"message" => "{\"usdCnyRate\": \"6.728\", \"futureIndex\": \"463.378\", \"timestamp\": \"1532933222335\"}", "@timestamp" => 2018-07-31T10:48:48.600Z, "host" => "logstashvm0", "path" => "/usr/share/logstash/wb.cond/test.log", "@version" => "1", "logdate" => 2018-07-30T06:47:02.335Z, "usdCnyRate" => 6.728, "timestamp" => "1532933222335", "futureIndex" => 463.378 } { "message" => "{\"usdCnyRate\": \"6.728\", \"futureIndex\": \"463.252\", \"timestamp\": \"1532933366866\"}", "@timestamp" => 2018-07-31T10:48:48.602Z, "host" => "logstashvm0", "path" => "/usr/share/logstash/wb.cond/test.log", "@version" => "1", "logdate" => 2018-07-30T06:49:26.866Z, "usdCnyRate" => 6.728, "timestamp" => "1532933366866", "futureIndex" => 463.252 } { "message" => "{\"usdCnyRate\": \"6.728\", \"futureIndex\": \"463.31\", \"timestamp\": \"1532933372350\"}", "@timestamp" => 2018-07-31T10:48:48.602Z, "host" => "logstashvm0", "path" => "/usr/share/logstash/wb.cond/test.log", "@version" => "1", "logdate" => 2018-07-30T06:49:32.350Z, "usdCnyRate" => 6.728, "timestamp" => "1532933372350", "futureIndex" => 463.31 }