一個簡單定製的Logstash filter

官方參考：https://www.elastic.co/guide/en/logstash/current/how_to_write_a_logstashfilter_plugin.html

官方實例：https://github.com/logstash-plugins/logstash-filter-example/

網上guide：http://www.cnblogs.com/xing901022/p/5259750.html

插件plugin命名：logstash-filter-seq

通過測試，實際只須要最少兩個文件：

1 logstash-filter-seq.gemspec

2 lib\logstash\filters\seq.rb

內容分別以下：

Gem::Specification.new do |s|
  s.name = 'logstash-filter-seq'
  s.version         = '1.0.0'
  s.licenses = ['Apache License (2.0)']
  s.summary = "This seq filter adds sequence to each document during filtering."
  s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
  s.authors = ["zhaoxp"]
  s.email = 'zhaoxp2@lenovo.com'
  s.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html"
  s.require_paths = ["lib"]

  # Files
  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md']
   # Tests
  s.test_files = s.files.grep(%r{^(test|spec|features)/})

  # Special flag to let us know this is actually a logstash plugin
  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "filter" }

  # Gem dependencies
  #s.add_runtime_dependency "logstash-core", ">= 2.0.0", "< 3.0.0"
  #s.add_development_dependency 'logstash-devutils'
end

 seq.rb

# encoding: utf-8
require "logstash/filters/base"
require "logstash/namespace"

# This example filter will replace the contents of the default 
# message field with whatever you specify in the configuration.
#
# It is only intended to be used as an example.
class LogStash::Filters::Seq < LogStash::Filters::Base

  # Setting the config_name here is required. This is how you
  # configure this filter from your Logstash config.
  #
  # filter {
  #   seq {
  #     message => "My message..."
  #   }
  # }
  #
  config_name "seq"
  
  # Replace the message with this value.
  #config :message, :validate => :string, :default => "Hello World!"
  config :seqname, :validate => :string, :default => "seq"
  

  public
  def register
    # Add instance variables 
    @lineindex=0
  end # def register

  public
  def filter(event)

#    if @message
      # Replace the event message with our message as configured in the
      # config file.
#      event["message"] = @message
#    end

    # filter_matched should go in the last line of our successful code
    #filter_matched(event)
    @lineindex=@lineindex+1
	event[seqname]=@lineindex
  end # def filter
end # class LogStash::Filters::Seq

部署方法：

1 將logstash-filter-seq目錄放入logstash下vendor\bundle\jruby\1.9\gems中。

2 修改logstash下的Gemfile文件，增長一行：

gem "logstash-filter-seq", :path => "vendor/bundle/jruby/1.9/gems/logstash-filter-seq-1.0.0"

 

使用方法：

filter{
  seq{
    seqname=> "testseq"
  }
}

這樣的話，使用這個filter最終會產生一個屬性名爲testseq，值爲數值而且逐個加一的屬性值。

當初使用這個filter的緣由是由於如下的filter達不到目的。由於這是以時間值做爲屬性值，若是logstash處理速度過快，就會出現相鄰兩條記錄的daytag值同樣。

ruby {
  code => "event['daytag'] = event.timestamp.time.localtime.strftime('%Y-%m-%d');event['seq'] = Time.now.strftime('%Y%m%d%H%M%S%L').to_i"
#  code => "event['daytag'] = event.timestamp.time.localtime.strftime('%Y-%m-%d')"
}

 

這個例子太多簡單，實際要考慮到的問題還有：

若是logstash重啓，那麼sequence會從1開始從新計算，因此有兩種考慮方案：

1 再增長一個屬性，來表示sequence開始計時時的時間。

2 在plugin中增長一個屬性，表示一個存放文件的路徑。這個文件中記錄了上次sequence最後的值，這樣涉及到一個問題是sequence刷新時間。這參考一下logstash-input-file這個plugin。