logstash 操做redis

在實際應用中,Logstash進程會被氛圍兩個不一樣的角色。


運行在應用服務器上的儘可能減輕運行壓力,只作讀取和轉發,這個角色叫作shipper


運行在獨立的服務器上完成數據解析處理,負責寫入到Elasticsearch的角色,叫作Indexer



5.1.1 讀取redis 數據:

zjtest7-frontend:/usr/local/logstash-2.3.4/config# ../bin/logstash -f redis.comf 
Settings: Default pipeline workers: 1
Pipeline main started
{
       "message" => "Hello world",
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
    "@timestamp" => "2016-08-19T06:26:12.854Z"
}

zjtest7-frontend:/usr/local/logstash-2.3.4/config# cat redis.comf 
input {
    redis {
       data_type =>"pattern_channel"
       key =>"logstash-*"
       host=>"192.168.32.67"
       port=>6379
       password => "1234567"
          }
}

output {
 stdout {
  codec=>rubydebug{}
   }
}



採用list類型擴展Logstash：

127.0.0.1:6379> PUBLISH logstash-list "Hello xxxxxx"
(integer) 2
127.0.0.1:6379> PUBLISH logstash-list "Hello yyyyy"



zjtest7-frontend:/usr/local/logstash-2.3.4/config# ../bin/logstash -f redis.comf 

Settings: Default pipeline workers: 1
Pipeline main started
{
       "message" => "Hello xxxxxx",
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
    "@timestamp" => "2016-08-19T07:46:27.031Z"
}
{
       "message" => "Hello yyyyy",
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
    "@timestamp" => "2016-08-19T07:46:37.365Z"





zjtest7-frontend:/usr/local/logstash-2.3.4/config# ../bin/logstash -f redis.comf 

Settings: Default pipeline workers: 1
Pipeline main started
{
       "message" => "Hello xxxxxx",
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
    "@timestamp" => "2016-08-19T07:46:26.964Z"
}
{
       "message" => "Hello yyyyy",
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
    "@timestamp" => "2016-08-19T07:46:37.362Z"
}


zjtest7-frontend:/usr/local/logstash-2.3.4/config# cat redis.comf 
input {
    redis {
       data_type =>"pattern_channel"
       key =>"logstash-list"
       host=>"192.168.32.67"
       port=>6379
       password => "1234567"
          }
}

output {
 stdout {
  codec=>rubydebug{}
   }
}


兩個終端同時啓動logstash -f redis.conf 進程,結果會是兩個終端都輸出消息。




這個時候,就須要用list 類型,在這種類型中,數據輸入到Redis 服務器上暫存,Logstash 則連上Redis 服務器取走(BLPOP命令,因此只要logstash不堵塞,redis 服務器上也不會

有數據堆積佔用空間)數據。




1.配置示例:

zjtest7-frontend:/usr/local/logstash-2.3.4/config# cat redis.comf 
input {
    redis {
       data_type =>"list"
       key =>"logstash-list"
       host=>"192.168.32.67"
       port=>6379
       password => "1234567"
          }
}

output {
 stdout {
  codec=>rubydebug{}
   }
}


這時候能夠看到, 只有一個終端輸出告終果

連續RPUSH幾回, 能夠看到兩個終端近乎各自輸出一半條目。



3.批量推送:
RPUSH 支持batch 方式,修改Logstash 配置中的batch_count值,





5.1.3  輸出到Redis：

1.配置示例


zjtest7-frontend:/usr/local/logstash-2.3.4/config# cat inputredis.conf 
input {stdin {} }
     output {
     redis {
          data_type =>"channel"
          key=>"logstash-chan-%{+yyyy.MM.dd}"
          host=>"192.168.32.67"
          port=>6379
          password => "1234567"
 }
}



127.0.0.1:6379> SUBSCRIBE logstash-chan-2016.08.19
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "logstash-chan-2016.08.19"
3) (integer) 1
1) "message"
2) "logstash-chan-2016.08.19"
3) "{\"message\":\"\",\"@version\":\"1\",\"@timestamp\":\"2016-08-19T08:27:07.190Z\",\"host\":\"0.0.0.0\"}"
1) "message"
2) "logstash-chan-2016.08.19"
3) "{\"message\":\"\",\"@version\":\"1\",\"@timestamp\":\"2016-08-19T08:27:07.736Z\",\"host\":\"0.0.0.0\"}"
1) "message"
2) "logstash-chan-2016.08.19"
3) "{\"message\":\"\",\"@version\":\"1\",\"@timestamp\":\"2016-08-19T08:27:07.772Z\",\"host\":\"0.0.0.0\"}"
1) "message"
2) "logstash-chan-2016.08.19"
3) "{\"message\":\"\",\"@version\":\"1\",\"@timestamp\":\"2016-08-19T08:27:07.808Z\",\"host\":\"0.0.0.0\"}"
1) "message"
2) "logstash-chan-2016.08.19"
3) "{\"message\":\"\",\"@version\":\"1\",\"@timestamp\":\"2016-08-19T08:27:07.844Z\",\"host\":\"0.0.0.0\"}"
1) "message"
2) "logstash-chan-2016.08.19"
3) "{\"message\":\"\",\"@version\":\"1\",\"@timestamp\":\"2016-08-19T08:27:07.880Z\",\"host\":\"0.0.0.0\"}"
1) "message"
2) "logstash-chan-2016.08.19"
3) "{\"message\":\"\",\"@version\":\"1\",\"@timestamp\":\"2016-08-19T08:27:07.916Z\",\"host\":\"0.0.0.0\"}"
1) "message"
2) "logstash-chan-2016.08.19"
3) "{\"message\":\"hello world\",\"@version\":\"1\",\"@timestamp\":\"2016-08-19T08:27:10.486Z\",\"host\":\"0.0.0.0\"}"
1) "message"
2) "logstash-chan-2016.08.19"
3) "{\"message\":\"what'sup scab\",\"@version\":\"1\",\"@timestamp\":\"2016-08-19T08:27:37.919Z\",\"host\":\"0.0.0.0\"}"