此jar是由Cloudera的CDH發行版提供的一個工具類,經過配置,能夠將log4j的日誌直接輸出到flume,方便日誌的採集。java
在CDH5.3.0版本中是:flume-ng-log4jappender-1.5.0-cdh5.3.0-jar-with-dependencies.jarapache
所在目錄是:/opt/cloudera/parcels/CDH/lib/flume-ng/tools/服務器
log4j.category.com.xxx=INFO,console,flume log4j.appender.console=org.apache.log4j.ConsoleAppender log4j.appender.console.target=System.out log4j.appender.console.layout=org.apache.log4j.PatternLayout log4j.appender.console.layout.ConversionPattern="%d{yyyy-MM-dd HH:mm:ss} %p [%c:%L] - %m%n log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender log4j.appender.flume.Hostname = localhost log4j.appender.flume.Port = 4444 log4j.appender.flume.UnsafeMode = true log4j.appender.flume.layout=org.apache.log4j.PatternLayout log4j.appender.flume.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %p [%c:%L] - %m%n
在classpath中要包含log4j.properties和flume-ng-log4jappender-1.5.0-cdh5.3.0-jar-with-dependencies.jarapp
import java.util.Date; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; public class WriteLog { protected static final Log logger = LogFactory.getLog(WriteLog.class); public static void main(String[] args) throws InterruptedException { // TODO Auto-generated method stub while (true) { // 每隔兩秒log輸出一下當前系統時間戳 logger.info(new Date().getTime()); Thread.sleep(2000); try { throw new Exception("exception msg"); } catch (Exception e) { logger.error("error:" + e.getMessage()); } } } }
a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe/configure the source a1.sources.r1.type = avro a1.sources.r1.bind = localhost a1.sources.r1.port = 4444 # Describe the sink a1.sinks.k1.type = file_roll a1.sinks.k1.sink.directory = /data/soft/flume/tmp a1.sinks.k1.sink.rollInterval=86400 a1.sinks.k1.sink.batchSize=100 a1.sinks.k1.sink.serializer=text a1.sinks.k1.sink.serializer.appendNewline = false # Use a channel which buffers events in memory a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 1000 # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1
將相關程序上傳到服務器,並先啓動agent
工具
進入flume安裝目錄後,執行
測試
bin/flume-ng agent -c conf -f conf/avro.conf --name a1 -Dflume.root.logger=INFO,consolespa
執行測試程序
日誌
java -classpath ./:flume-ng-log4jappender-1.5.0-cdh5.3.0-jar-with-dependencies.jar:test.jar com.xxx.WriteLogcode
tail -f /data/soft/flume/tmp/1436164166461-1 ci
2015-07-06 14:51:36 ERROR [com.xxx.WriteLog:27] - error:exception msg 2015-07-06 14:51:36 ERROR [com.xxx.WriteLog:28] - error:stack 2015-07-06 14:51:36 INFO [com.xxx.WriteLog:21] - 1436165496975 2015-07-06 14:51:38 ERROR [com.xxx.WriteLog:27] - error:exception msg 2015-07-06 14:51:38 ERROR [com.xxx.WriteLog:28] - error:stack 2015-07-06 14:51:38 INFO [com.xxx.WriteLog:21] - 1436165498977