kafka+log4j配置

前言。好多網上的整合都不能使用了。現更新最新的。

1、安裝和配置kafka.參考http://kafka.apache.org/quickstart 

tar -xzf kafka_2.11-0.10.2.0.tgz
cd kafka_2.11-0.10.2.0

修改config/server.properties. 打開listeners. 根據實際狀況修改zookeeper.connect

listeners=PLAINTEXT://192.168.1.9:9092
zookeeper.connect=ser9:2181,ser40:2181

若是增長爲集羣模式：broker.id爲集羣中不一樣的編號，要惟一。參考http://kafka.apache.org/documentation/#prodconfig

broker.id=1
listeners=PLAINTEXT://192.168.1.40:9092
log.dirs=/var/log/kafka-logs
num.partitions=8
default.replication.factor=3
min.insync.replicas=2
auto.create.topics.enable=true

開啓服務

bin/kafka-server-start.sh config/server.properties &

建立主題

bin/kafka-topics.sh --create --zookeeper ser9:2181 --replication-factor 1 --partitions 1 --topic test

2、修改pom.xml

<dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>0.10.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-log4j-appender</artifactId>
            <version>0.10.2.0</version>
        </dependency>

log4j.properties

log4j.logger.org.apache = DEBUG,kafka

# appender kafka
log4j.appender.kafka=org.apache.kafka.log4jappender.KafkaLog4jAppender
log4j.appender.kafka.topic=test
# multiple brokers are separated by comma ",".
log4j.appender.kafka.brokerList=192.168.1.9:9092
log4j.appender.kafka.compressionType=none
log4j.appender.kafka.syncSend=true
log4j.appender.kafka.layout=org.apache.log4j.PatternLayout
log4j.appender.kafka.layout.ConversionPattern=%-d{yyyy-MM-dd HH\:mm\:ss,SSS} [%5p] (%F\:%L) - %m%n

3、打開consumer

bin/kafka-console-consumer.sh --bootstrap-server ser9:9092 --topic test --from-beginning