Nginx集成Kafka

host：xupan002

 

librdkafka簡介

librdkafka 是一個C實現的高性能 Apache Kafka 客戶端，爲生產環境提供了一個可靠和高性能的客戶端。 librdkafka 一樣也提供了傳統的 C++ 接口。

 

性能

librdkafka 是一個基於現代硬件設計的多線程庫， 而且試圖保持最少的內存拷貝。 
若是應用程序願意，生產和消費消息的載體能夠不經過任何拷貝實現讓消息大小不受限制。librdkafka 一樣適用於高吞吐仍是低延時的場景，均可以經過屬性配置接口來知足。

 

librdkafka

cd /usr/local/devtools/nginx/plugin
git clone https://github.com/edenhill/librdkafka
cd librdkafka 
./configure
make
make install

 

ngx_kafka_module：

cd /usr/local/devtools/nginx/plugin
git clone https://github.com/brg-liuwei/ngx_kafka_module

cd /usr/local/devtools/nginx/nginx-1.12.2
./configure --add-module=/usr/local/devtools/nginx/plugin/ngx_kafka_module/
make
make install

 

配置nginx.conf

kafka;
kafka_broker_list xupan001:9092 xupan002:9093 xupan003:9093;

location = /kafka/track {
     kafka_topic track;
}

location = /kafka/user {
 kafka_topic user;
}
=================================================

echo "/usr/local/lib" >> /etc/ld.so.conf
ldconfig

 

 

測試

--kafka建立topic
kafka-topics.sh -zookeeper xupan001:2181,xupan002:2181,xupan003:2181 --create --topic track --replication-factor 3 --partitions 3     
kafka-topics.sh -zookeeper xupan001:2181,xupan002:2181,xupan003:2181 --create --topic user --replication-factor 3 --partitions 3      

--kafka建立消費者
./bin/kafka-console-consumer.sh --zookeeper xupan001:2181,xupan002:2181,xupan003:2181 --topic track --from-beginning

curl xupan002/kafka/track  -d "hello ngx_kafka_module"