kafka/zookeeper學習記錄

 

1，kafka依賴於zookeeper，下載：

kafka2.10-0.10.00包下載，zookeeper3.4.10下載；

2，配置啓動ZOOKEEPER

    配置項：ZOOKEEPER_HOME，和PATH;參考：

修改zookeeper-3.4.10/conf下，zoo.conf文件：

設置項：

dataDir=/home/t/source/zookeeper-3.4.10/dataDir
dataLogDir=/home/t/source/zookeeper-3.4.10/dataLogDir

zookeeper啓動：

./zkServer.sh start

3，配置啓動kafka

修改kafka配置項：

啓動kafka

./kafka-server-start.sh  ../config/server.properties

建立topic（消息類型）

./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

生產消息：

./kafka-console-producer.sh  --broker-list localhost:9092 --topic test

消費消息：

./kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

最終效果：

生產端輸入什麼，消費端輸出什麼。

partition

分區，對於一個topic，3個分區，則同一組消費者數量應當<=3，不然有消費者接受不到數據；

http://www.cnblogs.com/liuwei6/p/6900686.html

Topic在邏輯上能夠被認爲是一個queue，每條消費都必須指定它的Topic，能夠簡單理解爲必須指明把這條消息放進哪一個queue裏。爲了使得Kafka的吞吐率能夠線性提升，物理上把Topic分紅一個或多個Partition，每一個Partition在物理上對應一個文件夾，該文件夾下存儲這個Partition的全部消息和索引文件。

kafka外網訪問 advertised.listeners=PLAINTEXT://x.x.x.x:9092

kafka讀寫

import org.apache.kafka.clients.consumer.ConsumerRecord;  
import org.apache.kafka.clients.consumer.ConsumerRecords;  
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.TopicPartition;

import java.util.Arrays;
import java.util.Date;
import java.util.Properties;

import javax.print.attribute.standard.PrinterLocation;  
  
public class KafkaConsumerExample {  
    public static void main(String[] args) throws InterruptedException {  
        Properties props = new Properties();  
        props.put("bootstrap.servers", "192.168.1.166:9092");  
        props.put("group.id", "test13");  
        props.put("enable.auto.commit", "true");  
        props.put("auto.commit.interval.ms", "1000");  
        props.put("session.timeout.ms", "30000");  
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");  
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");  
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");  
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");  
        props.put("auto.offset.reset", "earliest");        
        props.put(ProducerConfig.BATCH_SIZE_CONFIG, 1024*1024*5);  
        //往kafka服務器提交消息間隔時間，0則當即提交不等待  
        props.put(ProducerConfig.LINGER_MS_CONFIG,0);  
        
        
        //Kafka Reader
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);  
        consumer.subscribe(Arrays.asList("test"));  
        consumer.seek(new TopicPartition("test", 1), 1);
        while (true) {  
            ConsumerRecords<String, String> records = consumer.poll(2000);
            System.out.println("-------------"+new Date());
            for (ConsumerRecord<String, String> record : records)  
                System.out.printf("offset = %d, key = %s, value = %s\n", record.offset(), record.key(), record.value());  
        }
        
        /*
        //KafkaWriter
        KafkaProducer<String, String> productor = new KafkaProducer<>(props);
        productor.send(new ProducerRecord<String, String>("test", "aaa", "xiaoxiaoxiao2018"));
        */
    }  
}

名詞解釋：

bootstrap.servers：Kafka集羣鏈接串，能夠由多個host:port組成【your.host.name:9092】