bigdata-02-kafka

下載：http://kafka.apache.org/downloads
安裝：tar -zxvf kafka_2.11-0.9.0.0.tgz -C ~/app

配置環境變量
    vi ~/.bash_profile

    export KAFKA_HOME=/Users/liujinjie/app/kafka_2.11-0.9.0.0
    export PATH=$KAFKA_HOME/bin:$PATH

使環境變量生效
    source ~/.bash_profile

~~~~~~~~~~~~~~~~~~~~~單節點單broker的部署(一個機器上配置一個broker)~~~~~~~~~~~~~~~~~~~~~~

修改配置文件
$KAFKA_HOME/config/server.properties
    broker.id=0   # 一個kafka一個實例
    listeners=PLAINTEXT://:9092
    host.name=localhost
    log.dirs=/Users/liujinjie/app/tmp/kafka-logs
    zookeeper.connect=localhost:2181

官方啓動文檔
http://kafka.apache.org/quickstart

kafka-server-start.sh
USAGE: /Users/liujinjie/app/kafka_2.11-0.9.0.0/bin/kafka-server-start.sh [-daemon] server.properties [--override property=value]*

一、啓動
kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties

二、建立topic
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic hello_topic

三、發送消息
kafka-console-producer.sh --broker-list localhost:9092 --topic test-topic-180216-01

四、消費消息

（1）kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic-180216-01

（2）kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

--from-beginning 每次都從頭消費

# 老版本的寫法
kafka-console-consumer.sh --zookeeper localhost:2181 --topic test-topic-180216-01

五、查看描述
kafka-topics.sh --describe --zookeeper localhost:2181

返回
Topic:hello_topic    PartitionCount:1    ReplicationFactor:1    Configs:
    Topic: hello_topic    Partition: 0    Leader: 0    Replicas: 0    Isr: 0

六、查看全部topic

kafka-topics.sh --list --zookeeper localhost:2181

七、根據topic名稱查看內容

kafka-topics.sh --describe --zookeeper localhost:2181 --topic hello_topic

~~~~~~~~~~~~~~~~~~~~~~單節點多broker部署(一個機器上部署多個broker)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

cd $KAFKA_HOME/conf
cp server.proerties server-1.properties
cp server.proerties server-2.properties
cp server.proerties server-3.properties

vi server-1.properties
    broker.id=1
    listeners=PLAINTEXT://:9093
    log.dirs=/Users/liujinjie/app/tmp/kafka-logs-1

vi server-2.properties
    broker.id=2
    listeners=PLAINTEXT://:9094
    log.dirs=/Users/liujinjie/app/tmp/kafka-logs-2

vi server-1.properties
    broker.id=3
    listeners=PLAINTEXT://:9095
    log.dirs=/Users/liujinjie/app/tmp/kafka-logs-3

啓動
kafka-server-start.sh -daemon $KAFKA_HOME/config/server-1.properties &

kafka-server-start.sh -daemon $KAFKA_HOME/config/server-2.properties &

kafka-server-start.sh -daemon $KAFKA_HOME/config/server-3.properties &

建立topic
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic

看topic詳情
    kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

返回
Topic:my-replicated-topic    PartitionCount:1    ReplicationFactor:3    Configs:
    Topic: my-replicated-topic    Partition: 0    Leader: 1    Replicas: 1,3,2    Isr: 1,3,2

PartitionCount:1         分區數量是1
ReplicationFactor:3     副本系數是3
Leader: 1                 broker-id=1是leader
Replicas:1,3,2           3個副本
Isr: 1,3,2              活着的3個

啓動生產者
kafka-console-producer.sh --broker-list localhost:9093,localhost:9094,localhost:9095 --topic my-replicated-topic

消費消息
kafka-console-consumer.sh --zookeeper localhost:2181 --topic my-replicated-topic

Now let's test out fault-tolerance. Broker 1 was acting as the leader so let's kill it:

> ps aux | grep server-1.properties 7564 ttys002    0:15.91 /System/Library/Frameworks/JavaVM.framework/Versions/1.8/Home/bin/java... > kill -9 7564