kafka 經常使用命令

Kafka

解壓，進入kafka目錄下的bin目錄

啓動zk

nohup ./zookeeper-server-start.sh ../config/zookeeper.properties &

nohup ./bin/zookeeper-server-start.sh ./config/zookeeper.properties &

啓動kafka

nohup ./kafka-server-start.sh ../config/server.properties &

nohup ./bin/kafka-server-start.sh ./config/server.properties &

建立topic

./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

或

./kafka-topics.sh --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test2 --create

列出topic

./kafka-topics.sh --list --zookeeper localhost:2181

或

./kafka-topic --zookeeper localhost:2181 --list

啓動生產者併發送消息

./kafka-console-producer.sh --broker-list localhost:9092 --topic zyl

輸入消息

nihao

china

另外開個終端，啓動消費者接受消息

./kafka-console-consumer.sh --zookeeper localhost:2181 --topic zyl --from-beginning

能夠接受到

nihao

china

另外開個終端，能夠看到有4個進程（分別爲zk、kafka、控制檯消費者、控制檯生產者）

[root@aa bin]# jps | grep -v Jps

9802 QuorumPeerMain

10392 Kafka

10956 ConsoleConsumer

10887 ConsoleProducer

使用集羣3個節點（多個節點也同理,加上以前的，我一共弄了4個節點）

拷貝並修改配置文件

cp server.properties server1.properties

cp server.properties server2.properties

cp server.properties server3.properties

nohup ./kafka-server-start.sh ../config/server1.properties &

nohup ./kafka-server-start.sh ../config/server2.properties &

nohup ./kafka-server-start.sh ../config/server3.properties &

[root@aa bin]# ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic（注意，指定的是3）

Created topic "my-replicated-topic".

[root@aa bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:

Topic: my-replicated-topic Partition: 0 Leader: 2 Replicas: 2,3,0 Isr: 2,3,0

[root@aa bin]# ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 4 --partitions 1 --topic my-replicated-topic

Error while executing topic command Topic "my-replicated-topic" already exists.

kafka.common.TopicExistsException: Topic "my-replicated-topic" already exists.

at kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:171)

at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:156)

at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:86)

at kafka.admin.TopicCommand$.main(TopicCommand.scala:50)

at kafka.admin.TopicCommand.main(TopicCommand.scala)

[root@aa bin]# ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 4 --partitions 1 --topic my-replicated-topic2

Created topic "my-replicated-topic2".

[root@aa bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic2

Topic:my-replicated-topic2 PartitionCount:1 ReplicationFactor:4 Configs:

Topic: my-replicated-topic2 Partition: 0 Leader: 1 Replicas: 1,2,3,0 Isr: 1,2,3,0

[root@aa bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic zyl

Topic:zyl PartitionCount:1 ReplicationFactor:1 Configs:

Topic: zyl Partition: 0 Leader: 0 Replicas: 0 Isr: 0

啓動生產者併發送消息

./kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic

另外開個終端，啓動消費者接受消息

./kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic my-replicated-topic

另外開個終端，能夠看到有9個進程（分別爲1個zk、4個kafka、2個控制檯消費者、2個控制檯生產者）

[root@aa ~]# jps | grep -v Jps

9802 QuorumPeerMain

10392 Kafka

11547 Kafka

12007 ConsoleProducer

10956 ConsoleConsumer

10887 ConsoleProducer

11469 Kafka

12054 ConsoleConsumer

11710 Kafka

查看單獨的進程：

ps -ef | grep server.properties | grep -v grep

ps -ef | grep "server1.properties" | grep -v grep

測試leader

殺掉進程，而後--describe查看

[root@aaa bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:

Topic: my-replicated-topic Partition: 0 Leader: 2 Replicas: 2,3,0 Isr: 2,3,0

能夠看到leader是2.

 

 [root@aa bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic2

Topic:my-replicated-topic2 PartitionCount:1 ReplicationFactor:4 Configs:

Topic: my-replicated-topic2 Partition: 0 Leader: 1 Replicas: 1,2,3,0 Isr: 1,2,3,0

能夠看到leader是2.

[root@aa bin]# ps -ef | grep server1.properties | grep -v grep

殺掉某個kafka進程

kill -9 11469

 ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic2

Topic:my-replicated-topic2 PartitionCount:1 ReplicationFactor:4 Configs:

Topic: my-replicated-topic2 Partition: 0 Leader: 2 Replicas: 1,2,3,0 Isr: 2,3,0

[root@aa bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:

Topic: my-replicated-topic Partition: 0 Leader: 2 Replicas: 2,3,0 Isr: 2,3,0

能夠看到my-replicated-topic2的leader變成了2。

問題：

1.zookeeper-shell.sh如何使用

2.能不能經過kafka-server-stop.sh中止某個kafka呢

Command must include exactly one action: --list, --describe, --create or --alter

Option                                  Description

------                                  -----------

--alter                                 Alter the configuration for the topic.

--config <name=value>                   A topic configuration override for the

                                          topic being created or altered.

--create                                Create a new topic.

--deleteConfig <name>                   A topic configuration override to be

                                          removed for an existing topic

--describe                              List details for the given topics.

--help                                  Print usage information.

--list                                  List all available topics.

--partitions <Integer: # of partitions> The number of partitions for the topic

                                          being created or altered (WARNING:

                                          If partitions are increased for a

                                          topic that has a key, the partition

                                          logic or ordering of the messages

                                          will be affected

--replica-assignment                    A list of manual partition-to-broker

  <broker_id_for_part1_replica1 :         assignments for the topic being

  broker_id_for_part1_replica2 ,          created or altered.

  broker_id_for_part2_replica1 :

  broker_id_for_part2_replica2 , ...>

--replication-factor <Integer:          The replication factor for each

  replication factor>                     partition in the topic being created.

--topic <topic>                         The topic to be create, alter or

                                          describe. Can also accept a regular

                                          expression except for --create option

--topics-with-overrides                 if set when describing topics, only

                                          show topics that have overridden

                                          configs

--unavailable-partitions                if set when describing topics, only

                                          show partitions whose leader is not

                                          available

--under-replicated-partitions           if set when describing topics, only

                                          show under replicated partitions

--zookeeper <urls>                      REQUIRED: The connection string for

                                          the zookeeper connection in the form

                                          host:port. Multiple URLS can be

                                          given to allow fail-over.