kafka總結

一、啓動kafka

sh kafka-server-start.sh ../config/server.properties &

注意：若是報Exception in thread "main" java.net.UnknownHostException錯誤，是由於kafka獲取ip信息的時候使用的是java.net.InetAddress.getCanonicalHostName()，在linux下該方法獲取的是主機名。

 

說明：broker註冊到zookeeper的IP是用的server.properties配置的信息，查看server.properties的註釋:（配置了，使用advertised.host.name的信息，否則使用host.name的信息，若是Host.name也是空，則使用java.net.InetAddress.getCanonicalHostName()）　

# Hostname the broker will bind to. If not set, the server will bind to all interfaces
host.name=xxx.xxx.xxx.xxx

# Hostname the broker will advertise to producers and consumers. If not set, it uses the
# value for "host.name" if configured.  Otherwise, it will use the value returned from
# java.net.InetAddress.getCanonicalHostName().
advertised.host.name=xxx.xxx.xxx.xxx

 

解決辦法：

　　1>、host裏配置：localhost 主機名。

　　2>、配置kafka 的server.properties中，host.name=本機IP（通常是內網IP）　　

　　若是使用方式1，在本機啓動producer是OK的，可是在遠程就不行了，由於返回給client的是zookeeper裏註冊的節點信息localhost，client經過localhost確定是獲取不到broker地址的。因此此處直接配置內網IP便可解決

另外，在linux中，consumer啓動的時候，註冊到zookeeper的IP信息應該也是經過java.net.InetAddress.getCanonicalHostName()獲取的，因此，在linux上啓動consumer須要配置host

 

二、zookeeper上查看節點信息 

sh zkCli.sh -server 127.0.0.1:2181
ls /brokers/ids

 

三、建立topic（2個分區，3個副本）

./kafka-topics.sh --create --topic topic1 --replication-factor 3 --partitions 2 --zookeeper localhost:2181

四、查看topic list

 ./kafka-topics.sh --list --zookeeper localhost:2181  

五、zookeeper上查看topic list信息

　 ls /brokers/topics

六、zookeeper上查看節點具體信息

[zk: 127.0.0.1:2181(CONNECTED) 9] get /brokers/topics/topic1
{"version":1,"partitions":{"1":[1,2,3],"0":[3,1,2]}}
cZxid = 0x200000179
ctime = Wed Mar 30 23:24:25 EDT 2016
mZxid = 0x200000179
mtime = Wed Mar 30 23:24:25 EDT 2016
pZxid = 0x20000017d
cversion = 1
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 52
numChildren = 1

七、kafka目錄中查看節點信息

cd /tmp/kafka-logs-1 #kafka server.config中配置的目錄，topic1-0 和topic1-1分別記錄topic1的2個分區的日誌數據。查看另外2個kafka日誌目錄，也會存在這樣的2個文件夾，對應的是topic建立的3個副本
drwxr-xr-x 2 root root 4096 Mar 30 23:24 topic1-0
drwxr-xr-x 2 root root 4096 Mar 30 23:24 topic1-1

八、建立消費者

 ./kafka-console-consumer.sh --zookeeper localhost:2181 --topic topic1 --from-beginning

九、zookeeper中查看對應的節點信息

#自動建立group_id
[zk: 127.0.0.1:2181(CONNECTED) 18] ls /consumers
[console-consumer-75348]
#對應三個節點
[zk: 127.0.0.1:2181(CONNECTED) 19] ls /consumers/console-consumer-75348
[offsets, owners, ids]
#查看剛剛建立的消費者ID：consumerId，用來標記一個消費者
[zk: 127.0.0.1:2181(CONNECTED) 15] ls /consumers/console-consumer-75348/ids
[console-consumer-75348_e2ef967443a3-1459395856932-a0028f17]
[zk: 127.0.0.1:2181(CONNECTED) 17] get /consumers/console-consumer-75348/ids/console-consumer-75348_e2ef967443a3-1459395856932-a0028f17
{"version":1,"subscription":{"topic1":1},"pattern":"white_list","timestamp":"1459395857222"}
cZxid = 0x200000190
ctime = Wed Mar 30 23:44:17 EDT 2016
mZxid = 0x200000190
mtime = Wed Mar 30 23:44:17 EDT 2016
pZxid = 0x200000190
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x153413bc26e002d
dataLength = 92
numChildren = 0
[zk: 127.0.0.1:2181(CONNECTED) 20] ls /consumers/console-consumer-75348/owners
[topic1]
[zk: 127.0.0.1:2181(CONNECTED) 21] ls /consumers/console-consumer-75348/owners/topic1
[1, 0]
#topic1的第一個分區正被xxx消費
[zk: 127.0.0.1:2181(CONNECTED) 23] get /consumers/console-consumer-75348/owners/topic1/1
console-consumer-75348_e2ef967443a3-1459395856932-a0028f17-0
cZxid = 0x200000195
ctime = Wed Mar 30 23:44:18 EDT 2016
mZxid = 0x200000195
mtime = Wed Mar 30 23:44:18 EDT 2016
pZxid = 0x200000195
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x153413bc26e002d
dataLength = 60
numChildren = 0
[zk: 127.0.0.1:2181(CONNECTED) 24] get /consumers/console-consumer-75348/owners/topic1/0
console-consumer-75348_e2ef967443a3-1459395856932-a0028f17-0
cZxid = 0x200000196
ctime = Wed Mar 30 23:44:18 EDT 2016
mZxid = 0x200000196
mtime = Wed Mar 30 23:44:18 EDT 2016
pZxid = 0x200000196
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x153413bc26e002d
dataLength = 60
numChildren = 0
[zk: 127.0.0.1:2181(CONNECTED) 26] ls /consumers/console-consumer-75348/offsets 
[topic1]
[zk: 127.0.0.1:2181(CONNECTED) 27] ls /consumers/console-consumer-75348/offsets/topic1
[1, 0]
#topic1的1分區被消費的offset位置。和consumer_id無關，和group_id關聯
#就是說，每一個分區的信息能夠被不一樣的group_id消費，可是同一時間只能由一個group的其中一個consumer消費
[zk: 127.0.0.1:2181(CONNECTED) 29] get /consumers/console-consumer-75348/offsets/topic1/1
0
cZxid = 0x20000019e
ctime = Wed Mar 30 23:45:17 EDT 2016
mZxid = 0x20000019e
mtime = Wed Mar 30 23:45:17 EDT 2016
pZxid = 0x20000019e
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 1
numChildren = 0
[zk: 127.0.0.1:2181(CONNECTED) 30] get /consumers/console-consumer-75348/offsets/topic1/0
0
cZxid = 0x20000019b
ctime = Wed Mar 30 23:45:17 EDT 2016
mZxid = 0x20000019b
mtime = Wed Mar 30 23:45:17 EDT 2016
pZxid = 0x20000019b
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 1
numChildren = 0

十、建立生產者

sh kafka-console-producer.sh --broker-list localhost:9092  --topic topic1

十一、隨便輸入消息，能夠在消費者控制檯看到對應的信息。

十二、zk上查看對應的信息

[zk: localhost:2181(CONNECTED) 42] get /consumers/console-consumer-19167/offsets/topic1/0
3
cZxid = 0x2000001fc
ctime = Thu Mar 31 01:50:52 EDT 2016
mZxid = 0x200000204
mtime = Thu Mar 31 01:52:51 EDT 2016
pZxid = 0x2000001fc
cversion = 0
dataVersion = 1
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 1
numChildren = 0

能夠看到offset有變化

好了，初步進行了總結，仍是多動手纔能有更深入的體會