kafka中 FailedToSendMessageException解決

這兩天在學kafka,把官網上的demo部署到本身的虛擬機上運行,正常;

而後部署到公司的研發線主機上,發現producer一直沒法發送消息;

部分錯誤日誌以下:

[2014-11-13 09:58:09,660] WARN Error while fetching metadata [{TopicMetadata for topic mor ->
No partition metadata for topic mor due to kafka.common.LeaderNotAvailableException}] for topic [mor]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2014-11-13 09:58:09,660] ERROR Failed to send requests for topics mor with correlation ids in [17,24] (kafka.producer.async.DefaultEventHandler)
[2014-11-13 09:58:09,660] ERROR Error in handling batch of 17 events (kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
        at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
        at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
        at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
        at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
        at scala.collection.immutable.Stream.foreach(Stream.scala:526)
        at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
        at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)

操做步驟徹底同樣,只是在研發線上修改了zookeeper的監聽端口,輸入命令時也是按照對應的端口輸入的,因此不是參數有問題致使的;

對比了研發線和虛擬機上的配置,發現除了這個端口以外,沒有區別;

鼓搗了一天,沒找到問題出在哪裏;

次日來上班,把server.properties文件裏

#host.name=localhost

的註釋去掉,而後運行,發現問題解決了;

(可是在虛擬機裏明明沒有修改這個東西也是能夠正常運行的;

懷疑是發行版不一樣,一些內部的配置有區別致使的;

虛擬機:

Distributor ID:    Ubuntu
Description:    Ubuntu 14.04.1 LTS
Release:    14.04
Codename:    trusty

研發線:

LSB Version:    :
Distributor ID: RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 5.4 (Tikanga)
Release:        5.4
Codename:       Tikanga

)

---

在解決問題的過程當中,還發現其餘的問題,可是目前沒發現影響運行;

就是zookeeper和broker跑起來以後,建立producer,建立話題,建立Consumer,的時候,zookeeper都會報異常,部分記錄以下:

[2014-11-13 09:12:13,486] INFO Got user-level KeeperException when processing sessionid:0x149a6b3e36c0001 type:setData cxid:0x3 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/config/topics/morning Error:KeeperErrorCode = NoNode for /config/topics/morning (org.apache.zookeeper.server.PrepRequestProcessor)

[2014-11-13 09:12:13,506] INFO Got user-level KeeperException when processing sessionid:0x149a6b3e36c0001 type:create cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/config/topics Error:KeeperErrorCode = NodeExists for /config/topics (org.apache.zookeeper.server.PrepRequestProcessor)

[2014-11-13 09:12:13,535] INFO Processed session termination for sessionid: 0x149a6b3e36c0001 (org.apache.zookeeper.server.PrepRequestProcessor)

[2014-11-13 09:19:50,958] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf50000 type:create cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/brokers Error:KeeperErrorCode = NoNode for /brokers (org.apache.zookeeper.server.PrepRequestProcessor)
[2014-11-13 09:19:50,982] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf50000 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/config Error:KeeperErrorCode = NoNode for /config (org.apache.zookeeper.server.PrepRequestProcessor)
[2014-11-13 09:19:50,998] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf50000 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/admin Error:KeeperErrorCode = NoNode for /admin (org.apache.zookeeper.server.PrepRequestProcessor)
[2014-11-13 09:19:51,295] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf50000 type:setData cxid:0x19 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/controller_epoch Error:KeeperErrorCode = NoNode for /controller_epoch (org.apache.zookeeper.server.PrepRequestProcessor)
[2014-11-13 09:19:51,374] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf50000 type:delete cxid:0x27 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for /admin/preferred_replica_election (org.apache.zookeeper.server.PrepRequestProcessor)

[2014-11-13 10:31:50,651] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf5001a type:setData cxid:0x19 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/consumers/test-consumer-group/offsets/mor/0 Error:KeeperErrorCode = NoNode for /consumers/test-consumer-group/offsets/mor/0 (org.apache.zookeeper.server.PrepRequestProcessor)
[2014-11-13 10:31:50,661] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf5001a type:create cxid:0x1a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/consumers/test-consumer-group/offsets Error:KeeperErrorCode = NoNode for /consumers/test-consumer-group/offsets (org.apache.zookeeper.server.PrepRequestProcessor)

一開始producer沒法發送消息,覺得是和這些異常有關,可是在虛擬機裏運行的時候,一樣出現這些異常,可是也沒有影響producer發送消息;

在網上查了半天,有人說是不正確關閉zookeeper和server的緣由,有人說是沒有刪除/tmp下的zookeeper日誌和kafka日誌的緣由,

總之,我試了上面提到的方法,仍是報這些異常;

若是有哪位仁兄知道這些異常是什麼緣由引發的,請告訴我,十分感謝;

---

另外,從本機運行java代碼的Consumer鏈接研發線的時候,鏈接迅速被關閉了,收不到producer發來的消息;

緣由是配置的超時時間過短,zookeeper沒有讀完Consumer的數據,鏈接就被Consumer斷開了,部分日誌以下:

[2014-11-13 10:28:47,989] INFO Accepted socket connection from /192.168.50.33:2676 (org.apache.zookeeper.server.NIOServerCnxn)
[2014-11-13 10:28:47,989] WARN EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)

‍解決辦法,就是把配置裏的超時時間配置的長點兒,以下:

props.put("zookeeper.session.timeout.ms", "400000");

---

對了,還有個問題,kafka官網上關於java版producer部分的代碼,有個地方寫的不是很明確,以下:

props.put("metadata.broker.list", "broker1:9092,broker2:9092");

 

broker1和broker2表明broker對應的hostname,而不是broker的id.正確示例以下:

 props.put("metadata.broker.list", "localhost:9092,localhost:9093");