轉載-zookeeper在kafka中的做用

1)Broker註冊

　　Broker在zookeeper中保存爲一個臨時節點，節點的路徑是/brokers/ids/[brokerid],每一個節點會保存對應broker的IP以及端口等信息.

　　2)Topic註冊

　　在kafka中,一個topic會被分紅多個區並被分到多個broker上，分區的信息以及broker的分佈狀況都保存在zookeeper中，根節點路徑爲/brokers/topics,每一個topic都會在topics下創建獨立的子節點，每一個topic節點下都會包含分區以及broker的對應信息，例以下圖中的狀態

　　3)生產者負載均衡

　　當Broker啓動時，會註冊該Broker的信息，以及可訂閱的topic信息。生產者經過註冊在Broker以及Topic上的watcher動態的感知Broker以及Topic的分區狀況，從而將Topic的分區動態的分配到broker上.

　　4)消費者

　　kafka有消費者分組的概念，每一個分組中能夠包含多個消費者，每條消息只會發給分組中的一個消費者，且每一個分組之間是相互獨立互不影響的。

　　5)消費者與分區的對應關係

　　對於每一個消費者分組，kafka都會爲其分配一個全局惟一的Group ID,分組內的全部消費者會共享該ID,kafka還會爲每一個消費者分配一個consumer ID,一般採用hostname:uuid的形式。在kafka的設計中規定，對於topic的每一個分區，最多隻能被一個消費者進行消費，也就是消費者與分區的關係是一對多的關係。消費者與分區的關係也被存儲在zookeeper中節點的路勁爲 /consumers/[group_id]/owners/[topic]/[broker_id-partition_id],該節點的內容就是消費者的Consumer ID

　　6)消費者負載均衡

　　消費者服務啓動時，會建立一個屬於消費者節點的臨時節點，節點的路徑爲 /consumers/[group_id]/ids/[consumer_id],該節點的內容是該消費者訂閱的Topic信息。每一個消費者會對/consumers/[group_id]/ids節點註冊Watcher監聽器，一旦消費者的數量增長或減小就會觸發消費者的負載均衡。消費者還會對/brokers/ids/[brokerid]節點進行監聽，若是發現服務器的Broker服務器列表發生變化，也會進行消費者的負載均衡

　　7)消費者的offset

　　在kafka的消費者API分爲兩種(1)High Level Api：由zookeeper維護消費者的offset (2) Low Level API,本身的代碼實現對offset的維護。因爲本身維護offset每每比較複雜，因此多數狀況下都是使用High Level的APIoffset在zookeeper中的節點路徑爲/consumers/[group_id]/offsets/[topic]/[broker_id-part_id],該節點的值就是對應的offset

　　2、kakfa在zookeeper中的結構

　　

　kafka.javaapi.consumer.ZookeeperConsumerConnector.java

　　/**

　　* This class handles the consumers interaction with zookeeper

　　*

　　* Directories:

　　* 1. Consumer id registry:

　　* /consumers/[group_id]/ids[consumer_id] -> topic1,...topicN

　　* A consumer has a unique consumer id within a consumer group. A consumer registers its id as an ephemeral znode

　　* and puts all topics that it subscribes to as the value of the znode. The znode is deleted when the client is gone.

　　* A consumer subscribes to event changes of the consumer id registry within its group.

　　*

　　* The consumer id is picked up from configuration, instead of the sequential id assigned by ZK. Generated sequential

　　* ids are hard to recover during temporary connection loss to ZK, since it's difficult for the client to figure out

　　* whether the creation of a sequential znode has succeeded or not. More details can be found at

　　*

　　* 2. Broker node registry:

　　* /brokers/[0...N] --> { "host" : "host:port",

　　* "topics" : {"topic1": ["partition1" ... "partitionN"], ...,

　　* "topicN": ["partition1" ... "partitionN"] } }

　　* This is a list of all present broker brokers. A unique logical node id is configured on each broker node. A broker

　　* node registers itself on start-up and creates a znode with the logical node id under /brokers. The value of the znode

　　* is a JSON String that contains (1) the host name and the port the broker is listening to, (2) a list of topics that

　　* the broker serves, (3) a list of logical partitions assigned to each topic on the broker.

　　* A consumer subscribes to event changes of the broker node registry.

　　*

　　* 3. Partition owner registry:

　　* /consumers/[group_id]/owner/[topic]/[broker_id-partition_id] --> consumer_node_id

　　* This stores the mapping before broker partitions and consumers. Each partition is owned by a unique consumer

　　* within a consumer group. The mapping is reestablished after each rebalancing.

　　*

　　* 4. Consumer offset tracking:

　　* /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] --> offset_counter_value

　　* Each consumer tracks the offset of the latest message consumed for each partition.

　　*

　　*/

歡迎工做一到五年的Java工程師朋友們加入Java架構開發：855801563

羣內提供免費的Java架構學習資料（裏面有高可用、高併發、高性能及分佈式、Jvm性能調優、Spring源碼，MyBatis，Netty,Redis,Kafka,Mysql,Zookeeper,Tomcat,Docker,Dubbo,Nginx等多個知識點的架構資料）合理利用本身每一分每一秒的時間來學習提高本身，不要再用"沒有時間「來掩飾本身思想上的懶惰！趁年輕，使勁拼，給將來的本身一個交代