10-kafka集羣擴容

第一步 配置新得broker

1. 將現有的集羣上任一個服務器上的kafka目錄拷貝到新的服務器上
2. 修改config/server.properties中的broker.id、log.dirs、listeners
3. 建立logs.dirs指定的目錄，並設定讀寫權限(chomd -R 777 XXX)

broker.id=3
log.dirs=kafka-logs
listeners=PLAINTEXT://172.16.49.174:9092

第二步 啓動新的broker

bin/kafka-server-start.sh config/server.properties  &

第三步 遷移指定topic的數據到新的broker

雖然通過上面兩個步驟後已經完成了集羣的擴容；可是集羣上原有的topic的數據不會自動遷移到新的broker上。能夠在新的broker所在的服務器上經過 ls /home/lxh/kafka_2.11-0.10.0.0/kafka-logs 查看到並無一原有的topic名稱的文件目錄（由於建立topic後會在config/server.properties中的配置的log.dirs 目錄中生產以topic名稱+分區編號的文件目錄）；那麼就須要手動的區遷移數據

(一)、生成遷移分配規則json文件

建立編輯要遷移的topic的 json文件

vi topic-to-move.json

好比要將topic名稱爲test和paritioer_test的數據從新平衡到集羣中，就能夠新增如下內容

{"topics": [{"topic": "test"},
            {"topic": "paritioer_test"}],
 "version":1
}

生成遷移分配規則json文件

bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file topic-to-move.json --broker-list "0,1,2" --generate

獲得的結果爲

Current partition replica assignment 

{"version":1,"partitions":[{"topic":"test","partition":4,"replicas":[0,1]},{"topic":"test","partition":1,"replicas":[1,0]},{"topic":"paritioer_test","partition":0,"replicas":[0]},{"topic":"test","partition":2,"replicas":[0,1]},{"topic":"test","partition":0,"replicas":[0,1]},{"topic":"test","partition":3,"replicas":[1,0]}]}
Proposed partition reassignment configuration

{"version":1,"partitions":[{"topic":"test","partition":4,"replicas":[1,0]},{"topic":"test","partition":1,"replicas":[1,2]},{"topic":"test","partition":2,"replicas":[2,0]},{"topic":"paritioer_test","partition":0,"replicas":[0]},{"topic":"test","partition":0,"replicas":[0,1]},{"topic":"test","partition":3,"replicas":[0,2]}]}

其中的Current partition replica assignment指的是遷移前的partition replica；Proposed partition reassignment configuration 指的就是遷移分配規則json。須要將該json文件保存到json文件中(如expand-cluster-reassignment.json)

(二)、執行遷移分配

bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file expand-cluster-reassignment.json --execute

注意：在遷移過程當中不能人爲的結束或中止kafka服務，否則會有數據不一致的問題

(三)、驗證分配

在執行的過程當中，能夠新開一個終端執行如下命令來查看執行是否正確完成

bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file expand-cluster-reassignment.json --verify

輸出

Status of partition reassignment:
Reassignment of partition [test,4] completed successfully
Reassignment of partition [test,0] completed successfully
Reassignment of partition [paritioer_test,0] completed successfully
Reassignment of partition [test,3] completed successfully
Reassignment of partition [test,2] completed successfully
Reassignment of partition [test,1] completed successfully

在遷移完成過程後，可使用如下命令看下topic的每一個partitions的分佈狀況

bin/kafka-topics.sh --zookeeper 172.16.49.173:2181 --describe --topic test

Topic:test  PartitionCount:5    ReplicationFactor:2 Configs:
    Topic: test Partition: 0    Leader: 0   Replicas: 0,1   Isr: 0,1
    Topic: test Partition: 1    Leader: 1   Replicas: 1,2   Isr: 1,2
    Topic: test Partition: 2    Leader: 2   Replicas: 2,0   Isr: 0,2
    Topic: test Partition: 3    Leader: 0   Replicas: 0,2   Isr: 0,2
    Topic: test Partition: 4    Leader: 0   Replicas: 1,0   Isr: 0,1

能夠看到名爲test的topic的有的數據以及存在於編號爲2的新broker上了