如何使用kafka增長topic的備份數量，讓業務更上一層樓

本文由雲+社區發表

1、困難點

創建topic的時候，能夠經過指定參數 --replication-factor 設置備份數量。可是，一旦完成創建topic，則沒法經過kafka-topic.sh 或者 命令修改replica數量。

2、解決辦法

​ 實際上，咱們能夠考慮一種 「另類」 的辦法：能夠利用 kafka-reassign-partitions.sh 命令對全部分區進行從新分佈，在作分區從新分佈的時候，經過增長每一個分區的replica備份數量來達到目的。

​ 本文將介紹如何利用 kafka-reassign-partitions.sh 命令增長topic的備份數量。

注意：如下命令使用到的topic名稱、zookeeper的ip和port，須要讀者替換成爲實際集羣的參數。

（假設kafka集羣有4個broker，id分別爲：1001,1002,1003,1004）

2.一、獲取當前topic的全部分區分佈在broker的狀況

[root@tbds bin]# ./kafka-topics.sh --zookeeper 172.16.32.13:2181 --topic ranger_audits --describe
Topic:ranger_audits     PartitionCount:10       ReplicationFactor:1     Configs:
        Topic: ranger_audits    Partition: 0    Leader: 1001    Replicas: 1001  Isr: 1001
        Topic: ranger_audits    Partition: 1    Leader: 1002    Replicas: 1002  Isr: 1002
        Topic: ranger_audits    Partition: 2    Leader: 1001    Replicas: 1001  Isr: 1001
        Topic: ranger_audits    Partition: 3    Leader: 1002    Replicas: 1002  Isr: 1002
        Topic: ranger_audits    Partition: 4    Leader: 1001    Replicas: 1001  Isr: 1001
        Topic: ranger_audits    Partition: 5    Leader: 1002    Replicas: 1002  Isr: 1002
        Topic: ranger_audits    Partition: 6    Leader: 1001    Replicas: 1001  Isr: 1001
        Topic: ranger_audits    Partition: 7    Leader: 1002    Replicas: 1002  Isr: 1002
        Topic: ranger_audits    Partition: 8    Leader: 1001    Replicas: 1001  Isr: 1001
        Topic: ranger_audits    Partition: 9    Leader: 1002    Replicas: 1002  Isr: 1002

能夠看出，ranger_audits 這個topic有10個分區，每一個分區只有一個feplica備份，分佈在1001和1002兩臺broker上面。

下面咱們須要將ranger_audits 的每一個分區數據都增長到2個replica備份，且分佈到4個broker上面。

2.二、建立增長replica備份數量的配置文件

（注意：儘可能保持topic的原有每一個分區的主備份不變化。所以，配置文件的每一個分區的第一個broker保持不變。）

[root@tbds bin]# vim ../config/increase-replication-factor.json
{"version":1,
"partitions":[
{"topic":"ranger_audits","partition":0,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":1,"replicas":[1002,1004]},
{"topic":"ranger_audits","partition":2,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":3,"replicas":[1002,1004]},
{"topic":"ranger_audits","partition":4,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":5,"replicas":[1002,1004]},
{"topic":"ranger_audits","partition":6,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":7,"replicas":[1002,1004]},
{"topic":"ranger_audits","partition":8,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":9,"replicas":[1002,1004]}
]}

上面的配置文件說明，咱們將topic的每一個分區都增長了一個replica，且保持每一個分區原有的主備份所在broker不變化，將每一個分區新增的replica備份數據放到到1003和1004兩個broker上面。

2.三、開始執行增長分區

[root@tbds bin]# ./kafka-reassign-partitions.sh -zookeeper 172.16.32.13:2181 --reassignment-json-file ../config/increase-replication-factor.json --execute
Current partition replica assignment
{"version":1,"partitions":[{"topic":"ranger_audits","partition":3,"replicas":[1002]},{"topic":"ranger_audits","partition":9,"replicas":[1002]},{"topic":"ranger_audits","partition":8,"replicas":[1001]},{"topic":"ranger_audits","partition":1,"replicas":[1002]},{"topic":"ranger_audits","partition":4,"replicas":[1001]},{"topic":"ranger_audits","partition":2,"replicas":[1001]},{"topic":"ranger_audits","partition":5,"replicas":[1002]},{"topic":"ranger_audits","partition":0,"replicas":[1001]},{"topic":"ranger_audits","partition":6,"replicas":[1001]},{"topic":"ranger_audits","partition":7,"replicas":[1002]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions 
{"version":1,"partitions":[{"topic":"ranger_audits","partition":0,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":8,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":5,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":2,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":9,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":1,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":3,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":4,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":7,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":6,"replicas":[1001,1003]}]}

2.四、查看執行進度

[root@tbds bin]# ./kafka-reassign-partitions.sh -zookeeper 172.16.32.13:2181 --reassignment-json-file ../config/increase-replication-factor.json --verify
Status of partition reassignment:
Reassignment of partition [ranger_audits,0] completed successfully
Reassignment of partition [ranger_audits,8] completed successfully
Reassignment of partition [ranger_audits,5] completed successfully
Reassignment of partition [ranger_audits,2] completed successfully
Reassignment of partition [ranger_audits,9] completed successfully
Reassignment of partition [ranger_audits,1] completed successfully
Reassignment of partition [ranger_audits,3] completed successfully
Reassignment of partition [ranger_audits,4] completed successfully
Reassignment of partition [ranger_audits,7] completed successfully
Reassignment of partition [ranger_audits,6] completed successfully

上面顯示增長分區操做成功

2.五、再次查看topic的狀況

[root@tbds bin]# ./kafka-topics.sh --zookeeper 172.16.32.13:2181 --topic ranger_audits --describe
Topic:ranger_audits     PartitionCount:10       ReplicationFactor:2     Configs:
        Topic: ranger_audits    Partition: 0    Leader: 1001    Replicas: 1001,1003     Isr: 1001,1003
        Topic: ranger_audits    Partition: 1    Leader: 1002    Replicas: 1002,1004     Isr: 1002,1004
        Topic: ranger_audits    Partition: 2    Leader: 1001    Replicas: 1001,1003     Isr: 1001,1003
        Topic: ranger_audits    Partition: 3    Leader: 1002    Replicas: 1002,1004     Isr: 1002,1004
        Topic: ranger_audits    Partition: 4    Leader: 1001    Replicas: 1001,1003     Isr: 1001,1003
        Topic: ranger_audits    Partition: 5    Leader: 1002    Replicas: 1002,1004     Isr: 1002,1004
        Topic: ranger_audits    Partition: 6    Leader: 1001    Replicas: 1001,1003     Isr: 1001,1003
        Topic: ranger_audits    Partition: 7    Leader: 1002    Replicas: 1002,1004     Isr: 1002,1004
        Topic: ranger_audits    Partition: 8    Leader: 1001    Replicas: 1001,1003     Isr: 1001,1003
        Topic: ranger_audits    Partition: 9    Leader: 1002    Replicas: 1002,1004     Isr: 1002,1004

從上面能夠看出，備份數量增長成功

3、進一步思考

​ 利用上述介紹的辦法，除了能夠用來增長topic的備份數量以外，還可以處理如下幾個場景：

**一、對topic的全部分區數據進行總體遷移。**怎麼理解呢？假如集羣有N個broker，後來新擴容M個broker。因爲新擴容的broker磁盤都是空的，原有的broker磁盤佔用都很滿。那麼咱們能夠利用上述方法，將存儲在原有N個broker的某些topic總體搬遷到新擴容的M個broker，進而實現kafka集羣的總體數據均衡。

​ 具體使用方法就是：經過編寫2.2章節的配置文件，將topic的全部分區都配置到新的M個broker上面去，再執行excute，便可完成topic的全部分區數據總體遷移到新擴容的M個broker節點。

***二、broker壞掉的狀況。***致使某些topic的某些分區的replica數量減小，能夠利用kafka-reassign-partitions.sh增長replica；

***三、kafka 某些broker磁盤佔用很滿，某些磁盤佔用又不多。***能夠利用kafka-reassign-partitions.sh遷移某些topic的分區數據到磁盤佔用少的broker，實現數據均衡；

***四、kafka集羣擴容。***須要把原來broker的topic數據總體遷移到新的broker，合理利用新擴容的broker，實現負載均衡。

此文已由做者受權騰訊雲+社區在各渠道發佈

獲取更多新鮮技術乾貨，能夠關注咱們騰訊雲技術社區-雲加社區官方號及知乎機構號