kafka2.3.1+zookeeper3.5.6+kafka-manager2.0.0.2集羣部署（centos7.7）

1、準備三臺服務器，配置好主機名和ip地址

 2、服務器初始化：包括安裝經常使用命令工具，修改系統時區，校對系統時間，關閉selinux，關閉firewalld，修改主機名，修改系統文件描述符，優化內核參數，優化數據盤掛載參數

一、安裝經常使用命令工具

yum install vim net-tools bash-completion wget unzip ntp bzip2 epel-release -y

二、修改系統時區，校對系統時間

timedatectl set-timezone Asia/Shanghai

ntpdate pool.ntp.org

三、關閉selinux

vim /etc/selinux/config

SELINUX=disabled

四、關閉防火牆

systemctl stop firewalld.service

systemctl disable firewalld.service

五、修改主機名

vim /etc/hostname

kafka-broker1

六、修改系統文件描述符大小

vim /etc/security/limits.conf

最後添加：

* soft nofile 655360
* hard nofile 655360
* soft nproc 655360
* hard nproc 655360
* soft memlock unlimited
* hard memlock unlimited

7.優化內核參數

vim /etc/sysctl.conf

最後添加：

vm.max_map_count = 655360
net.core.rmem_max = 2097152
net.core.wmem_max = 2097152
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_max_syn_backlog = 8192
net.core.netdev_max_backlog = 10000
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

8.優化數據盤掛載參數，個人數據盤掛載在opt下，對應本身數據盤做相應調整

vim /etc/fstab

/dev/mapper/centos-opt  /opt                    xfs     defaults,noatime,largeio        0 0

 9.重啓系統使配置生效

init 6

3、安裝zookeeper集羣

1.因zookeeper和kafka須要java啓動

首先安裝jdk1.8環境

yum install java-1.8.0-openjdk-devel.x86_64 -y

2.Kakfa集羣須要依賴ZooKeeper存儲Broker、Topic等信息，因此咱們先安裝zookeeper集羣

cd /opt

wget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.5.6/apache-zookeeper-3.5.6-bin.tar.gz

tar -zxvf apache-zookeeper-3.5.6-bin.tar.gz

mv apache-zookeeper-3.5.6 zookeeper-3.5.6

3.修改zookeeper配置文件

cd zookeeper-3.5.6/conf/

cp zoo_sample.cfg zoo.cfg

vim zoo.cfg

修改：

dataDir=/opt/zookeeper-3.5.6/data

末尾添加集羣其餘節點信息

server.1=192.168.0.13:2888:3888
server.2=192.168.0.14:2888:3888
server.3=192.168.0.15:2888:3888

4.添加zookeeper數據目錄

建立/opt/zookeeper-3.5.6/data目錄

mkdir /opt/zookeeper-3.5.6/data

在data目錄裏建立myid文件，寫上該節點id

 而後將opt下zookeeper-3.5.6目錄上傳到其餘兩個節點上

scp -r zookeeper-3.5.6 192.168.0.14:/opt
scp -r zookeeper-3.5.6 192.168.0.15:/opt

修改其餘兩個節點data下myid文件內容分別爲2和3

5.啓動zookeeper服務

三臺節點分別啓動zookeeper服務

/opt/zookeeper-3.5.6/bin/zkServer.sh start

6.查看節點zookeeper狀態

/opt/zookeeper-3.5.6/bin/zkServer.sh status

 4、安裝kafka集羣

1.官網下載kafka

cd /opt

wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.3.1/kafka_2.12-2.3.1.tgz

tar -zxvf kafka_2.12-2.3.1.tgz

2.修改kafka配置文件

cd kafka_2.12-2.3.1/config

vim server.properties

修改如下參數

broker.id=1
host.name=192.168.0.13
listeners=PLAINTEXT://192.168.0.13:9092
delete.topic.enable=true
log.cleanup.policy=delete
message.max.byte=5242880
default.replication.factor=2
replica.fetch.max.bytes=5242880
auto.create.topics.enable=true
num.network.threads=24
num.io.threads=48
log.dirs=/opt/kafka_2.12-2.3.1/kafka-logs
num.partitions=3
zookeeper.connect=192.168.0.13:2181,192.168.0.14:2181,192.168.0.15:2181

3.優化調整kafka jvm堆內存大小

 vim /opt/kafka_2.12-2.3.1/bin/kafka-server-start.sh

export KAFKA_HEAP_OPTS="-Xmx5G -Xms5G"

4.開啓kafka JMX監控

 vim /opt/kafka_2.12-2.3.1/bin/kafka-server-start.sh

export JMX_PORT="9999"

4.其餘兩臺節點按照上面步驟同步安裝kafka

5.三臺節點分別啓動kafka

/opt/kafka_2.12-2.3.1/bin/kafka-server-start.sh -daemon /opt/kafka_2.12-2.3.1/config/server.properties

5、安裝kafka-manager，方便管理kafka

1.下載kafka-manager源碼

cd /opt

wget https://github.com/yahoo/kafka-manager/archive/2.0.0.2.zip

mv 2.0.0.2.zip kafka-manager-2.0.0.2.zip

2.解壓zip包

unzip kafka-manager-2.0.0.2.zip

cd kafka-manager-2.0.0.2

3.yum安裝sbt(由於kafka-manager須要sbt編譯)

curl https://bintray.com/sbt/rpm/rpm > /etc/yum.repos.d/bintray-sbt-rpm.repo
yum install sbt -y

4.編譯kafka-manager

./sbt clean dist

可能要等好幾個小時。

 

 看到Your package is ready in /opt/kafka-manager-2.0.0.2/target/universal/kafka-manager-2.0.0.2.zip表明編譯成功了。

5.而後將編譯好的zip包拷貝到/opt下，並刪除原來kafka-manager-2.0.0.2文件夾

cp /opt/kafka-manager-2.0.0.2/target/universal/kafka-manager-2.0.0.2.zip /opt

rm -rf kafka-manager-2.0.0.2

unzip kafka-manager-2.0.0.2.zip

cd kafka-manager-2.0.0.2

6. 修改kafka-manager配置

vim conf/application.conf

kafka-manager.zkhosts="192.168.0.13:2181,192.168.0.14:2181,192.168.0.15:2181"

7.啓動kafka-manager

nohup /opt/kafka-manager-2.0.0.2/bin/kafka-manager &>>/dev/null &

8.用瀏覽器訪問http://192.168.0.13:9000/，添加集羣

 

 9.添加成功