kafka集羣搭建

一直據說過kafka跟rabbitmq，redis類似，可是沒試過，今天特地抽了點時間學習下kafka集羣搭建，瞭解下。

1.環境準備

3臺虛擬機（1C4G）

192.168.77.31　　zk-1

192.168.77.32　　zk-2

192.168.77.33　　zk-3

經過官網下載好安裝包.

 

安裝kafka集羣，須要先安裝zookeeper，這裏也一併把zookeeper集羣也裝了一遍。

 

2.安裝過zookeeper集羣

1. 安裝jdk1.8，並設置環境變量加載
echo 'export JAVA_HOME=/usr/java/jdk1.8.0_181-amd64/' >>/etc/profile
echo 'export PATH=$JAVA_HOME/bin:$PATH' >>/etc/profile
echo 'export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar' >>/etc/profile

source /etc/profile

2. 安裝zookeeper
#安裝zookeeper
tar -xf ./zookeeper-3.4.14.tar.gz -C /opt && ln -s /opt/zookeeper-3.4.14/ /usr/local/zookeeper

cd /usr/local/zookeeper/conf
mv ./zoo_sample.cfg zoo.cfg

修改配置文件
tickTime=2000
initLimit=10
syncLimit=5
dataLogDir=/opt/zookeeper/logs
dataDir=/opt/zookeeper/data
clientPort=2181
autopurge.snapRetainCount=500
autopurge.purgeInterval=24

#對應機器的ip設置爲0.0.0.0
server.1= 192.168.77.31:2888:3888
server.2= 192.168.77.32:2888:3888
server.3= 192.168.77.33:2888:3888

#建立相關目錄，三臺節點都須要
mkdir -p /opt/zookeeper/{logs,data}

建立ServerID標識，每臺機器的ServerID都不一致
echo "1" > /opt/zookeeper/data/myid
#添加環境變量
sed -i '$a\export PATH=$PATH:/usr/local/zookeeper/bin' /etc/profile
#刷新環境變量
source /etc/profile
zkServer.sh start
#查看zk集羣狀態
zkServer.sh status

 

3.安裝kafka集羣

tar -xf kafka_2.12-2.3.0.tgz -C /usr/local/
cd /usr/local/kafka_2.12-2.3.0/config/
修改server.properties文件
config/server.properties:
broker.id=0 #id惟一，三臺不能重複
listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs
zookeeper.connect=zk-1:2181,zk-2:2181,zk-3:2181

echo 'export PATH=$PATH:/usr/local/kafka_2.12-2.3.0/bin/' >>/etc/profile

source /etc/profile

 

後臺啓動kafka

nohup kafka-server-start.sh /usr/local/kafka_2.12-2.3.0/config/server.properties &

 啓動後查看進程是否啓動完成

 

一些經常使用的kafka命令

Create a topic

kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test

replication-factor 複製印子1

partitions 分區

列出主題
kafka-topics.sh --list --bootstrap-server localhost:9092

Send some messages
kafka-console-producer.sh --broker-list localhost:9092 --topic test

 

Now create a new topic with a replication factor of three:
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 3 --partitions 1 --topic my-replicated-topic

Okay but now that we have a cluster how can we know which broker is doing what? To see that run the "describe topics" command:
kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic my-replicated-topic

 

消費信息kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic