Kubernetes搭建Zookeeper和Kafka集羣

時間 2019-11-10

標籤 kubernetes 搭建 zookeeper kafka 集羣欄目 Zookeeper 简体版

原文原文鏈接

主要參考了https://www.cnblogs.com/00986014w/p/9561901.html 這篇博文，但他zookeeper使用的不是官方鏡像html

我使用的是3個節點的Zookeeper集羣，能夠參照着修改。node

搭建Zookeeper集羣

集羣Service的`zookeeper-svc.yaml`

apiVersion: v1
kind: Service
metadata:
  labels:
    app: zookeeper-cluster-service-1
  name: zookeeper-cluster1
spec:
  ports:
  - name: client 
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zookeeper-cluster-service-1
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: zookeeper-cluster-service-2
  name: zookeeper-cluster2
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zookeeper-cluster-service-2
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: zookeeper-cluster-service-3
  name: zookeeper-cluster3
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zookeeper-cluster-service-3

經過sudo kubectl create -f zookeeper-svc.yaml建立3個Service。bootstrap

集羣Deployment的`zookeeper-deployment.yaml`

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: zookeeper-cluster-service-1 
  name: zookeeper-cluster-1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: zookeeper-cluster-service-1
        name: zookeeper-cluster-1 
    spec:
      containers:
      - image: zookeeper
        imagePullPolicy: IfNotPresent
        name: zookeeper-cluster-1 
        ports:
        - containerPort: 2181
        env:
        - name: ZOO_MY_ID
          value: "1"
        - name: ZOO_SERVERS
          value: "server.1=0.0.0.0:2888:3888 server.2=zookeeper-cluster2:2888:3888 server.3=zookeeper-cluster3:2888:3888"
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: zookeeper-cluster-service-2
  name: zookeeper-cluster-2
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: zookeeper-cluster-service-2
        name: zookeeper-cluster-2
    spec:
      containers:
      - image: zookeeper
        imagePullPolicy: IfNotPresent
        name: zookeeper-cluster-2
        ports:
        - containerPort: 2181
        env:
        - name: ZOO_MY_ID
          value: "2"
        - name: ZOO_SERVERS
          value: "server.1=zookeeper-cluster1:2888:3888 server.2=0.0.0.0:2888:3888 server.3=zookeeper-cluster3:2888:3888"
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: zookeeper-cluster-service-1
  name: zookeeper-cluster-3
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: zookeeper-cluster-service-3
        name: zookeeper-cluster-3
    spec:
      containers:
      - image: zookeeper
        imagePullPolicy: IfNotPresent
        name: zookeeper-cluster-3
        ports:
        - containerPort: 2181
        env:
        - name: ZOO_MY_ID
          value: "3"
        - name: ZOO_SERVERS
          value: "server.1=zookeeper-cluster1:2888:3888 server.2=zookeeper-cluster2:2888:3888 server.3=0.0.0.0:2888:3888"

經過sudo kubectl create -f zookeeper-deployment.yaml建立3個Deployment。api

檢查集羣是否啓動成功

等3個Pod成爲Running狀態後，經過sudo kubectl log zookeeper-cluster-1-xxxxxx檢查日誌中是否有錯誤。bash

而後經過sudo kubectl exec -it zookeeper-cluster-1-676df4686f-c7b6d /bin/bash分別進入兩個Pod，執行/bin/zkCli.sh分別建立查看，試試是否能成功。app

搭建Kafka集羣

集羣Service的`kafka-svc.yaml`

apiVersion: v1
kind: Service
metadata:
  name: kafka-cluster1
  labels:
    app: kafka-cluster-1
spec:
  type: NodePort
  ports:
  - port: 9092
    name: kafka-cluster-1
    targetPort: 9092
    nodePort: 30091
    protocol: TCP
  selector:
    app: kafka-cluster-1
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-cluster2
  labels:
    app: kafka-cluster-2
spec:
  type: NodePort
  ports:
  - port: 9092
    name: kafka-cluster-2
    targetPort: 9092
    nodePort: 30092
    protocol: TCP
  selector:
    app: kafka-cluster-2
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-cluster3
  labels:
    app: kafka-cluster-3
spec:
  type: NodePort
  ports:
  - port: 9092
    name: kafka-cluster-3
    targetPort: 9092
    nodePort: 30093
    protocol: TCP
  selector:
    app: kafka-cluster-3

經過sudo kubectl create -f kafka-svc.yaml建立3個Service。日誌

集羣Deployment的`kafka-deployment.yaml`

這裏須要注意，要把env中KAFKA_ADVERTISED_HOST_NAME改爲各個Pod對應Service的ClusterIP。code

PS：若是上面zookeeper的service和我定義的不一樣，就對應着修改KAFKA_ZOOKEEPER_CONNECTserver

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-cluster-1 
spec:
  replicas: 1
  selector:
    matchLabels:
      name: kafka-cluster-1 
  template:
    metadata:
      labels:
        name: kafka-cluster-1
        app: kafka-cluster-1
    spec:
      containers:
      - name: kafka-cluster-1
        image: wurstmeister/kafka
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9092
        env:
        - name: KAFKA_ADVERTISED_PORT
          value: "9092"
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: "[zookeeper-cluster1  的 ClusterIP]"
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zookeeper-cluster1:2181,zookeeper-cluster2:2181,zookeeper-cluster3:2181
        - name: KAFKA_BROKER_ID
          value: "1"
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-cluster-2
spec:
  replicas: 1
  selector:
    matchLabels:
      name: kafka-cluster-2
  template:
    metadata:
      labels:
        name: kafka-cluster-2
        app: kafka-cluster-2
    spec:
      containers:
      - name: kafka-cluster-2
        image: wurstmeister/kafka
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9092
        env:
        - name: KAFKA_ADVERTISED_PORT
          value: "9092"
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: "[zookeeper-cluster2 的  ClusterIP]"
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zookeeper-cluster1:2181,zookeeper-cluster2:2181,zookeeper-cluster3:2181
        - name: KAFKA_BROKER_ID
          value: "2"
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-cluster-3
spec:
  replicas: 1
  selector:
    matchLabels:
      name: kafka-cluster-3
  template:
    metadata:
      labels:
        name: kafka-cluster-3
        app: kafka-cluster-3
    spec:
      containers:
      - name: kafka-cluster-3
        image: wurstmeister/kafka
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9092
        env:
        - name: KAFKA_ADVERTISED_PORT
          value: "9092"
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: "[zookeeper-cluster3  的  ClusterIP]"
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zookeeper-cluster1:2181,zookeeper-cluster2:2181,zookeeper-cluster3:2181
        - name: KAFKA_BROKER_ID
          value: "3"

經過sudo kubectl create -f zookeeper-deployment.yaml建立3個Deployment。htm

檢查集羣是否啓動成功

等3個Pod成爲Running狀態後，經過sudo kubectl log kafka-cluster-1-xxxxxx檢查日誌中是否有錯誤。

而後經過sudo kubectl exec -it -sudo kubectl exec -it kafka-cluster-1-558747bc7d-5n94p /bin/bash進入Pod，執行kafka-console-producer.sh --broker-list [zookeeper-cluster1 的 ClusterIP]:9092 --topic test建立了topic test。

經過sudo kubectl exec -it kafka-cluster-2-66c88f759b-8wlvp /bin/bash進入Pod，執行kafka-console-consumer.sh --bootstrap-server [zookeeper-cluster2 的 ClusterIP]:9092 --topic test --from-beginning接收topic test的消息。

而後試着在cluster-1中發送消息，看看在cluster-2中是否能接收。