zk集羣docker-compose.ymljava
docker network create --driver bridge --subnet 172.23.0.0/25 --gateway 172.23.0.1 zookeeper_network
version: '3.4' services: zoo1: image: zookeeper restart: always hostname: zoo1 container_name: zoo1 ports: - 2181:2181 volumes: - "./zoo1/data:/data" - "./zoo1/datalog:/datalog" environment: ZOO_MY_ID: 1 ZOO_SERVERS: server.1=0.0.0.0:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888 networks: default: ipv4_address: 172.23.0.11 zoo2: image: zookeeper restart: always hostname: zoo2 container_name: zoo2 ports: - 2180:2181 volumes: - "./zoo2/data:/data" - "./zoo2/datalog:/datalog" environment: ZOO_MY_ID: 2 ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=0.0.0.0:2888:3888 server.3=zoo3:2888:3888 networks: default: ipv4_address: 172.23.0.12 zoo3: image: zookeeper restart: always hostname: zoo3 container_name: zoo3 ports: - 2179:2181 volumes: - "./zoo3/data:/data" - "./zoo3/datalog:/datalog" environment: ZOO_MY_ID: 3 ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=0.0.0.0:2888:3888 networks: default: ipv4_address: 172.23.0.13 networks: default: external: name: zookeeper_network
version: '2' services: kafka1: image: wurstmeister/kafka restart: always hostname: kafka1 container_name: kafka1 ports: - "9092:9092" environment: KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.0.6:9092 KAFKA_ADVERTISED_HOST_NAME: 192.168.0.6 KAFKA_ADVERTISED_PORT: 9092 KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181 JMX_PORT: 9999 volumes: - ./kafka1/logs:/kafka external_links: - zoo1 - zoo2 - zoo3 networks: default: ipv4_address: 172.23.0.14 kafka2: image: wurstmeister/kafka restart: always hostname: kafka2 container_name: kafka2 ports: - "9093:9092" environment: KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.0.6:9093 KAFKA_ADVERTISED_HOST_NAME: 192.168.0.6 KAFKA_ADVERTISED_PORT: 9093 KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181 JMX_PORT: 9988 volumes: - ./kafka2/logs:/kafka external_links: # 鏈接本compose文件之外的container - zoo1 - zoo2 - zoo3 networks: default: ipv4_address: 172.23.0.15 kafka3: image: wurstmeister/kafka restart: always hostname: kafka3 container_name: kafka3 ports: - "9094:9092" environment: KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.0.6:9094 KAFKA_ADVERTISED_HOST_NAME: 192.168.0.6 KAFKA_ADVERTISED_PORT: 9094 KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181 JMX_PORT: 9977 volumes: - ./kafka3/logs:/kafka external_links: # 鏈接本compose文件之外的container - zoo1 - zoo2 - zoo3 networks: default: ipv4_address: 172.23.0.16 kafka-manager: image: sheepkiller/kafka-manager:latest restart: always container_name: kafa-manager hostname: kafka-manager ports: - "9002:9000" links: # 鏈接本compose文件建立的container - kafka1 - kafka2 - kafka3 external_links: # 鏈接本compose文件之外的container - zoo1 - zoo2 - zoo3 environment: ZK_HOSTS: zoo1:2181,zoo2:2181,zoo3:2181 KAFKA_BROKERS: kafka1:9092,kafka2:9093,kafka3:9094 APPLICATION_SECRET: letmein KM_ARGS: -Djava.net.preferIPv4Stack=true networks: default: ipv4_address: 172.23.0.10 networks: default: external: # 使用已建立的網絡 name: zookeeper_network
錯誤:node
INFO Opening socket connection to server zk2.zookeeper_default/172.22.0.3:2180. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) docker-kafka | [2019-01-19 05:22:45,206] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) docker-kafka | java.net.ConnectException: Connection refused docker-kafka | at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) docker-kafka | at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) docker-kafka | at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) docker-kafka | at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) docker-kafka | [2019-01-19 05:22:45,311] INFO Opening socket connection to server zk3.zookeeper_default/172.22.0.2:2179. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
解決辦法:docker
兩個docker-compose使用相同的網絡apache
kafka3 | [2019-01-19 05:49:11,086] WARN [Controller id=1001, targetBrokerId=1003] Connection to node 1003 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) kafka3 | [2019-01-19 05:49:11,146] WARN [Controller id=1001, targetBrokerId=1001] Connection to node 1001 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) kafka3 | [2019-01-19 05:49:11,187] WARN [Controller id=1001, targetBrokerId=1003] Connection to node 1003 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
找了挺多靠譜和不靠譜的解決方案,都沒能解決問題,但也給我提示。網絡
這個問題無非就是kafka集羣,這裏是經過宿主機鏈接節點的,socket
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:9094
KAFKA_ADVERTISED_HOST_NAME: kafka3
這兩個配置能夠配置宿主機的IP或者容器的hostname均可,可是有兩點:一、端口須要暴露出來 二、而且能夠經過宿主機訪問進去
我是第二點被坑了,fire-wall開發端口,就沒有上面這個錯誤了。
而後點擊連接開啓jmx工具
因而就出現下面的錯誤ui
[error] k.m.j.KafkaJMX$ - Failed to connect to service:jmx:rmi:///jndi/rmi://9.11.8.48:-1/jmxrmi java.lang.IllegalArgumentException: requirement failed: No jmx port but jmx polling enabled!
解決辦法是:spa
在每個kafka節點加上環境變量 JMX_PORT=端口.net
加上以後發現連不上,又是網絡鏈接的問題,因而又把每一個jmx端口暴露出來,而後fire-wall放行, 解決問題。
KAFKA_ADVERTISED_HOST_NAME: 這個最好設置宿主機的ip,宿主機之外的代碼或者工具來鏈接,後面的端口也須要設置暴露的端口。