Connect to kafka docker from outside docker network not working ?

This tutorial guides you on how to connect to kafka-docker from outside docker network i.e., how to connect to kafka running in docker container from outside docker network.

Connect to kafka docker from outside docker network

If you are using wurstmeister/kafka-docker for your deployments, then use the docker-compose.yml provided here so that you can connect to kafka-docker outside of docker network.

The wurstmeister image is available directly from Docker Hub.

Pre-requisites

Install docker-compose.

docker-compose.yml

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka:latest
    ports:
      - "9092:9092"
      - "9094:9094"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: INTERNAL://:9092,EXTERNAL://:9094
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://:9092,EXTERNAL://:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - zookeeper
    restart: always

Below is the kafkacat test response.

$ kafkacat -b localhost:9094 -L                                                                                                   
Metadata for all topics (from broker -1: localhost:9094/bootstrap):                                                                                     
 1 brokers:                                                                                                                                             
  broker 1001 at 1243061b0e0d:9094

Listener Configuration – Kafka Docker

I would recommend you to go through kafka documentation and understand the various broker listener configuration options.

Before Listeners, we had to use advertised.host.name and advertised.port, but later versions of Kafka have deprecated them. Note, advertised.host.name and advertised.port still work as expected, but these should not be used if we are configuring the listeners.

For example,

---
KAFKA_LISTENERS: INTERNAL://:9092,EXTERNAL://:9094
KAFKA_ADVERTISED_LISTENERS: INTERNAL://:9092,EXTERNAL://:9094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL

---

While configuring the listeners you need to keep in mind the following rules.

No listeners will share a port number.
An advertised.listener must be present by protocol name and port number in the list of listeners.

Injecting HOSTNAME_COMMAND into configuration

If you would like to use the value of HOSTNAME_COMMAND in any of the KAFKA_XXX variables, you can use the _{HOSTNAME_COMMAND} string in your variable value as shown below.

KAFKA_ADVERTISED_LISTENERS=SSL://_{HOSTNAME_COMMAND}:9094,PLAINTEXT://9092

For example,

HOSTNAME_COMMAND: curl http://35.192.169.252/latest/meta-data/public-hostname
KAFKA_ADVERTISED_LISTENERS: INTERNAL://:9092,EXTERNAL://_{HOSTNAME_COMMAND}:9094
KAFKA_LISTENERS: INTERNAL://:9092,EXTERNAL://:9094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL

Will result in the following kafka broker configuration

advertised.listeners = EXTERNAL://ec2-xx-xx-xxx-xx.us-west-2.compute.amazonaws.com:9094,INTERNAL://:9092
listeners = EXTERNAL://:9094,INTERNAL://:9092
inter.broker.listener.name = INTERNAL

That’s it. when you use the docker-compose.yml that’s provided you should be able to connect from outside the docker network and it’s working.

Hope it helped 🙂