Short Tip on Kafka:Quick and Easy Local Development Setup

Reading Time: 3 minutes

When it comes to a need to setup a local Apache Kafka instance for some experiments or tests it may turn into an exercise that is not easy to complete.

Problems that may arise are usually quite different, it can be connectivity between Kafka and Zookeeper (prior to KIP-500), exposing a listener for incoming connections, or plugging in any friendly UI for handy observability/management of the Kafka instance.

To simplify these preparational steps Docker Compose can be used. It allows to start all the components, wire them up in a single network and expose necessary ports to the host machine. But it also requires a bit of knowledge about Kafka internals and configuration properties. So making it from scratch may require some effort.

And here is a small guide on how this can be done faster.

TL;DR

Complete configuration for Docker Compose can be found here: docker-compose.yml. It contains the following components:

Kafka;
Zookeeper;
Kafdrop, web-basd UI.

After it is launched (via docker-compose up -d), in a matter of a minute, a fully functional local Kafka instance will be available.

Configuration Details

Here are some parts of the configuration that might be interesting to know about.

Zookeeper

Zookeeper exposes optional port 2181 for external connectivity. It is not mandatory as this port is usually used only inside the dedicated sub-network created by Docker Compose, but it also allows connections from the host machine.

Kafka

Kafka defines 9092 port for connections from the host machine, and also starts only after Zookeeper is ready (defined by depends_on property).

On top it defines the following environment variables:

environment:

# ...

# aliases used to map listener names and related protocols,
# PLAINTEXT means that neither authentication nor encryption is used
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT

# actual listener definitions that will be used by Kafka
# for incoming connections
- KAFKA_LISTENERS=INTERNAL://:29092,EXTERNAL://:9092

# metadata published to Zookeeper that will be shared to clients 
# to let them establish an incoming connections
- KAFKA_ADVERTISED_LISTENERS=INTERNAL://kafka:29092,EXTERNAL://localhost:9092

# ...

More details about Kafka configuration are available in the official documentation.

Kafdrop

Exposes port 9000 to the web interface that gives observability over Kafka instance.