Kafka Docker Compose Setup

A Docker Compose configuration for running Apache Kafka with Zookeeper and Kafbat UI for easy cluster management and monitoring.

Services

Zookeeper

Image: confluentinc/cp-zookeeper:7.6.1
Port: 2181
Purpose: Coordinates and manages the Kafka cluster

Kafka Broker

Image: confluentinc/cp-kafka:7.6.1
Ports:
- 9092: External access (use this from your host machine)
- 29092: Internal Docker network communication
Features:
- 7-day log retention (168 hours)
- Persistent data storage via Docker volume
- Single broker configuration (replication factor: 1)

Kafbat UI

Image: ghcr.io/kafbat/kafka-ui:latest
Port: 8080
Purpose: Web-based UI for managing and monitoring Kafka
Access: http://localhost:8080

Prerequisites

Docker installed
Docker Compose installed
Port availability: 2181, 8080, 9092, 29092

Getting Started

Start the services:
```
docker-compose up -d
```
Check service status:
```
docker-compose ps
```
Access Kafbat UI: Open your browser and navigate to http://localhost:8080
View logs:
```
docker-compose logs -f
```

Configuration

Kafbat UI Configuration

The UI expects a configuration file at ./kui/config.yml. Create this file to define your Kafka clusters and connection settings.

Example kui/config.yml:

kafka:
  clusters:
    - name: local
      bootstrapServers: kafka:29092

Connecting to Kafka

From Host Machine

Use connection string: localhost:9092

From Docker Containers

Use connection string: kafka:29092

Useful Commands

Stop all services:

docker-compose down

Stop and remove volumes (⚠️ deletes all Kafka data):

docker-compose down -v

Restart services:

docker-compose restart

View Kafka logs:

docker-compose logs -f kafka

Data Persistence

Kafka data is persisted in the kafka_data Docker volume, ensuring your topics and messages survive container restarts.

Troubleshooting

Services won't start: Ensure ports 2181, 8080, 9092, and 29092 are not already in use
Cannot connect to Kafka: Verify the broker is running with docker-compose ps
Kafbat UI not loading: Check that ./kui/config.yml exists and is properly formatted
Connection refused: Allow a few seconds for Kafka to fully initialize after starting

Notes

This is a single-broker setup suitable for development and testing
For production, consider using multiple brokers and higher replication factors
All services are configured with restart: unless-stopped for automatic recovery

services: zookeeper: image: confluentinc/cp-zookeeper:7.6.1 container_name: zookeeper restart: unless-stopped environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 ports: - "2181:2181" kafka: image: confluentinc/cp-kafka:7.6.1 container_name: kafka restart: unless-stopped depends_on: - zookeeper ports: - "9092:9092" # External access for remote clients (n8n) - "29092:29092" # Internal access within Docker network environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,PLAINTEXT_INTERNAL://0.0.0.0:29092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://<YOUR_HOST_IP>:9092,PLAINTEXT_INTERNAL://kafka:29092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT_INTERNAL KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_LOG_RETENTION_HOURS: 168 KAFKA_LOG_DIRS: /var/lib/kafka/data volumes: - kafka_data:/var/lib/kafka/data kafbat-ui: image: ghcr.io/kafbat/kafka-ui:latest container_name: kafbat-ui restart: unless-stopped depends_on: - kafka - zookeeper ports: - "8080:8080" environment: DYNAMIC_CONFIG_ENABLED: 'true' volumes: - ./kui/config.yml:/etc/kafkaui/dynamic_config.yaml volumes: kafka_data: