Andre Schütz - Learn. Think. Reflect - Tech developer
Workign with Kafka and Zookeeper within Docker or from source

Workign with Kafka and Zookeeper within Docker or from source

If you are working with Kafka and Zookeeper to create some test cases or for the work in bigger projects, you can install them directly from the respective project website or make an installation with a Docker container. The following hints can help you with your first steps to make progress when working with the tools.

Kafka and Zookeeper within Docker

The examples assumes that the client can access the instance within the Docker at port 33222.

Output the logs

docker logs -f my-project-kafka-c
docker logs -f my-project-zk4kafka-c

Output all messages for a specific topic

docker exec -it my-report-kafka-c /usr/bin/kafka-console-consumer --bootstrap-server localhost:33222 --topic TOPICNAME --from-beginning

Kafka and Zookeeper as direct installation

Download Apache Kafka from the respective Apache project website. Extract the package and you can execute the following tasks.

The following examples expect that you are within the respective folder of Apache Kafka. The version of Apache Kafka is kafka_2.12-1.0.0.

Start the Apache Zookeeper and Apache Kafka locally

bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties

Create a topic

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic TOPICNAME

Sent a message to the topic

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic TOPICNAME

Show all received messages for a specific topic

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic TOPICNAME --from-beginning

Show the existing topics

bin/kafka-topics.sh --list --zookeeper localhost:2181

Use local Apache Kafka instance to access data within Apache Kafka instance of Docker

The examples assumes that the client can access the Apache Kafka instance within the Docker at port 33222 and the Apache Zookeeper instance within Docker at port 33272.

Consume messages for a specific topic

bin/kafka-console-consumer.sh --bootstrap-server localhost:33222 --topic TOPICNAME --from-beginning

Produce messages to the instance within Docker

bin/kafka-console-producer.sh --broker-list localhost:33222 --topic TOPICNAME

List the topics from the Apache Zookeeper instance within Docker

bin/kafka-topics.sh --list --zookeeper localhost:33272

Additional tools

Look for processes that occur on the port for the local Apache Kafka within Docker

The example assumes that the client can access the Apache Kafka instance within the Docker at port 33222.

netstat -tulpe | grep 33222

Remove everything that was created within docker

docker-compose down -v

Leave a Reply