How to list all kafka topics using kafka testclient ?

This tutorial guides you on how to list all kafka topics using kafka testclient. Let’s say you are running kafka docker container in a kubernetes pod. And you would like to check or list all kafka topics then you are at the right place.

Deploy a test client in kubernetes cluster

You can use the following deployment YAML file (kafka-testclient.yaml) to deploy a kafka testclient pod in your kubernetes cluster.

kafka-testclient.yaml

apiVersion: v1
kind: Pod
metadata:
  name: kafka-testclient
  namespace: kafka
spec:
  containers:
  - name: kafka
    image: solsson/kafka:0.11.0.0
    command:
      - sh
      - -c
      - "exec tail -f /dev/null"

First, connect to the kubernetes cluster. In case of GKE cluster you need to run the following command.

$ gcloud container clusters get-credentials cluster1 --zone us-central1-c --project sne5g21

Then, run the following kubectl command to deploy the kafka-testclient Pod.

$ kubectl apply -f kafka-testclient

Then using this kafka-testclient, you can create test topics as shown below.

$ kubectl -n mv exec -ti kafka-testclient -- ./bin/kafka-topics.sh --zookeeper zookeeper:2181 --topic topic1 --create --partitions 1 --replication-factor 1
$ kubectl -n mv exec -ti kafka-testclient -- ./bin/kafka-topics.sh --zookeeper zookeeper:2181 --topic topic2 --create --partitions 1 --replication-factor 1
$ kubectl -n mv exec -ti kafka-testclient -- ./bin/kafka-topics.sh --zookeeper zookeeper:2181 --topic topic3 --create --partitions 1 --replication-factor 1

The above commands would have created topics topic1, topic2 and topic3. Note, here we need to use the correct hostname for zookeeper cluster and the topic configuration.

List all kafka topics using kafka testclient

Now, let’s see how to list all kafka topics using the kafka-testclient Pod.

Run the following command to list all the kafka topics created.

$ kubectl -n mv exec -ti kafka-testclient -- ./bin/kafka-topics.sh --zookeeper zookeeper.mv.svc.clusterset.local:2181 --list

Defaulting container name to kafka.
Use 'kubectl describe pod/testclient -n mv' to see all of the containers in this pod.
topic1
topic2
topic3

Similarly, you can test kafka producer by running the following command using kafka-testclient.

$ kubectl -n mv exec -ti kafka-testclient -- ./bin/kafka-console-producer.sh --broker-list kafka:9092 --topic topic1

> Hello !!
> How are you doing ?
>

And you can also test kafka consumer, so that we can consume the messages that were produced by running the following command in another terminal.

$ kubectl -n mv exec -ti testclient -- ./bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic topic1 --from-beginning

Hello !!
How are you doing ?

That’s it you had learnt how to deploy a kafka client as pod to communicate to zookeeper and kafka services deployed in the GKE Kubernetes Cluster. And did a quick test to list all the kafka topics using the testclient.

Hope it helped 🙂

Deploy a test client in kubernetes cluster

List all kafka topics using kafka testclient

References