How to view Kafka messages using kafka testclient ?

This tutorial guides you on how to view kafka messages using kafka testclient. Let’s say you are running kafka docker container in the kubernetes cluster. And you would like to view kafka messages sent to the specific topic then you are at the right place.

Install testclient to view Kafka messages

Use the following deployment YAML file to deploy a kafka testclient pod in your kubernetes cluster.

kafka-testclient.yaml

apiVersion: v1
kind: Pod
metadata:
  name: kafka-testclient
  namespace: kafka
spec:
  containers:
  - name: kafka
    image: solsson/kafka:0.11.0.0
    command:
      - sh
      - -c
      - "exec tail -f /dev/null"

First, you need to connect to the kubernetes cluster. In case of GKE cluster run the following command to make an entry in the kubeconfig.

$ gcloud container clusters get-credentials cluster1 --zone us-central1-c --project sne5g21

Then, run the following kubectl command to deploy the kafka-testclient Pod in the kubernetes cluster.

$ kubectl apply -f kafka-testclient

Also, you can check whether the specific topic exists or not using the following command.

$ kubectl -n mv exec -ti kafka-testclient -- ./bin/kafka-topics.sh --zookeeper zookeeper:2181 --list

The above command will list all the kafka topics.

View Kafka messages using kafka testclient

Now, let’s see how to view kafka messages in a specific topic using kafka testclient.

You can use kafka-console-consumer.sh script to view kafka messages produced on some topic as shown below.

$ kubectl -n mv exec -ti testclient -- ./bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic topic1 --from-beginning
 
Hello !!
How are you doing ?

Note, the kafka console consumer is a tool that reads data from Kafka topic and outputs it to standard output. The following are the options that you can use based on your needs. For example, if you would like kafka consumer to consume or read maximum 10 messages from a kafka topic before exiting, then you can use option –max-messages <Integer: num_messages>.

kafka-console-consumer options

Below are some of the important list of options that you might use with kafka-console-consumer when needed.

–bootstrap-server <server> – The <server> to connect to.

–zookeeper <urls> -The connection string for the zookeeper connection in the form host:port.

–topic <topic> – The topic id to consume on.

–partition <Integer: partition> – The partition to consume from.

–from-beginning – To consume messages starting from the beginning instead of the latest message.

–max-messages <Integer: num_messages> – The maximum number of messages to consume before exiting. If not set, consumption is continual.

–offset <consume offset> – The offset id to consume from (a non-negative number).

For other options details please check kafka-console-consumer options.

That’s it you had learnt how to view kafka messages using kafka testclient utility.

Hope it helped 🙂

Install testclient to view Kafka messages

View Kafka messages using kafka testclient

kafka-console-consumer options

References