Is there a way to return the number of messages sent to Kafka using KafkaExport?

HyeontaeJu

Is there a way to return the number of messages sent to Kafka using KafkaExport?

Jim_Knicely

You should be able to use KAFKACAT to list the messages sent to Kafka from Vertica.

Something like:

./kafkacat -C -t <Topic> -b <Broker>

See:
https://www.vertica.com/docs/latest/HTML/Content/Authoring/KafkaIntegrationGuide/TroubleShooting/UsingKafkacatToTroubleShootIssues.htm

SergeB

The number of messages produced to Kafka should match the number of rows returned by the query used in KafkaExport MINUS any error reported by KafkaExport.

HyeontaeJu

@Jim_Knicely, @SergeB
Then,, Is it impossible return the number of message sent to Kafka topic immediately after KafkaExport query executing?

SergeB

You could determine the number of rows from your select statement (select count(*)....) immediately before or after calling KafkaExport