Consuming data from kafka does not include key

jellang · November 2020

I am able to consume messages from a kafka topic and store them in a flex table using a COPY from KafkaSource statement. This however does not get the key set by kafka for each message.
The messages have a simple key i.e.: i-01234567 and the message itself is a simple json with one value: { "loginAt": "2020-08-28T23:59:59Z"}
Is there a way to also get the key?

SergeB · November 2020

Unfortunately, there is currently no mechanism to retrieve the Kafka message keys unless they're part of the message payload e.g { {"kafka_key":"i-01234567 ","loginAt": "2020-08-28T23:59:59Z"}

LauriPessi · December 2023

Is this something that's going to change in the (near) future?

It would be very beneficial to get the keys from messages - along with coordinates (partition, offset)

We're Moving!

Create My New Community Account Now

Consuming data from kafka does not include key

Best Answer

Answers

Leave a Comment