Consuming data from kafka does not include key

I am able to consume messages from a kafka topic and store them in a flex table using a COPY from KafkaSource statement. This however does not get the key set by kafka for each message.
The messages have a simple key i.e.: i-01234567 and the message itself is a simple json with one value: { "loginAt": "2020-08-28T23:59:59Z"}
Is there a way to also get the key?

Tagged:

Best Answer

  • SergeBSergeB Employee
    Answer ✓

    Unfortunately, there is currently no mechanism to retrieve the Kafka message keys unless they're part of the message payload e.g { {"kafka_key":"i-01234567 ","loginAt": "2020-08-28T23:59:59Z"}

Answers

  • Is this something that's going to change in the (near) future?

    It would be very beneficial to get the keys from messages - along with coordinates (partition, offset)

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file