The Vertica Forum recently got a makeover! Let us know what you think by filling out this short, anonymous survey.
Please take this survey to help us learn more about how you use third party tools. Your input is greatly appreciated!

If Vertica kafka Utilities support parsing Avro Enum type data?

In my avro schema, there are two enum type fields defined, like below.

 

"fields" : [ {"name" : "eventType", "type": {"name" : "EventType", "type": "enum", "symbols" : ["A", "C"]}}, {"name": "packetType", "type": {"name": "PacketType", "type": "enum", "symbols" : ["IIPMS", "SMS"]}}]

 

I use a kafka schedular to parse messages from kafka with KafkaAVROParser, but the values can't be persisted into db table.

 

Untitled picture.png

 

I want to know if vertica kafka utility support the Avro enum type data parsing and persistence. If no, I will change the enum type to string. If yes, why the data value can't be persisted correctly, is it because I miss some important configurations?

Comments

  • Hi

     

    Yes, KafkaAvroParser supports Avro Enum type.

    How are you writing the messages? see: https://community.dev.hpe.com/t5/Vertica-Forum/Loading-AVRO-data-using-KAFKA-Parser/m-p/235240

    Does your schema contains siteID and alarmString so those values were also loaded using KafkaAvroParser?

     

    E.g. Enum type is loaded using KafkaAvroParser with the following Avro schema (it is an Avro Record containing String fields and also the Enum fileds)

     

    {"type":"record","name":"Users","namespace":"test","fields":[{"name":"username","type":"string"},{"name":"age","type":"int"},{"name":"address","type":"string"},{"name" : "eventType", "type": {"name" : "EventType", "type": "enum", "symbols" : ["A", "C"]}}, {"name": "packetType", "type": {"name": "PacketType", "type": "enum", "symbols" : ["IIPMS", "SMS"]}}],"doc":"Record with enums"}

     

    So target FlexTable contains

    SELECT maptostring(__raw__) FROM myTargetTable;
    maptostring
    -----------------------------------------------------------------------------------------------------------------------------------------------------
    {
    "EventType" : "A",
    "PacketType" : "SMS",
    "__name__" : "Users",
    "address" : "Street 1 #54",
    "age" : "30",
    "username" : "John"
    }

    (1 row)

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file