Options

how to use KafkaJSONParser in scheduler?

I am trying to send messages from Kafka to Vertica flex table using KafkaJSONParser. 

However JSON formatted message does not load into table as JSON format.

 

This is my vkconfig topic command using KafkaJSONParser:

./vkconfig topic --edit --conf config.properties --target flextest1.kafka --rejection-table flextest1.kafka_rej --config-schema flextest1 --topic flextest1 --parser KafkaJSONParser

 

When I editted vkconfig topic, it is not updated in the table:

kafka=> select * from flextest1.kafka_targets;
target_schema | target_table | enabled | parser | rejection_table | parser_parameters | load_method | target_columns
---------------+--------------+---------+-------------+---------------------+-------------------+-------------+----------------
flextest1 | kafka | t | KafkaParser | flextest1.kafka_rej | | TRICKLE |
(1 row)

 

I even tried to change parser value manually to KafkaJSONParser, I am unable to parse as JSON formatted.

 

JSON formatted message I tried to send from Kafka:

{"mac": "f0:cb:d1:5b:b8:0f","ipAddress": "10.255.220.12"}

 

The message goes to __raw__ column as it is, but not JSON formatted.

 

 

Comments

  • Options
    SruthiASruthiA Vertica Employee Administrator

    Hi,

     

        Did you launch the scheduler to load data in JSON format? What is the command you used to lauch? Can you try executing it using entire path as specified in the documentation

     

     

    /opt/vertica/packages/kafka/bin/vkconfig topic --edit --target flextest1.kafka --parser KafkaJSONParser

    Now query your target table to see if it has changed

     

     

    Sruthi

  • Options

    Thank you Sruthi,

    It worked as charm when running vkconfig via full path.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file