Not able to compute keys vertica kafka




I am trying to integrate vertica kafka. I am storing data in flex table. Vertica is not able to parser data.



Select * from table. I get


1 | {"classname":"global", "name": "Everest", "type":"mountain", "height":29029, "hike_safety": 34.1}
2 | {"classname":"global", "name": "Mt St Helens", "type":"volcano", "height":29029, "hike_safety": 15.4}
3 | {"classname":"application", "name": "Denali", "type":"mountain", "height":17000, "hike_safety": 12.2}
4 | {"classname":"application", "name": "Kilimanjaro", "type":"mountain", "height":14000 }
5 | {"classname":"global", "name": "Everest", "type":"mountain", "height":29029, "hike_safety": 34.1}
250001 | {"classname":"global", "name": "Everest", "type":"mountain", "height":29029, "hike_safety": 34.1}
250002 | {"classname":"global", "name": "Mt St Helens", "type":"volcano", "height":29029, "hike_safety": 15.4}




But when I try to compute key using " select compute_flextable_keys('table');" I am getting this error.


ERROR 5861: Error calling processPartition() in User Function MapKeysInfo at [src/Dict.cpp:639], error code: 0, message: Exception while processing partition row: [1] on map keys information retrieval: [Tried to retrieve Flex Table data from a cell not containing an Flex Table map.]


  • Options



    After a quick look by one of the R&D team members we think you may be using the base Kafka Parser and not the Kafka JSON Parser.


    Check out this section of the documentation:





  • Options


    Hi Mark,


    Thanks for your response.

    With COPY statment, vertica is parsing the json input correctly. How to do it with scheduler?

    I am using the following command to assign the json parser to scheduler.


    /opt/vertica/packages/kafka/bin/vkconfig topic --add --target data_tgt --rejection-table data_rej --topic data --num-partitions 1 --config-schema data  --parser KafkaJSONParser

  • Options

    Hi phmandawa,


    I am having exactly same problem as yours.

    My vkconfig topic command is same as yours, but it seems it does not parsing json formatted message to the vertica flex table.


    Another thing that I noticed is everytime I edit topic with --parser KafkaJSONParser option, it does not reflect in kafka_targets table. Example below shows KafkaParser all the time. Shouldn't it be changed to KafkaJSONParser?  I even tried to update column value manually to KafkaJSONParser, Kafka does not want to parse as JSON format.


    target_schema | target_table | enabled | parser | rejection_table | parser_parameters | load_method | target_columns
    flextest1 | kafka | t | KafkaParser | flextest1.kafka_rej | | TRICKLE |


    Have you solved this issue?  Please help if you have solved yours.



  • Options



    I could not find a solution. We went with a different approach.


Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file