Not able to compute keys vertica kafka

phmandawaphmandawa Employee, Registered User

 

Hi,

 

I am trying to integrate vertica kafka. I am storing data in flex table. Vertica is not able to parser data.

 

 

Select * from table. I get

 

1 | {"classname":"global", "name": "Everest", "type":"mountain", "height":29029, "hike_safety": 34.1}
2 | {"classname":"global", "name": "Mt St Helens", "type":"volcano", "height":29029, "hike_safety": 15.4}
3 | {"classname":"application", "name": "Denali", "type":"mountain", "height":17000, "hike_safety": 12.2}
4 | {"classname":"application", "name": "Kilimanjaro", "type":"mountain", "height":14000 }
5 | {"classname":"global", "name": "Everest", "type":"mountain", "height":29029, "hike_safety": 34.1}
250001 | {"classname":"global", "name": "Everest", "type":"mountain", "height":29029, "hike_safety": 34.1}
250002 | {"classname":"global", "name": "Mt St Helens", "type":"volcano", "height":29029, "hike_safety": 15.4}

 

 

 

But when I try to compute key using " select compute_flextable_keys('table');" I am getting this error.

 

ERROR 5861: Error calling processPartition() in User Function MapKeysInfo at [src/Dict.cpp:639], error code: 0, message: Exception while processing partition row: [1] on map keys information retrieval: [Tried to retrieve Flex Table data from a cell not containing an Flex Table map.]

Comments

  • markclyonsmarkclyons Administrator, Moderator, Employee, Registered User, VerticaExpert, CABuser

    Hi,

     

    After a quick look by one of the R&D team members we think you may be using the base Kafka Parser and not the Kafka JSON Parser.

     

    Check out this section of the documentation:

    https://my.vertica.com/docs/7.2.x/HTML/index.htm#Authoring/KafkaIntegrationGuide/UsingCOPYwithKafka.htm%3FTocPath%3DIntegrating%2520with%2520Apache%2520Kafka%7C_____5

     

    Thanks,

    Mark

  • phmandawaphmandawa Employee, Registered User

     

    Hi Mark,

     

    Thanks for your response.

    With COPY statment, vertica is parsing the json input correctly. How to do it with scheduler?

    I am using the following command to assign the json parser to scheduler.

     

    /opt/vertica/packages/kafka/bin/vkconfig topic --add --target data_tgt --rejection-table data_rej --topic data --num-partitions 1 --config-schema data  --parser KafkaJSONParser

  • MoonvertMoonvert Employee, Registered User

    Hi phmandawa,

     

    I am having exactly same problem as yours.

    My vkconfig topic command is same as yours, but it seems it does not parsing json formatted message to the vertica flex table.

     

    Another thing that I noticed is everytime I edit topic with --parser KafkaJSONParser option, it does not reflect in kafka_targets table. Example below shows KafkaParser all the time. Shouldn't it be changed to KafkaJSONParser?  I even tried to update column value manually to KafkaJSONParser, Kafka does not want to parse as JSON format.

     

    target_schema | target_table | enabled | parser | rejection_table | parser_parameters | load_method | target_columns
    ---------------+--------------+---------+-------------+---------------------+-------------------+-------------+----------------
    flextest1 | kafka | t | KafkaParser | flextest1.kafka_rej | | TRICKLE |

     

    Have you solved this issue?  Please help if you have solved yours.

     

    Thanks,

  • phmandawaphmandawa Employee, Registered User

    HI,

     

    I could not find a solution. We went with a different approach.

     

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file