Not able to compute keys vertica kafka

phmandawa

 

Hi,

 

I am trying to integrate vertica kafka. I am storing data in flex table. Vertica is not able to parser data.

 

 

Select * from table. I get

 

1 | {"classname":"global", "name": "Everest", "type":"mountain", "height":29029, "hike_safety": 34.1}
2 | {"classname":"global", "name": "Mt St Helens", "type":"volcano", "height":29029, "hike_safety": 15.4}
3 | {"classname":"application", "name": "Denali", "type":"mountain", "height":17000, "hike_safety": 12.2}
4 | {"classname":"application", "name": "Kilimanjaro", "type":"mountain", "height":14000 }
5 | {"classname":"global", "name": "Everest", "type":"mountain", "height":29029, "hike_safety": 34.1}
250001 | {"classname":"global", "name": "Everest", "type":"mountain", "height":29029, "hike_safety": 34.1}
250002 | {"classname":"global", "name": "Mt St Helens", "type":"volcano", "height":29029, "hike_safety": 15.4}

 

 

 

But when I try to compute key using " select compute_flextable_keys('table');" I am getting this error.

 

ERROR 5861: Error calling processPartition() in User Function MapKeysInfo at [src/Dict.cpp:639], error code: 0, message: Exception while processing partition row: [1] on map keys information retrieval: [Tried to retrieve Flex Table data from a cell not containing an Flex Table map.]

[Deleted User]

Hi,

 

After a quick look by one of the R&D team members we think you may be using the base Kafka Parser and not the Kafka JSON Parser.

 

Check out this section of the documentation:

https://my.vertica.com/docs/7.2.x/HTML/index.htm#Authoring/KafkaIntegrationGuide/UsingCOPYwithKafka.htm%3FTocPath%3DIntegrating%2520with%2520Apache%2520Kafka%7C_____5

 

Thanks,

Mark

phmandawa

 

Hi Mark,

 

Thanks for your response.

With COPY statment, vertica is parsing the json input correctly. How to do it with scheduler?

I am using the following command to assign the json parser to scheduler.

 

/opt/vertica/packages/kafka/bin/vkconfig topic --add --target data_tgt --rejection-table data_rej --topic data --num-partitions 1 --config-schema data  --parser KafkaJSONParser

Moonvert

Hi phmandawa,

 

I am having exactly same problem as yours.

My vkconfig topic command is same as yours, but it seems it does not parsing json formatted message to the vertica flex table.

 

Another thing that I noticed is everytime I edit topic with --parser KafkaJSONParser option, it does not reflect in kafka_targets table. Example below shows KafkaParser all the time. Shouldn't it be changed to KafkaJSONParser?  I even tried to update column value manually to KafkaJSONParser, Kafka does not want to parse as JSON format.

 

target_schema | target_table | enabled | parser | rejection_table | parser_parameters | load_method | target_columns
---------------+--------------+---------+-------------+---------------------+-------------------+-------------+----------------
flextest1 | kafka | t | KafkaParser | flextest1.kafka_rej | | TRICKLE |

 

Have you solved this issue?  Please help if you have solved yours.

 

Thanks,

phmandawa

HI,

 

I could not find a solution. We went with a different approach.