store data from one file to multiple table in vertica kafka

phmandawa

HI,

 

I am trying to integrate vertica and kafka. I have a json file. Kafka scheduler is reading data from file using nodeJS and sending it to vertica. I want to store data to different tables in vertica, Is it possible to do?

 

Thanks in advance.

 

[Deleted User]

Hi,

 

Yes, you can load different tables using the Vertica-Kafka integration. 

 

First you create your target table(s) using regular sql

Then after creating a scheduler, you do vkconfig topic --add ...

To define a topic->table mapping that references the table you just created.

 

For your JSON message to load into two different tables we recommend to land the message in a FlexTable, then insert select out of that into the final destination(s).

 

Thanks,

Mark

phmandawa

 

HI Mark,

 

Thanks for the response. Please correct me if I am wrong.

1. Create flex table and put all data in the table.

2. From this flex table, put data into different tables

 

I need one more help. When I am trying to integrate kafka vertica. Sometimes messages are stored in vertica and sometimes not.

 

 

This COPY command is working properly.

COPY public.kafka_tgt SOURCE KafkaSource(stream='test|0|23', brokers='localhost:9092', duration=interval '9884 milliseconds', stop_on_eof=true, executionparallelism=1 ) PARSER KafkaJSONParser( ) REJECTED DATA AS TABLE public.kafka_rej TRICKLE;

 

Can you please tell me where to check for the problem? Do we have any blog or tutorial from where I can learn more about it.