We're Moving!

The Vertica Forum is moving to a new OpenText Analytics Database (Vertica) Community.

Join us there to post discussion topics, learn about

product releases, share tips, access the blog, and much more.

Create My New Community Account Now


store data from one file to multiple table in vertica kafka — Vertica Forum

store data from one file to multiple table in vertica kafka

HI,

 

I am trying to integrate vertica and kafka. I have a json file. Kafka scheduler is reading data from file using nodeJS and sending it to vertica. I want to store data to different tables in vertica, Is it possible to do?

 

Thanks in advance.

 

Comments

  • Hi,

     

    Yes, you can load different tables using the Vertica-Kafka integration. 

     

    First you create your target table(s) using regular sql

    Then after creating a scheduler, you do vkconfig topic --add ...

    To define a topic->table mapping that references the table you just created.

     

    For your JSON message to load into two different tables we recommend to land the message in a FlexTable, then insert select out of that into the final destination(s).

     

    Thanks,

    Mark

  •  

    HI Mark,

     

    Thanks for the response. Please correct me if I am wrong.

    1. Create flex table and put all data in the table.

    2. From this flex table, put data into different tables

     

    I need one more help. When I am trying to integrate kafka vertica. Sometimes messages are stored in vertica and sometimes not.

     

     

    This COPY command is working properly.

    COPY public.kafka_tgt SOURCE KafkaSource(stream='test|0|23', brokers='localhost:9092', duration=interval '9884 milliseconds', stop_on_eof=true, executionparallelism=1 ) PARSER KafkaJSONParser( ) REJECTED DATA AS TABLE public.kafka_rej TRICKLE;

     

    Can you please tell me where to check for the problem? Do we have any blog or tutorial from where I can learn more about it.

     

     

     

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file