The Vertica Forum recently got a makeover! Let us know what you think by filling out this short, anonymous survey.

store data from one file to multiple table in vertica kafka

HI,

 

I am trying to integrate vertica and kafka. I have a json file. Kafka scheduler is reading data from file using nodeJS and sending it to vertica. I want to store data to different tables in vertica, Is it possible to do?

 

Thanks in advance.

 

Comments

  • Hi,

     

    Yes, you can load different tables using the Vertica-Kafka integration. 

     

    First you create your target table(s) using regular sql

    Then after creating a scheduler, you do vkconfig topic --add ...

    To define a topic->table mapping that references the table you just created.

     

    For your JSON message to load into two different tables we recommend to land the message in a FlexTable, then insert select out of that into the final destination(s).

     

    Thanks,

    Mark

  •  

    HI Mark,

     

    Thanks for the response. Please correct me if I am wrong.

    1. Create flex table and put all data in the table.

    2. From this flex table, put data into different tables

     

    I need one more help. When I am trying to integrate kafka vertica. Sometimes messages are stored in vertica and sometimes not.

     

     

    This COPY command is working properly.

    COPY public.kafka_tgt SOURCE KafkaSource(stream='test|0|23', brokers='localhost:9092', duration=interval '9884 milliseconds', stop_on_eof=true, executionparallelism=1 ) PARSER KafkaJSONParser( ) REJECTED DATA AS TABLE public.kafka_rej TRICKLE;

     

    Can you please tell me where to check for the problem? Do we have any blog or tutorial from where I can learn more about it.

     

     

     

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file

Can't find what you're looking for? Search the Vertica Documentation, Knowledge Base, or Blog for more information.