store data from one file to multiple table in vertica kafka



I am trying to integrate vertica and kafka. I have a json file. Kafka scheduler is reading data from file using nodeJS and sending it to vertica. I want to store data to different tables in vertica, Is it possible to do?


Thanks in advance.



  • Options



    Yes, you can load different tables using the Vertica-Kafka integration. 


    First you create your target table(s) using regular sql

    Then after creating a scheduler, you do vkconfig topic --add ...

    To define a topic->table mapping that references the table you just created.


    For your JSON message to load into two different tables we recommend to land the message in a FlexTable, then insert select out of that into the final destination(s).




  • Options


    HI Mark,


    Thanks for the response. Please correct me if I am wrong.

    1. Create flex table and put all data in the table.

    2. From this flex table, put data into different tables


    I need one more help. When I am trying to integrate kafka vertica. Sometimes messages are stored in vertica and sometimes not.



    This COPY command is working properly.

    COPY public.kafka_tgt SOURCE KafkaSource(stream='test|0|23', brokers='localhost:9092', duration=interval '9884 milliseconds', stop_on_eof=true, executionparallelism=1 ) PARSER KafkaJSONParser( ) REJECTED DATA AS TABLE public.kafka_rej TRICKLE;


    Can you please tell me where to check for the problem? Do we have any blog or tutorial from where I can learn more about it.




Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file