I am trying to integrate vertica and kafka. I have a json file. Kafka scheduler is reading data from file using nodeJS and sending it to vertica. I want to store data to different tables in vertica, Is it possible to do?
Thanks in advance.
Yes, you can load different tables using the Vertica-Kafka integration.
First you create your target table(s) using regular sql
Then after creating a scheduler, you do vkconfig topic --add ...
To define a topic->table mapping that references the table you just created.
For your JSON message to load into two different tables we recommend to land the message in a FlexTable, then insert select out of that into the final destination(s).
Thanks for the response. Please correct me if I am wrong.
1. Create flex table and put all data in the table.
2. From this flex table, put data into different tables
I need one more help. When I am trying to integrate kafka vertica. Sometimes messages are stored in vertica and sometimes not.
This COPY command is working properly.
COPY public.kafka_tgt SOURCE KafkaSource(stream='test|0|23', brokers='localhost:9092', duration=interval '9884 milliseconds', stop_on_eof=true, executionparallelism=1 ) PARSER KafkaJSONParser( ) REJECTED DATA AS TABLE public.kafka_rej TRICKLE;
Can you please tell me where to check for the problem? Do we have any blog or tutorial from where I can learn more about it.