What is the upper limit on the number of Kafka topics for streaming data into Vertica 8.1
Hello,
I am trying to stream messages from 30 Kafka topics simultaneously into 30 columnar tables. Each topic has one partition. This fails to stream data into Vertica.
I tried the same with 25 topics and with one 1 partition each. This seems to work but the streaming seems to be slow after multiple hours of running the Vertica Kafka scheduler.
I have tried out other scenarios such as 15 topics with 6 partitions each. This also fails to stream data into Vertica.
I tried running three separate instances of Vertica Kafka Scheduler with 12 topics each with 4 partitions. And that worked.
In a way I am unable to conclude the upper limit of the number of topics or partitions that the Vertica Kafka Scheduler supports at once.
I am using Vertica 8.1. Does anyone have any ideas about this?
Thanks,
Sandeep.
I am trying to stream messages from 30 Kafka topics simultaneously into 30 columnar tables. Each topic has one partition. This fails to stream data into Vertica.
I tried the same with 25 topics and with one 1 partition each. This seems to work but the streaming seems to be slow after multiple hours of running the Vertica Kafka scheduler.
I have tried out other scenarios such as 15 topics with 6 partitions each. This also fails to stream data into Vertica.
I tried running three separate instances of Vertica Kafka Scheduler with 12 topics each with 4 partitions. And that worked.
In a way I am unable to conclude the upper limit of the number of topics or partitions that the Vertica Kafka Scheduler supports at once.
I am using Vertica 8.1. Does anyone have any ideas about this?
Thanks,
Sandeep.
0
Answers
Sandeep, I don't think there is upper limit on number of topics supported by vertica. what you mean by fails to stream data into vertica ? Can you be more specific about error that you get ?
If you have Vertica support license , please reach out to support for assistance.
Hello Kamat,
Yes, data streaming is failing into Vertica. I have a support ticket open and will further followup with the support team.
Thank you for the help.
Sandeep.