The Vertica Forum recently got a makeover! Let us know what you think by filling out this short, anonymous survey.
Please take this survey to help us learn more about how you use third party tools. Your input is greatly appreciated!
What is the upper limit on the number of Kafka topics for streaming data into Vertica 8.1
Hello,
I am trying to stream messages from 30 Kafka topics simultaneously into 30 columnar tables. Each topic has one partition. This fails to stream data into Vertica.
I tried the same with 25 topics and with one 1 partition each. This seems to work but the streaming seems to be slow after multiple hours of running the Vertica Kafka scheduler.
I have tried out other scenarios such as 15 topics with 6 partitions each. This also fails to stream data into Vertica.
I tried running three separate instances of Vertica Kafka Scheduler with 12 topics each with 4 partitions. And that worked.
In a way I am unable to conclude the upper limit of the number of topics or partitions that the Vertica Kafka Scheduler supports at once.
I am using Vertica 8.1. Does anyone have any ideas about this?
Thanks,
Sandeep.
I am trying to stream messages from 30 Kafka topics simultaneously into 30 columnar tables. Each topic has one partition. This fails to stream data into Vertica.
I tried the same with 25 topics and with one 1 partition each. This seems to work but the streaming seems to be slow after multiple hours of running the Vertica Kafka scheduler.
I have tried out other scenarios such as 15 topics with 6 partitions each. This also fails to stream data into Vertica.
I tried running three separate instances of Vertica Kafka Scheduler with 12 topics each with 4 partitions. And that worked.
In a way I am unable to conclude the upper limit of the number of topics or partitions that the Vertica Kafka Scheduler supports at once.
I am using Vertica 8.1. Does anyone have any ideas about this?
Thanks,
Sandeep.
0
Answers
Hello Sandeep,
I have tried with 34 Kafka topics with one partition each, and it works pretty well.
Thanks,
Raj