What is the upper limit on the number of Kafka topics for streaming data into Vertica 8.1

ksandeep

Hello,

I am trying to stream messages from 30 Kafka topics simultaneously into 30 columnar tables. Each topic has one partition. This fails to stream data into Vertica.

I tried the same with 25 topics and with one 1 partition each. This seems to work but the streaming seems to be slow after multiple hours of running the Vertica Kafka scheduler.

I have tried out other scenarios such as 15 topics with 6 partitions each. This also fails to stream data into Vertica.

I tried running three separate instances of Vertica Kafka Scheduler with 12 topics each with 4 partitions. And that worked.

In a way I am unable to conclude the upper limit of the number of topics or partitions that the Vertica Kafka Scheduler supports at once.

I am using Vertica 8.1. Does anyone have any ideas about this?

Thanks,
Sandeep.

rthota

Hello Sandeep,

I have tried with 34 Kafka topics with one partition each, and it works pretty well.

Thanks,
Raj