What is the upper limit on the number of Kafka topics for streaming data into Vertica 8.1

ksandeep

Hello,

I am trying to stream messages from 30 Kafka topics simultaneously into 30 columnar tables. Each topic has one partition. This fails to stream data into Vertica.

I tried the same with 25 topics and with one 1 partition each. This seems to work but the streaming seems to be slow after multiple hours of running the Vertica Kafka scheduler.

I have tried out other scenarios such as 15 topics with 6 partitions each. This also fails to stream data into Vertica.

I tried running three separate instances of Vertica Kafka Scheduler with 12 topics each with 4 partitions. And that worked.

In a way I am unable to conclude the upper limit of the number of topics or partitions that the Vertica Kafka Scheduler supports at once.

I am using Vertica 8.1. Does anyone have any ideas about this?

Thanks,
Sandeep.

[Deleted User]

Sandeep, I don't think there is upper limit on number of topics supported by vertica. what you mean by fails to stream data into vertica ? Can you be more specific about error that you get ?

If you have Vertica support license , please reach out to support for assistance.

ksandeep

Hello Kamat,

Yes, data streaming is failing into Vertica. I have a support ticket open and will further followup with the support team.

Thank you for the help.

Sandeep.