We're Moving!

The Vertica Forum is moving to a new OpenText Analytics Database (Vertica) Community.

Join us there to post discussion topics, learn about

product releases, share tips, access the blog, and much more.

Create My New Community Account Now


What is the upper limit on the number of Kafka topics for streaming data into Vertica 8.1 — Vertica Forum

What is the upper limit on the number of Kafka topics for streaming data into Vertica 8.1

Hello,

I am trying to stream messages from 30 Kafka topics simultaneously into 30 columnar tables. Each topic has one partition. This fails to stream data into Vertica.

I tried the same with 25 topics and with one 1 partition each. This seems to work but the streaming seems to be slow after multiple hours of running the Vertica Kafka scheduler.

I have tried out other scenarios such as 15 topics with 6 partitions each. This also fails to stream data into Vertica.

I tried running three separate instances of Vertica Kafka Scheduler with 12 topics each with 4 partitions. And that worked.

In a way I am unable to conclude the upper limit of the number of topics or partitions that the Vertica Kafka Scheduler supports at once.

I am using Vertica 8.1. Does anyone have any ideas about this?

Thanks,
Sandeep.

Answers

  • [Deleted User][Deleted User] Administrator

    Sandeep, I don't think there is upper limit on number of topics supported by vertica. what you mean by fails to stream data into vertica ? Can you be more specific about error that you get ?

    If you have Vertica support license , please reach out to support for assistance.

  • Hello Kamat,

    Yes, data streaming is failing into Vertica. I have a support ticket open and will further followup with the support team.

    Thank you for the help.

    Sandeep.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file