Please take this survey to help us learn more about how you use third party tools. Your input is greatly appreciated!

integration with apache kafka error No Source exists

Hello,
Could you please help me with the following issue?
I have an installed vertica 9.3 on one host and installed apache kafka cluster on 3 other hosts. I want integration vertica with apache kafka (2.4.0). I try to make source: vkconfig source --create --cluster kafka_weblog --source mytest --partitions 1 --conf /opt/vertica/packages/kafka/config/my.conf

I get error "No Source exists with name 'mytest'", although this topic exists
I check with kafkacat and I see that there aren't data about topics:
kafkacat -L -b 192.168.0.8:9092
Metadata for all topics (from broker 3: 192.168.0.8:9092/3):3 brokers:
broker 2 at 192.168.0.7:9092
broker 3 at 192.168.0.8:9092
broker 1 at 192.168.0.9:9092
0 topics:

Why is that?

Tagged:

Answers

  • SruthiASruthiA Employee

    Please complete step 3 and step 4 from the below documentation in kafka cluster and try to create source in vertica cluster

    https://kafka.apache.org/quickstart

  • I followed quickstart step 3 and step 4 (I created a new topic mytest2 and I wrote some messages to it), but unfortunately, this did not help me, the same error appears "No Source exists with name 'mytest2'"

  • Please send the output of following commands-
    1)bin/kafka-topics.sh --list --zookeeper 192.168.0.8:2181 (default listen port of zookeeper is 2181)
    2) kafkacat -L -b 192.168.0.8:9092 (just to verify again)

  • SruthiASruthiA Employee

    @aleksss55 Along with what @sahil_kumar requested, could you please share kafka server log and vkafka-sched.log?

  • skamatskamat Administrator

    @aleksss55 please open support ticket and mention Sruthi A. & Sahil Kumar. One of them will help you resolve this issue ASAP

  • Thanks a lot, the problem isn't actually

  • SergeBSergeB Employee

    Sounds like an issue on the Kafka side with the way your topic was created/populated as no topics are visible.
    Could you try running something like this from a Vertica node:
    echo 'this is a test message' | /opt/vertica/packages/kafka/bin/kafkacat -P -t my_new_topic -b 192.168.0.8:9092
    If some errors, you'll need to fix things on the Kafka side.
    If no errors, verify you can consume that one message on the Vertica node by running
    /opt/vertica/packages/kafka/bin/kafkacat -C -t my_new_topic -b 192.168.0.8:9092
    FYI: More kafkacat producer examples can be found here:
    https://docs.confluent.io/current/app-development/kafkacat-usage.html

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file

Can't find what you're looking for? Search the Vertica Documentation, Knowledge Base, or Blog for more information.