integration with apache kafka error No Source exists

aleksss55aleksss55 Community Edition User

Hello,
Could you please help me with the following issue?
I have an installed vertica 9.3 on one host and installed apache kafka cluster on 3 other hosts. I want integration vertica with apache kafka (2.4.0). I try to make source: vkconfig source --create --cluster kafka_weblog --source mytest --partitions 1 --conf /opt/vertica/packages/kafka/config/my.conf

I get error "No Source exists with name 'mytest'", although this topic exists
I check with kafkacat and I see that there aren't data about topics:
kafkacat -L -b 192.168.0.8:9092
Metadata for all topics (from broker 3: 192.168.0.8:9092/3):3 brokers:
broker 2 at 192.168.0.7:9092
broker 3 at 192.168.0.8:9092
broker 1 at 192.168.0.9:9092
0 topics:

Why is that?

Tagged:

Answers

  • SruthiASruthiA Vertica Employee Administrator

    Please complete step 3 and step 4 from the below documentation in kafka cluster and try to create source in vertica cluster

    https://kafka.apache.org/quickstart

  • aleksss55aleksss55 Community Edition User

    I followed quickstart step 3 and step 4 (I created a new topic mytest2 and I wrote some messages to it), but unfortunately, this did not help me, the same error appears "No Source exists with name 'mytest2'"

  • sahil_kumarsahil_kumar Vertica Employee Employee

    Please send the output of following commands-
    1)bin/kafka-topics.sh --list --zookeeper 192.168.0.8:2181 (default listen port of zookeeper is 2181)
    2) kafkacat -L -b 192.168.0.8:9092 (just to verify again)

  • SruthiASruthiA Vertica Employee Administrator

    @aleksss55 Along with what @sahil_kumar requested, could you please share kafka server log and vkafka-sched.log?

  • [Deleted User][Deleted User] Administrator

    @aleksss55 please open support ticket and mention Sruthi A. & Sahil Kumar. One of them will help you resolve this issue ASAP

  • aleksss55aleksss55 Community Edition User

    Thanks a lot, the problem isn't actually

  • SergeBSergeB - Select Field - Employee

    Sounds like an issue on the Kafka side with the way your topic was created/populated as no topics are visible.
    Could you try running something like this from a Vertica node:
    echo 'this is a test message' | /opt/vertica/packages/kafka/bin/kafkacat -P -t my_new_topic -b 192.168.0.8:9092
    If some errors, you'll need to fix things on the Kafka side.
    If no errors, verify you can consume that one message on the Vertica node by running
    /opt/vertica/packages/kafka/bin/kafkacat -C -t my_new_topic -b 192.168.0.8:9092
    FYI: More kafkacat producer examples can be found here:
    https://docs.confluent.io/current/app-development/kafkacat-usage.html

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file