integration with apache kafka error No Source exists
aleksss55
Community Edition User ✭
Hello,
Could you please help me with the following issue?
I have an installed vertica 9.3 on one host and installed apache kafka cluster on 3 other hosts. I want integration vertica with apache kafka (2.4.0). I try to make source: vkconfig source --create --cluster kafka_weblog --source mytest --partitions 1 --conf /opt/vertica/packages/kafka/config/my.conf
I get error "No Source exists with name 'mytest'", although this topic exists
I check with kafkacat and I see that there aren't data about topics:
kafkacat -L -b 192.168.0.8:9092
Metadata for all topics (from broker 3: 192.168.0.8:9092/3):3 brokers:
broker 2 at 192.168.0.7:9092
broker 3 at 192.168.0.8:9092
broker 1 at 192.168.0.9:9092
0 topics:
Why is that?
0
Answers
Please complete step 3 and step 4 from the below documentation in kafka cluster and try to create source in vertica cluster
https://kafka.apache.org/quickstart
I followed quickstart step 3 and step 4 (I created a new topic mytest2 and I wrote some messages to it), but unfortunately, this did not help me, the same error appears "No Source exists with name 'mytest2'"
Please send the output of following commands-
1)bin/kafka-topics.sh --list --zookeeper 192.168.0.8:2181 (default listen port of zookeeper is 2181)
2) kafkacat -L -b 192.168.0.8:9092 (just to verify again)
@aleksss55 Along with what @sahil_kumar requested, could you please share kafka server log and vkafka-sched.log?
@aleksss55 please open support ticket and mention Sruthi A. & Sahil Kumar. One of them will help you resolve this issue ASAP
Thanks a lot, the problem isn't actually
Sounds like an issue on the Kafka side with the way your topic was created/populated as no topics are visible.
Could you try running something like this from a Vertica node:
echo 'this is a test message' | /opt/vertica/packages/kafka/bin/kafkacat -P -t my_new_topic -b 192.168.0.8:9092
If some errors, you'll need to fix things on the Kafka side.
If no errors, verify you can consume that one message on the Vertica node by running
/opt/vertica/packages/kafka/bin/kafkacat -C -t my_new_topic -b 192.168.0.8:9092
FYI: More kafkacat producer examples can be found here:
https://docs.confluent.io/current/app-development/kafkacat-usage.html