Is the Kafka "Using Streaming Data with Vertica" documentation page still valid ?
Hi,
I'm trying to test the Kafka integration in Vertica 8.1, but the documentation tutorial does not work.
I'm stuck with the error No Source exists with name 'conversion' in cluster 'StreamCluster1' at the create a source step. It might be also nice to see the content of the various config-schema files, and to explain why we create a cluster two times.
I'm probably missing something obvious, I'm a beginner with Vertica, but I feel like the documentation is maybe not updated.
0
Comments
I have not tested it but it is probably because while creating cluster it has --cluster kafka and later on it is using StreamCluster1.
Could you please give it a try by keeping same names? If it succeeds then please file a documentation bug with vertica support.
It would help if Vertica would give an valid end-to-end example, especially on Vertica 8.1, as they completely changed vkconfig, and broke backward compatibility.
Nowhere in the documentation for Kafka Integration with Vertica (using vkconfig) does it mention how to specify which Kafka topics to pull into Vertica. It looks like COPY with KafkaSource is the only reliable way to go (unless we use Spark).
https://my.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/KafkaIntegrationGuide/Using_Kafka_with_HP_Vertica.htm?Highlight=vkconfig topic
Why nobody answares to this question. I'm stuk like antoinep!!!
I would post this question on StackOverflow, #vertica. Sometimes I get better responses there.
Antoine et al.
The documentation is inaccurate (JIRA created), the parameter source_name has to be a Kafka topic. The Vkconfig CLI will verify that the topic (source_name) exists in the previously defined cluster as reflected in the error you got.
No Source exists with name 'conversion' in cluster 'StreamCluster1'
means that we couldn't find the topic 'conversion' in the Kafka cluster.