Creating a Kafka source for a Vertica v8.1 scheduler

Green89

Hello,

I'm on Linux system and I'm using a Vertica Analytic Database v8.1.0-2. I'm trying to implement a Vertica scheduler to load data from a kafka topic into a Vertica table.

I successfully created a scheduler and a kafka cluster by launching the following two commands:
/opt/vertica/packages/kafka/bin/vkconfig scheduler --create --config-schema KafkaScheduler --operator dbadmin --frame-duration '00:00:30'

/opt/vertica/packages/kafka/bin/vkconfig cluster --create --config-schema KafkaScheduler --cluster kafka --hosts xxx:9092,xxx:9092,xxx:9092

Note: The KafkaScheduler schema has well been created with the appropriate set of tables.

However, when I'm getting to create a kafka source for the schduler by launching the following command: /opt/vertica/packages/kafka/bin/vkconfig source --create --config-schema KafkaScheduler --cluster kafka --source myKafkaTopicName --partitions 10

I'm guetting the following error:

Exception in thread "main" com.vertica.solutions.kafka.exception.ConfigurationException: ERROR: [[Vertica]VJDBC ERROR: Error calling processPartition() in User Function KafkaListTopics at [src/KafkaUtil.cpp:178], error code: 0, message: Error getting metadata: [Local: Broker transport failure]]
at com.vertica.solutions.kafka.model.StreamSource.validateConfiguration(StreamSource.java:171)
at com.vertica.solutions.kafka.model.StreamSource.setFromMapAndValidate(StreamSource.java:127)
at com.vertica.solutions.kafka.model.StreamModel.(StreamModel.java:89)
at com.vertica.solutions.kafka.model.StreamSource.(StreamSource.java:36)
at com.vertica.solutions.kafka.cli.SourceCLI.getNewModel(SourceCLI.java:48)
at com.vertica.solutions.kafka.cli.SourceCLI.getNewModel(SourceCLI.java:14)
at com.vertica.solutions.kafka.cli.CLI.run(CLI.java:56)
at com.vertica.solutions.kafka.cli.CLI._main(CLI.java:132)
at com.vertica.solutions.kafka.cli.SourceCLI.main(SourceCLI.java:20)
Caused by: java.sql.SQLNonTransientException: [Vertica]VJDBC ERROR: Error calling processPartition() in User Function KafkaListTopics at [src/KafkaUtil.cpp:178], error code: 0, message: Error getting metadata: [Local: Broker transport failure]
at com.vertica.util.ServerErrorData.buildException(Unknown Source)
at com.vertica.dataengine.VResultSet.fetchChunk(Unknown Source)
at com.vertica.dataengine.VResultSet.initialize(Unknown Source)
at com.vertica.dataengine.VQueryExecutor.readExecuteResponse(Unknown Source)
at com.vertica.dataengine.VQueryExecutor.handleExecuteResponse(Unknown Source)
at com.vertica.dataengine.VQueryExecutor.execute(Unknown Source)
at com.vertica.jdbc.common.SPreparedStatement.executeWithParams(Unknown Source)
at com.vertica.jdbc.common.SPreparedStatement.executeQuery(Unknown Source)
at com.vertica.solutions.kafka.model.StreamSource.validateConfiguration(StreamSource.java:150)
... 8 more
Caused by: com.vertica.support.exceptions.NonTransientException: [Vertica]VJDBC ERROR: Error calling processPartition() in User Function KafkaListTopics at [src/KafkaUtil.cpp:178], error code: 0, message: Error getting metadata: [Local: Broker transport failure]
... 17 more

I checked the topic name and number of partitions they are same as corresponding topic in kafka cluster.

Any help would be appreciated.

avi120

Hi. Were you able to resolve this problem. I have the same exact error. Thanks

SergeB

Can you run KafkaCheckBrokers on xxx:9092 (one of the values you passed when running vkconfig --create --cluster ?
Then check the hostnames returned by that meta function as these are the ones the scheduler will use.