Why do I get "KAFKASOURCE() does not exist" Error, when loading from Kafka

joergschaber

Hi,

I have set up Kafka server with ** no permission** . When I consume data from a topic using a .NET client lib and configurw the consumer with

_config = new ConsumerConfig
                      {
                          BootstrapServers = "<server:port>",
                          GroupId = "<groupName>",
                          AutoOffsetReset = AutoOffsetReset.Earliest
                      };

I can readily consumer messages.

However, when I try to load messages from a topic (2 paritions) into Vertica using COPY with

COPY <schema.table> SOURCE KAFKASOURCE(
                stream='<topics>|0|-2,<topics>|1|-2', 
                brokers = '<server:port>', duration = interval '10000 milliseconds', 
                group_id = '<groupName>', executionparallelism = 2) 
                PARSER KafkaJSONParser(flatten_maps=false, flatten_arrays=false,reject_on_duplicate=false)
                NO COMMIT;

I get an error Function KAFKASOURCE() does not exist, or permission is denied for KAFKASOURCE()'.
I ran

GRANT USAGE ON library kafkalib to <user>;
GRANT EXECUTE ON source KafkaSource() to <user>;;

When I use the dbadmin, the command does not return/erminate.
Even when I close the corresponding session, the session still appears in the session table.

Unsing Vertica 10.0, Any idea?

SruthiA

Could you please try running copy from vertica node? does it still throw same error? is kafka package installed in vertica cluster?

joergschaber

Good question, I thought that is standard. How do I check?

SruthiA

select * from user_libraries WHERE lib_name = 'KafkaLib';

joergschaber

KafkaLib is installed.

joergschaber

I ran the COPY command also from the vertica node as dbadmin. Same here, the command does not return and the session that was openeds cannot be deleted. Is there a more detialed log I can look into?

SruthiA

so looks like it s hanging when run via vsql. vertica.log should provide more information