We're Moving!

The Vertica Forum is moving to a new OpenText Analytics Database (Vertica) Community.

Join us there to post discussion topics, learn about

product releases, share tips, access the blog, and much more.

Create My New Community Account Now


Why do I get "KAFKASOURCE() does not exist" Error, when loading from Kafka — Vertica Forum

Why do I get "KAFKASOURCE() does not exist" Error, when loading from Kafka

joergschaberjoergschaber Vertica Customer ✭
in General Discussion

Hi,

I have set up Kafka server with ** no permission** . When I consume data from a topic using a .NET client lib and configurw the consumer with

_config = new ConsumerConfig
                      {
                          BootstrapServers = "<server:port>",
                          GroupId = "<groupName>",
                          AutoOffsetReset = AutoOffsetReset.Earliest
                      };

I can readily consumer messages.

However, when I try to load messages from a topic (2 paritions) into Vertica using COPY with

COPY <schema.table> SOURCE KAFKASOURCE(
                stream='<topics>|0|-2,<topics>|1|-2', 
                brokers = '<server:port>', duration = interval '10000 milliseconds', 
                group_id = '<groupName>', executionparallelism = 2) 
                PARSER KafkaJSONParser(flatten_maps=false, flatten_arrays=false,reject_on_duplicate=false)
                NO COMMIT;

I get an error Function KAFKASOURCE() does not exist, or permission is denied for KAFKASOURCE()'.
I ran

GRANT USAGE ON library kafkalib to <user>;
GRANT EXECUTE ON source KafkaSource() to <user>;;

When I use the dbadmin, the command does not return/erminate.
Even when I close the corresponding session, the session still appears in the session table.

Unsing Vertica 10.0, Any idea?

Answers

  • SruthiASruthiA Administrator

    Could you please try running copy from vertica node? does it still throw same error? is kafka package installed in vertica cluster?

  • joergschaberjoergschaber Vertica Customer ✭

    Good question, I thought that is standard. How do I check?

  • SruthiASruthiA Administrator

    select * from user_libraries WHERE lib_name = 'KafkaLib';

  • joergschaberjoergschaber Vertica Customer ✭

    KafkaLib is installed.

  • joergschaberjoergschaber Vertica Customer ✭

    I ran the COPY command also from the vertica node as dbadmin. Same here, the command does not return and the session that was openeds cannot be deleted. Is there a more detialed log I can look into?

  • SruthiASruthiA Administrator

    so looks like it s hanging when run via vsql. vertica.log should provide more information

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Format
SpoilerCodeQuote
Heading 2Heading 1
Emoji
:):D:(;):/:o:s:p:'(:|B):#:*<3o:)>:)
Url
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file
Home • General DiscussionComment As ...