The Vertica Forum recently got a makeover! Let us know what you think by filling out this short, anonymous survey.

kafka integration with vertica in sandbox

Hi,

i am trying to integrate with kafka with vertica in sandbox as per exmaple in documenation. but i am not able ...can you any one help me on this...

 

 

Regards

Mahesh

 

Comments

  • Hi Mahesh,

        What exactly seems to be the issue?

     

    -Chris

  • i am executing follwong command on sand box(vm image)
    /opt/vertica/packages/kafka/bin/vkconfig scheduler --add --config-schema myScheduler --operator dbadmin        ---> no error
    CREATE FLEX TABLE public.kafka_tgt();     --> no error
     
    /opt/vertica/packages/kafka/bin/vkconfig topic --add --target public.kafka_tgt --rejection-table public.kafka_rej --topic messages --num-partitions 4      --> error
     
    if i execute the above statment. i am getting on error kafka_target schema doestn't exit...
  • I'm seeing two things that might be creating issue for you.

     

    First, in you scheduler, you are not defining the brokers.  You should add --brokers (server):9092

    Secondly in the topic you should add the --config-schema flag, in your case, it should be --config-schema myScheduler

     

    -Chris

  •  

    /opt/vertica/packages/kafka/bin/vkconfig scheduler --add  brokers 127.0.0.1:9092 --config-schema myScheduler --operator dbadmin......still no luck....

  • Hmmm.. What is the exact error message?

     

    Thanks,

    -Chris

  • the eror is " no such agrument to config"...

  • Ok, I think I see the problem.  First, the brokers file doesn't have the -- before it.  But since you are going local host, you can ignore it and it will assume local host as the default.  Command should be:

     

    /opt/vertica/packages/kafka/bin/vkconfig scheduler --add --config-schema myScheduler --operator dbadmin

  •  

     

    now i am getting error for this command

     

    /opt/vertica/packages/kafka/bin/vkconfig topic --add --target public.kafka_tgt --rejection-table public.kafka_rej --topic messages --num-partitions 4

     

    The error is " The schema kafka_config doesn't exits.......

  • Topic add requires the --config-schema as well.  Command should be:

     

    /opt/vertica/packages/kafka/bin/vkconfig topic --add --target public.kafka_tgt --rejection-table public.kafka_rej --topic messages --num-partitions 4  --config-schema myScheduler

     

    -Chris

  • now i am getting error for commad
     
    /opt/vertica/packages/kafka/bin/vkconfig launch --conf configFile.propertis
     
    the error is " expection in the thread "main""   java error...

  • Are you using the conf file?  if not, maybe try just a /vkconfig launch &

     

    In older versions, the launch command also needed the --config-schema  flag as well, however documentation does not indicate that.  You might want to also try /vkconfig launch --config-schema myScheduler &

     

    -Chris

  • Here is the out put..

    [[email protected] bin]$ ./vkconfig launch &
    [1] 16212
    [[email protected] bin]$ Exception in thread "Main" java.sql.SQLException: [Vertica][VJDBC](4650) ERROR: Schema "kafka_config" does not exist
    at com.vertica.util.ServerErrorData.buildException(Unknown Source)
    at com.vertica.dataengine.VQueryExecutor.executeSimpleProtocol(Unknown Source)
    at com.vertica.dataengine.VQueryExecutor.execute(Unknown Source)
    at com.vertica.jdbc.common.SStatement.executeNoParams(Unknown Source)
    at com.vertica.jdbc.common.SStatement.executeQuery(Unknown Source)
    at com.vertica.solutions.kafka.cli.CLIUtil.validateCopySyntaxes(CLIUtil.java:34)
    at com.vertica.solutions.kafka.Launcher.run(Launcher.java:69)
    at com.vertica.solutions.kafka.Launcher.main(Launcher.java:136)
    Caused by: com.vertica.support.exceptions.ErrorException: [Vertica][VJDBC](4650) ERROR: Schema "kafka_config" does not exist
    ... 8 more

    [1]+ Exit 1 ./vkconfig launch
    [[email protected] bin]$ ./vkconfig launch -- config-schema myScheduler &
    [1] 16296
    [[email protected] bin]$ No such argument

  • So the first error indicates that it hinks the schema is kafka_config

     

    And the where you issue the command with the --config-schema you placed a space between -- and config.  Try this:

     

    ./vkconfig launch --config-schema myScheduler &

     

    -chris

  • i got this message now..

    [[email protected] bin]$ ./vkconfig launch --config-schema myScheduler &
    [1] 18002

     

    after this do i need to remove broker information in this script
    COPY
    schema.target_
    table
    SOURCE
    KafkaSource
    (stream='topic1|1|1,topic2|2|2',
    brokers='host1:9092,host2:9092',duration=
    INTERVAL'timeslice',stop_
    on_
    eof=TRUE,
    eof_
    timeout=
    INTERVAL'timeslice')
    PARSER
    KafkaJSONParser
    (flatten_
    arrays=False,
    flatten_
    maps=False)
    REJECTED
    DATA
    AS
    TABLE
    schema.rejection_
    table
    TRICKLE

  • Hi,

        At this point the data with in the messages topic on kafka should be loading in microbatches into the public.kafka_tgt table in Vertica.  

     

    you can check this by going into vsql and issuing this command:

     

    Select count(*) from public.kafka_tgt;

     

    -Chris

  • dbadmin=> Select count(*) from public.kafka_tgt;
     count
    -------
         0
    (1 row)

    dbadmin=> COPY schema.target_table SOURCE KafkaSource (stream='topic1|1|1,topic2|2|2',brokers='host1:9092,host2:9092',duration=INTERVAL'timeslice',stop_on_eof=TRUE,eof_timeout=INTERVAL'timeslice') PARSER KafkaJSONPars(flatten_arrays=False,flatten_maps=False) REJECTED DATA AS TABLE schema.rejection_table TRICKLE;
    ERROR 4856:  Syntax error at or near "schema" at character 6
    LINE 1: COPY schema.target_table SOURCE KafkaSource (stream='topic1|...
                 ^
    dbadmin=>

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file

Can't find what you're looking for? Search the Vertica Documentation, Knowledge Base, or Blog for more information.