kafka integration with vertica in sandbox

Mahesh_Benne

Hi,

i am trying to integrate with kafka with vertica in sandbox as per exmaple in documenation. but i am not able ...can you any one help me on this...

 

 

Regards

Mahesh

 

Chris_Daly_HPE

Hi Mahesh,

    What exactly seems to be the issue?

 

-Chris

Mahesh_Benne

i am executing follwong command on sand box(vm image)
/opt/vertica/packages/kafka/bin/vkconfig scheduler --add --config-schema myScheduler --operator dbadmin        ---> no error
CREATE FLEX TABLE public.kafka_tgt();     --> no error
 
/opt/vertica/packages/kafka/bin/vkconfig topic --add --target public.kafka_tgt --rejection-table public.kafka_rej --topic messages --num-partitions 4      --> error
 
if i execute the above statment. i am getting on error kafka_target schema doestn't exit...

Chris_Daly_HPE

I'm seeing two things that might be creating issue for you.

 

First, in you scheduler, you are not defining the brokers.  You should add --brokers (server):9092

Secondly in the topic you should add the --config-schema flag, in your case, it should be --config-schema myScheduler

 

-Chris

Mahesh_Benne

 

/opt/vertica/packages/kafka/bin/vkconfig scheduler --add  brokers 127.0.0.1:9092 --config-schema myScheduler --operator dbadmin......still no luck....

Chris_Daly_HPE

Hmmm.. What is the exact error message?

 

Thanks,

-Chris

Mahesh_Benne

the eror is " no such agrument to config"...

Chris_Daly_HPE

Ok, I think I see the problem.  First, the brokers file doesn't have the -- before it.  But since you are going local host, you can ignore it and it will assume local host as the default.  Command should be:

 

/opt/vertica/packages/kafka/bin/vkconfig scheduler --add --config-schema myScheduler --operator dbadmin

Mahesh_Benne

 

 

now i am getting error for this command

 

/opt/vertica/packages/kafka/bin/vkconfig topic --add --target public.kafka_tgt --rejection-table public.kafka_rej --topic messages --num-partitions 4

 

The error is " The schema kafka_config doesn't exits.......

Chris_Daly_HPE

Topic add requires the --config-schema as well.  Command should be:

 

/opt/vertica/packages/kafka/bin/vkconfig topic --add --target public.kafka_tgt --rejection-table public.kafka_rej --topic messages --num-partitions 4  --config-schema myScheduler

 

-Chris

Mahesh_Benne

now i am getting error for commad
 
/opt/vertica/packages/kafka/bin/vkconfig launch --conf configFile.propertis
 
the error is " expection in the thread "main""   java error...

Chris_Daly_HPE

Are you using the conf file?  if not, maybe try just a /vkconfig launch &

 

In older versions, the launch command also needed the --config-schema  flag as well, however documentation does not indicate that.  You might want to also try /vkconfig launch --config-schema myScheduler &

 

-Chris

Mahesh_Benne

Here is the out put..

[dbadmin@localhost bin]$ ./vkconfig launch &
[1] 16212
[dbadmin@localhost bin]$ Exception in thread "Main" java.sql.SQLException: [Vertica][VJDBC](4650) ERROR: Schema "kafka_config" does not exist
at com.vertica.util.ServerErrorData.buildException(Unknown Source)
at com.vertica.dataengine.VQueryExecutor.executeSimpleProtocol(Unknown Source)
at com.vertica.dataengine.VQueryExecutor.execute(Unknown Source)
at com.vertica.jdbc.common.SStatement.executeNoParams(Unknown Source)
at com.vertica.jdbc.common.SStatement.executeQuery(Unknown Source)
at com.vertica.solutions.kafka.cli.CLIUtil.validateCopySyntaxes(CLIUtil.java:34)
at com.vertica.solutions.kafka.Launcher.run(Launcher.java:69)
at com.vertica.solutions.kafka.Launcher.main(Launcher.java:136)
Caused by: com.vertica.support.exceptions.ErrorException: [Vertica][VJDBC](4650) ERROR: Schema "kafka_config" does not exist
... 8 more

[1]+ Exit 1 ./vkconfig launch
[dbadmin@localhost bin]$ ./vkconfig launch -- config-schema myScheduler &
[1] 16296
[dbadmin@localhost bin]$ No such argument

Chris_Daly_HPE

So the first error indicates that it hinks the schema is kafka_config

 

And the where you issue the command with the --config-schema you placed a space between -- and config.  Try this:

 

./vkconfig launch --config-schema myScheduler &

 

-chris

Mahesh_Benne

i got this message now..

[dbadmin@localhost bin]$ ./vkconfig launch --config-schema myScheduler &
[1] 18002

 

after this do i need to remove broker information in this script
COPY
schema.target_
table
SOURCE
KafkaSource
(stream='topic1|1|1,topic2|2|2',
brokers='host1:9092,host2:9092',duration=
INTERVAL'timeslice',stop_
on_
eof=TRUE,
eof_
timeout=
INTERVAL'timeslice')
PARSER
KafkaJSONParser
(flatten_
arrays=False,
flatten_
maps=False)
REJECTED
DATA
AS
TABLE
schema.rejection_
table
TRICKLE

Chris_Daly_HPE

Hi,

    At this point the data with in the messages topic on kafka should be loading in microbatches into the public.kafka_tgt table in Vertica.  

 

you can check this by going into vsql and issuing this command:

 

Select count(*) from public.kafka_tgt;

 

-Chris

Mahesh_Benne

dbadmin=> Select count(*) from public.kafka_tgt;
 count
-------
     0
(1 row)

dbadmin=> COPY schema.target_table SOURCE KafkaSource (stream='topic1|1|1,topic2|2|2',brokers='host1:9092,host2:9092',duration=INTERVAL'timeslice',stop_on_eof=TRUE,eof_timeout=INTERVAL'timeslice') PARSER KafkaJSONPars(flatten_arrays=False,flatten_maps=False) REJECTED DATA AS TABLE schema.rejection_table TRICKLE;
ERROR 4856:  Syntax error at or near "schema" at character 6
LINE 1: COPY schema.target_table SOURCE KafkaSource (stream='topic1|...
             ^
dbadmin=>