KafkaAvroParser problem

alekon

I have a problem when I try to load the data from avro-formatted kafka stream to Vertica flex table. I use KafkaAvroParser, schema registry server and the query:

COPY "public"."test" SOURCE KafkaSource(stream='public-test|0|1', brokers='127.0.0.1:9092', duration=interval '9941 milliseconds', eof_timeout=interval '1000 milliseconds', stop_on_eof=true, message_max_bytes=1048576 ) PARSER KafkaAvroParser(schema_registry_url='http://127.0.0.1:8081/') REJECTED DATA AS TABLE "public"."test_rej" TRICKLE STREAM NAME 'vkstream_stream_config_microbatch_8000001_conversion' NO COMMIT;

When I use the command line avro consumer:
kafka-avro-console-consumer --topic public-test --zookeeper 127.0.0.1:2181 --property print.key=true

The result is:
{"id":{"int":1}} {"id":{"int":1},"value":{"string":"test1"}}

But in my Vertica flex table I get just the empty values:

select maptostring(____raw____) from public.test;

{
"name" : "test",
"id" : "",
"value" : ""
}

I have no idea, what I'm doing wrong...

skeswani

in KafkaAvroParser can you provide the json representation of external_schema={}.
and see if that works? wonder if its related to the schema registry.

furthermore, what was writte to test_rej?

aurorak

What version of Vertica and Kafka are you using?

alekon

Skeswani, schema registry works fine. I can get the schema. And I guess the avro console consumer uses the same one.
And nothing was written in test_rej.

alekon

aurorak, I use Vertica 8.1 and Kafka 2.11

aurorak

@alekon I would try this against version 8.0x if you have access to it.

alekon

aurorak, I have tried 8.0.0 version. All the same.