KafkaAvroParser problem

I have a problem when I try to load the data from avro-formatted kafka stream to Vertica flex table. I use KafkaAvroParser, schema registry server and the query:

COPY "public"."test" SOURCE KafkaSource(stream='public-test|0|1', brokers='', duration=interval '9941 milliseconds', eof_timeout=interval '1000 milliseconds', stop_on_eof=true, message_max_bytes=1048576 ) PARSER KafkaAvroParser(schema_registry_url='') REJECTED DATA AS TABLE "public"."test_rej" TRICKLE STREAM NAME 'vkstream_stream_config_microbatch_8000001_conversion' NO COMMIT;

When I use the command line avro consumer:
kafka-avro-console-consumer --topic public-test --zookeeper --property print.key=true

The result is:
{"id":{"int":1}} {"id":{"int":1},"value":{"string":"test1"}}

But in my Vertica flex table I get just the empty values:

select maptostring(____raw____) from public.test;

"name" : "test",
"id" : "",
"value" : ""

I have no idea, what I'm doing wrong...


  • Options
    skeswaniskeswani - Select Field - Employee

    in KafkaAvroParser can you provide the json representation of external_schema={}.
    and see if that works? wonder if its related to the schema registry.

    furthermore, what was writte to test_rej?

  • Options

    What version of Vertica and Kafka are you using?

  • Options
    edited July 2017
    Skeswani, schema registry works fine. I can get the schema. And I guess the avro console consumer uses the same one.
    And nothing was written in test_rej.
  • Options
    aurorak, I use Vertica 8.1 and Kafka 2.11
  • Options

    @alekon I would try this against version 8.0x if you have access to it.

  • Options

    aurorak, I have tried 8.0.0 version. All the same.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file