favroparser throwing error
I am using favroparser to COPY data to a table. The schema is attached in the avro file itself and below is the schema .
{"type":"record","name":"Person","fields":[{"name":"ID","type":{"type":"long"}},{"name":"First","type":{"type":"string"}},{"name":"Last","type":{"type":"string"}},{"name":"Phone","type":{"type":"string"}},{"name":"Age","type":{"type":"int"}}]}
When I try to copy the file with above schema , I am getting above "File schema not supported". Can you let me know what is the issue?
0
Comments
Had no problems in a simple test with your avro schema and a flex table.
Could you provide more details about your own setup? Vertica version, sample avro file, target table description, copy command ...???
Hi SergeB,
Our Vertica version is 9.0. Below is the copy command and target table and attached is the avro file. I renamed it as txt file to allow the attachment, you can rename it to avro after downloading.
create table t (station varchar, time int, temp int);
dbadmin=> copy t from local '~/work/examples/weather.avro' with parser favroParser(flatten_records=true,flatten_arrays=true,flatten_maps=true);
ERROR 5861: Error calling destroy() in User Function UDParser at [src/AvroParser.cpp:979], error code: 0, message: Error: Avro file schema not supported
Regards
Venkaiah
I get some errors when trying to parse the avro file you uploaded using standard java avro apis. Those would seem consistent with the error you're getting during the COPY command.
As an experiment and in case there was a problem with the attachment , i created an avro record from this schema (from the file you updated)
{"type":"record","name":"Weather","namespace":"test",
"fields":[{"name":"station","type":{"type":"string"}},{"name":"time","type":{"type":"long"}},
{"name":"temp","type":{"type":"int"}}],
"doc":"A weather reading"}
and this json:
{"station":"Sunnyvale","time":8291829812,"temp":99}
{"station":"Cambridge","time":8291829812,"temp":30}
java -jar ./avro-tools-1.8.2.jar fromjson --schema-file aschema.avsc arecord.json > arecord.avro
No problem parsing in Java and no problem loading via COPY
Could you try to build a simple record as above?
Hi SergeB
I am able to parse the the avro file I attached using avro-tools.jar .
When I generated avro using above schema you have given and data using avro-tools.jar, the avro file generated has schema file like this where type will become norma so obviously java avro is working fine. {"type":"record","name":"Weather","namespace":"test",
"fields":[{"name":"station","type":"string"},{"name":"time","type":"long"},
{"name":"temp","type":"int"}],
"doc":"A weather reading"}
The avro file I generated uses avro-c library for writing, so it should be compatible with other language avro libs, I am not sure why it is not working fine with favroparser? Can I use KafkaAvroParser with streming it through KafkaSource?
Regards
Venkat
Hi SergeB,
I fixed the avro file schema issue by changing the avro-c code in our repo, but now the problem is our avro file as codec as "deflate" and it seems that favroparser doesn't have support to load avro files with codec. If I generated the same avro with codec set to null the file is loading perfectly. Is there a way to avro file with codec?
Regards
Venkat
Hi Venkat
Could you open a support ticket and upload some example avro messages? It's difficult to assess your parsing issues without looking at some examples. Unlike the forum, the support site should not mangle attachments.
Thanks
hi SergeRB,
I sent email for support ticket to verticahelp@microfocus.com and verticahelp@hpe.com but didn't get any response until now. Can you let me know the way to open support ticket.
Regards
Venkat
@Venkat,
Please visit our "Software Support Website" to open a case:
https://softwaresupport.softwaregrp.com/