Please take this survey to help us learn more about how you use third party tools. Your input is greatly appreciated!
Loading parquet files with complex data type
Is it possible to load a parquet file with the schema, that has complex data type like structs or lists?
I am trying to load a file with the shema below, however vertica gives me "complex types are not supported" message, even though per documentation structs are supported
Id: BYTE_ARRAY UTF8 capturedTimestamp: INT64 location.latitude: DOUBLE location.longitude: DOUBLE location.postalCode: BYTE_ARRAY UTF8 event.metadata.triptype: BYTE_ARRAY UTF8
Thanks for any help/suggestions.
Leave a Comment
Can't find what you're looking for? Search the Vertica Documentation, Knowledge Base, or Blog for more information.
What is your external table definition? What is the data type you are specifying in Vertica for BYTE_ARRAY type in the schema?
Which version of Vertica are you on? Structs were introduced in 9.2.
I'm certain that the documentation says that structs are supported in the context of External Tables. But you can attempt to load the data with structs using a COPY.
Use this guide as reference: https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/AdministratorsGuide/Tables/ExternalTables/Structs.htm
So a table in Hive with nested structs like the following:
Can be written as an External Table in Vertica as:
You can create a regular table like below and even do a COPY.
Attached atomix.parquet that I used.
Ah, that make sense, thank you. Using 9.1.0-1. OK, going to upgrade to 9.2 and will give it a try, thank you!