Drop_partitions for exported parquet files and for external tables
Hi,
I had smallish data change in very sizeable table, affecting one partition. This table is exported to partitioned Parquet on S3. I need to sync changes in Vertica table data to Parquet files on S3.
Vertica do not offer option to override single partition for EXPORT TO PARQUET. Options are either to drop whole table dir structure "ifDirExists=overwrite", or append new files "ifDirExists=append".
It is a big hassle to refresh only single partition for large table exported to Parquet. You need to go to S3 and manually drop dir with content. That creates problems with giving users unnecessary access to S3. I prefer to have SQL interface to drop partition. Apart from being very handy, access to drop partition can be controlled by Vertica means.
Would be ideal if Vertica can offer drop_partition for partitioned EXPORT TO PARQUET. Would be even better if drop_partition would work on external table.
Please file new feature request to Engineering, hopefully they will find it useful and will agree to implement.
Thank you
Sergey