The best way of extracting data from flex tables
We store some types of data in the flex tables because they are quite unstructured and we believe that flex tables are the best solution for it.
From time to time we need to extract quite big amount of data (about 2 billion records)
We noticed that extracting data from the flex table using 'select field1, field2, …, fieldN from flex_table' is bottleneck.
For example, the extraction of 400000 records with 10 non-materialized fields takes about 58 seconds. Please note that all these 10 fields exist in stored JSON.
What is the best way (in terms of processing time) of exporting data from a flex table?