Fast bulkload for flex tables
we have a primary table designed as a flex table. We are looking for a fastest way to load data in it. Currently, we store all upcoming events to a local disk JSON file, and then every 2 minutes we roll this file and execute copy local command using fjsonparser. We observed that a file with around 500000 (half million) events takes around 200 seconds which looks a bit slower. I know that JSON parser is quite slower than the CSV parser used for the normal table. We are looking for the best practices to speed upload:
1. JSON parser can be optimized?
2. copy would be better than copy local? (if we choose to copy the file to a shared file system and execute copy on the vertica server).
3. Does bulkload need a specific resource pool and which parameters of the resource pool can impact the performance?
4. Copying a few large files vs copy more small files - which is better?
5. How many huge copy load commands can Vertica execute in parallel?