Selectively retaining data from data collectors

kxu

While the execution_engine_profiles data collector is very useful for capturing query metrics, the table grows large very quickly. Is there a way to selectively retain certain metrics?

More generally, can a data collector retain information in a query optimized format (regular permanent tables)?

ChuckB

There are several answers to this.
1. You can up the retention for the data collector tables.
2. You can send the DC data elsewhere, using Kafka: https://www.vertica.com/blog/publish-data-collector-tables-apache-kafka/
3. You can copy the DC data to a table; vertica generates example scripts on how to do this and puts them in your DataCollector directory when it starts up.
4. For the execution_engine_profiles table specifically, consider increasing the time threshold for recording queries, so that the shortest ones aren't retained. See "SaveDCEEProfileThresholdUS".