Kafka streaming to Flex table with columns
we are on vertica 8.1.1. I'm streaming clickstream data (JSON format) from a Kafka topic into vertica flex table using vkconfig.
1) I created the flex table with default timestamp column as follows and data is loaded to raw column from the input stream. But the insert_timestamp column has all NULL values. With vkconfig, we can't do ETL while streaming. But this is a default timestamp column added to FLEX table. Is there a way to get this column populated with the insert time, while streaming data from Kafka? appreciate any help on this.
create flex table topic_1_tgt (
INSERT_TIMESTAMP timestamp(0) DEFAULT getdate() );
2) Also stream_microbatch_history has a lot of history data. How can we cleanup this table and delete old data?