MOVE_PARTITIONS_TO_TABLE breaks ROS containers

Sajan_Sangraula

We have a table where partitions are merged to a single ROS container using MERGE__PARTITIONS. When these merged partitions are moved to a new table using MOVE_PARTITIONS_TO_TABLE, the partitions are separated to new ROS containers of their own. Since the source and destination tables have the same structure with the same partition clause, shouldn't the MOVE_PARTITION_TO_TABLE function recognize partition merges if all the partitions being moved are in the same container?

[Deleted User]

Hi Sajan, thanks for reporting your problem! We'll see if we can find a fix for you.

[Deleted User]

Hi Sajan, just wanted to let you know that I am testing out the behavior you are seeing, and I will let you know what I come up with as soon as I do.

Bhawana

Hi Sajan, MOVE_PARTITION_TO_TABLE function doesn't recognize partition merges if all the partitions being moved are in the same container. On the source side, when you have the 1 container and you move one of it's partitions, a new single container with the remaining ones would be created. So the source side it would seem like should retain a single container. The target side might behave differently though. The target side is not originally a merged partition, it's individual partitions moved one at a time. So If I have 12 months on source side, and I move partition 12 to the target I'd get one container, then if I moved partition 11 to the target I think I'd get a second container. Only if I did the merge_partitions on the target side would it consolidate back to a single container. Please let me know if you have any questions. Regards, Bhawana

Nimmi_gupta

Current version of Vertica doesn't respect MERGE__PARTITIONS with MOVE_PARTITION_TO_TABLE function. It is working as it has been designed for now. A product enhancement request has been entered for future release.

Nimmi_gupta

Current version of Vertica doesn't respect MERGE__PARTITIONS with MOVE_PARTITION_TO_TABLE function.  It is working as it has been designed for now.

A product enhancement request has been entered for future release.

Mike_Mansell

In our case the MOVE_PARTITION_TO_TABLE function appears to be reorganizing every storage container on the source table that has merged partitions which I'm assuming is not "working as it has been designed". For example, Assume a table with 5 years of data partitioned by month and merged by year. With 2 buddy projections on a 4 node cluster I end up with 40 ROS containers for this table. If I perform a MOVE_PARTITION_TO_TABLE to move a single month partition from this table I end up with 479 ROS containers on the source table. Every merged ROS container has been split out by partition regardless of which partition has been moved or if that partition was included in any of the containers. In fact I can issue a MOVE_PARTITION_TO_TABLE on a non-existent partition in the source table and it will still reorganize and split up every partition that I have ever merged. This can obviously be pretty expensive and effectively makes the ability to merge partitions useless as any partition movement nullifies previous attempts to merge any partitions. Is this still a product enhancement or does this qualify as a separate issue?

Sergey_Cherepan

Problem is wider than originally stated. If you will merge several partitions into single ros container, following operations will split them back (actually entire table will be split): 1) move partitions to another table 2) drop partition 3) purge on projection, table and database levels (if delete vector is present in corresponding partitions) 4) Partition move to another storage location Additionally, you cannot rerganize merged ros container. For expample, you merged 4 partitions into one container, and then want to split it into 2 ros containers with 2 partitions each. This operation require split of whole table and merge again. Vertica badly needs handling of "partitions groups". This can be an answer to "very high file count" problem with too many partitions. Unfortunately it is not planned. Despite some concerns in previuos posts, this feature works exactly as designed, no bugs.