file handle count calculation

Navin_C · February 2016

Hi All,

I have been struggling to find a relation between Open file handle count and Vertica process on each node.

How can we determine, how many files does vertica open for a query.

I know we can use resource acquisitions table, but I need to understand the logic behind these numbers.

Logically.

A ROS container can have many files, one file for each column of that container.

If a select query with only one column is executed, then it should open only the index file and file containing the column values in that ROs container as a result of partition pruning, but this is not observed in simple cases.

I tried querying the single column of a partitioned table with 20 values. The data was distributed on all nodes of a 3 node cluster, but the query shows file handle count as

Node 1 : 7 (initiator)

Node 2 : 4

Node 3 : 4

I query two columns and the result changes

Node 1 : 9 (initiator)

Node 2 : 6

Node 3 : 6

Any expalnation to logic would be appreciated.

Thanks

eli_revach · February 2016

Hi ,

Looks like from 7.X and above you no longer have pidx and fdb files , vertica do not maintain the index file in seperate file , for now on , you will see file with extention _0.gt .

Now for your specific use case i see few options :

Mutipule ROS files per specific columns
Mutipule ROS files per specific columns + temp files (Sort_Temp_<id>.dat) , the temp files are create and delete at the end of the query

Very easy way to monitor which files are open is using lsof utility , you can corellect the name of the files you get from the lsof utility (try to grep the results by data location you have on the file system ) with the sal_storage_id column exists in the storage_containers table .

Hope you will find it useful

Thanks .

file handle count calculation

Comments

Leave a Comment