Vertica catalog - memory issue

eli_revach · July 2013

Hi , I am checking vertica as an database solution to a product we develop that will run on SAAS , when we design SAAS solution one of the basic test we run is to check how database behave to big catalogs (many tables) . Our test show that vertica probably load the entire catalog into memory as we see dramatic memory growth when we run out test 10K tables+ super projection. Btw – the memory growth was not show on the resource pools data dictionary tables , so checking them don’t present any issue . We look behave was present on the level of OS monitor tools that we run on top of vertica process . I would like to know if this us expect behave .

[Deleted User] · August 2013

Hi Eli, This is correct -- Vertica currently keeps a full copy of its catalog in memory. As a result, it may not work well for cases where the catalog is extremely large. We do have customers running with many more than 10k tables, though. It depends heavily on the schemas of the tables, the number and type of projections, etc. (And, of course, on the amount of RAM available on each node.) If you're seeing memory usage that would make even 10k tables untenable on solid modern hardware, feel free to post a more-specific question or to check with Vertica Support. Adam

eli_revach · August 2013

Hi thanks for the replay ,i understand that it related to the amount of projections ,but how it related to the amounts of schemas ? Can you elaborate on this ?

[Deleted User] · August 2013

Hey Eli, No problem. Sorry, though, I was unclear: I didn't mean number of schemas created by CREATE SCHEMA; I meant the exact definition of each table, the table's columns, etc. (That definition is also sometimes referred to as the table's schema.) If you have, for example, lots and lots of columns, that's more metadata for the catalog to track. Adam

eli_revach · August 2013

Hi Adam , Any idea why resource_pool_status table do not show any special memory allocation ? The general pool and all others pools show very minimal allocation comparing to what is going on the level of the DB server . Eli .

[Deleted User] · August 2013

Hey Eli, Yes -- resource pools govern resources granted to queries (and to the WOS, as a special case), but the catalog isn't a query, so it's not managed by resource pools. Unfortunately, I don't know offhand where the best place to look would be, for catalog data structures. The system-table documentation might or might not have some suggestions for you. Adam

Vertica catalog - memory issue

Comments

Leave a Comment