Relationship between Cores and loading

Navin_C

Hello All,

I wanted to understand the relationship between number of cores and loading in Vertica.
I remember there is some dependency on number of cores while inserting data into vertica. whether it is the INSERT statement or COPY statements.

Any limitations / Any dependency / any points to remember

Thanks 

[Deleted User]

Hi Navin,

The only relationship here, generally speaking, is that COPY will parse up to "#cores" files at once, when it is loading files from local disk on the server.  (Parsing is often CPU-bound, so we hope this is a reasonable limit.)  If you give us a list of 100 files, but you only have 24 CPU cores, then we will process the first 24 files; then, when the first of those files finishes parsing, we'll pick up the 25th, then the 26th, etc.

(Many versions ago, Vertica's COPY used to simply try to load all files at once.  This worked ok for smaller numbers of files, but used rather a lot of system resources when loading tens or hundreds of thousands of files.)

Say you have 24 cores and are loading 25 equal-sized files.  You might think that this would take twice as long as loading 24 files, because we'd load 24 files at first, then have a single thread loading the last file.  In Vertica 6, this was an issue.  As of Vertica 7, when we notice that a CPU core has no files left to parse, we will assign that core to help parse an existing file.  So we should generally be efficient even in this case.

Adam

Sai_Pasupulati

Hi Adam,

We are talking loading  into Vertica, I want utilize same thread ask this question.

Could you please help on this

I want find out  some load analysis to monitor the health of the system, like system is reaching its max load capacity and writes per seconds  , This is very critical for us do some capacity planning.

 thank you

Sai

Sai_Pasupulati

need some answer please

[Deleted User]

Thanks for your interest -- if you need prompt support, please contact Vertica Support at my.vertica.com.  These are public forums; you may or may not get an answer, and you are very unlikely to get a response within half an hour.

If you're looking for a quick response, I'll give you a quick answer -- Vertica's Management Console provides you with these tools.