VerticaCopyStream processing speed

Anatoly_Deyneka

We use vertica 7 community edition on 20cores 32GB physical server. When we run VerticaCopyStream from our java program stream of 6.5MB (42k rows) is processed in 42seconds. CPU and disk utilisation is about 1%. How to improve it? I need to import handreds of GB. COPY #tmp_table_name FROM LOCAL STDIN DELIMITER '|' DIRECT ENFORCELENGTH We import the data to tmp table and then merge the data to other tables. BTW merge is fast enough , it takes few milliseconds.

eli_revach

Hi ,

How much it take when you try to load  the same  file via simple copy command from vsql ?   do your java process run on the db server or on some other server , if on other server what is the CPU and memory utilization you see their ?   

One of the way to scale data load is to run parallel load streams  on different nodes .

Anatoly_Deyneka

We run our java app on different server. Import via vsql on the same node takes milliseconds. It's network problem. Thank you for the help.