Options

VerticaCopyStream processing speed

Anatoly_DeynekaAnatoly_Deyneka
in General Discussion
We use vertica 7 community edition on 20cores 32GB physical server. When we run VerticaCopyStream from our java program stream of 6.5MB (42k rows) is processed in 42seconds. CPU and disk utilisation is about 1%. How to improve it? I need to import handreds of GB. COPY #tmp_table_name FROM LOCAL STDIN DELIMITER '|' DIRECT ENFORCELENGTH We import the data to tmp table and then merge the data to other tables. BTW merge is fast enough , it takes few milliseconds.

Comments

  • Options
    eli_revacheli_revach
    Hi ,

    How much it take when you try to load  the same  file via simple copy command from vsql ?   do your java process run on the db server or on some other server , if on other server what is the CPU and memory utilization you see their ?   

    One of the way to scale data load is to run parallel load streams  on different nodes .

  • Options
    Anatoly_DeynekaAnatoly_Deyneka
    We run our java app on different server. Import via vsql on the same node takes milliseconds. It's network problem. Thank you for the help.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Format
SpoilerCodeQuote
Heading 2Heading 1
Emoji
:):D:(;):/:o:s:p:'(:|B):#:*<3o:)>:)
Url
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file
Home • General DiscussionComment As ...