Max rows per file in COPY command

Hi

 

I am using COPY command to load 1 file with 27 million rows in it and i am getting an error [HY000] Unexpected protocol message: DataRow

 

when doing it on a file with 14 million rows there is no problem.

 

Is there a limit on the number of rows in a file?

Comments

  • Hi ,

    Their is no limit on the amount of the rows per file , you should provide more deatils about the content of the file  the method you using the COPY command , and your vertica setup , it may for getting more info about what you trying to do 

     

    Thanks 

  • Hi!

     

     

    Limit of cause exist: number of rows per load - 2^63 = 9,223,372,036,854,775,808 rows

    http://my.vertica.com/docs/7.1.x/HTML/index.htm#Authoring/SQLReferenceManual/SystemLimits.htm

     

    >> Unexpected protocol message: DataRow

    Here is a problem. DataRow - is response message generated by SELECT, not by COPY.

     

    Can you provide more info?

    • db version
    • client (vsql/JDBC/ODBC/ADO/Tableu/...???)
    • file is locally or remotely
    • post your COPY statement please.
  • Hi

     

    1 Vertica 7.1.2-2

    2 .NET (ADO)

    3 Locally

    4 copy command sample:

    COPY DB.SCHEMA.TABLE (col1,col2,col3)
    FROM LOCAL C:\MyFile.csv UNCOMPRESSED DELIMITER ',' NULL AS 'NULL' 
    ESCAPE AS '
    \\' ENCLOSED BY '\"' DIRECT STREAM NAME 'StreamName'
    EXCEPTIONS 'C:\exceptions' REJECTED DATA 'C:\rejections' 
    ABORT ON ERROR NO COMMIT;
     

     

  • Did you try to run this copy command from vsql ? 

  • Hi!

     

    I think that DataRow comes here to describe a rejected row and fails. Have you successfully uploaded 14M of data(without any rejection)?

     

    BTW: your COPY statement is strange: exceptions+rejections and abort on error+no commit. Can you explain your logic? I can't get it.

     

    Disclaimer: Im not an expert in ADO(.NET) so maybe someone else can help you, but 99.9% problem in ADO.

     

    PS

    http://www.akadia.com/services/dotnet_handling_row_errors.html

     

  • Hi!

     

    I don't know why, but my replay deleted and I will not repost it :(

    Sorry.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file