s3export

Hi,

Where can I find documentation about this function ? limitations, errors ?

 

I have a table with 1130 columns/ 13 millions rows and I am trying to export it to s3.

All columns are exported (about 400 files created) but not all rows.

I obtained the following error:

 

SQL Error [5861] [VP001]: [Vertica][VJDBC](5861) ERROR: Error calling processPartition() in User Function s3export at [src/S3.cpp:540], error code: 0, message: Numeric output buffer too small
[Vertica][VJDBC](5861) ERROR: Error calling processPartition() in User Function s3export at [src/S3.cpp:540], error code: 0, message: Numeric output buffer too small
com.vertica.util.ServerException: [Vertica][VJDBC](5861) ERROR: Error calling processPartition() in User Function s3export at [src/S3.cpp:540], error code: 0, message: Numeric output buffer too small

 

Thank you,

Veronica

Comments

  • Hello,

    I am having the same problem.

    Were you ever able to resolve this?

    Thanks,
    Drew

  • Hi @ckotsidimos -- I've read the manual many times. I'm not sure what I'm missing.

    Here is the exact error message I get:

    17:09:47 [SELECT - 0 rows, 1778.340 secs] [Code: 5861, SQL State: VP001] [Vertica]VJDBC ERROR: Error calling processPartition() in User Function s3export at [src/AWSHttpClient.cpp:259], error code: 0, message: MethodNotAllowed (405): The specified method is not allowed against this resource.
    ... 4 statement(s) executed, 0 rows affected, exec/fetch time: 1778.856/0.000 sec [3 successful, 1 errors]

  • asaeidiasaeidi - Select Field - Employee

    Hi @ScubaDrew
    What version of vertica are you using?
    You may encounter that error sometimes when s3export aborts. It tries to clean up by deleting partial uploads, but there is nothing to clean up.

  • @asaeidi Is there any way to determine why it aborted?

  • From my vertica logs:

    2017-06-02 13:00:42.178 Init Session:0x7f6669aec700-a0000003c75556 [UserMessage] <UDx> s3export - Buffer size: 10485760
    2017-06-02 13:00:42.241 Init Session:0x7f6669aec700-a0000003c75556 [UserMessage] <UDx> s3export - Buffer size: 10485760
    2017-06-02 13:00:42.241 Init Session:0x7f6669aec700-a0000003c75556 [UserMessage] <UDx> s3export - Buffer size: 10485760
    2017-06-02 13:00:42.242 Init Session:0x7f6669aec700-a0000003c75556 [UserMessage] <UDx> s3export - Buffer size: 10485760
    2017-06-02 13:00:42.317 EEcmdq:0x7f65105e05d0 [UserMessage] <UDx> s3export - Export url: s3://devops.xxx/redshift/dev/import/events7.3f9df5bd.csv
    2017-06-02 13:00:42.317 EEcmdq:0x7f6510624790 [UserMessage] <UDx> s3export - Export url: s3://devops.xxx/redshift/dev/import/events7.352b2c07.csv
    2017-06-02 13:00:42.319 EEcmdq:0x7f6511671370 [UserMessage] <UDx> s3export - Export url: s3://devops.xxx/redshift/dev/import/events7.80eeb602.csv
    2017-06-02 13:00:42.324 EEcmdq:0x7f65126e45c0 [UserMessage] <UDx> s3export - Export url: s3://devops.xxx/redshift/dev/import/events7.64368286.csv
    2017-06-02 13:06:37.376 Init Session:0x7f6669aec700-a0000003c75556 <ERROR> @v_db_node0001: VP001/5861: Error calling processPartition() in User Function s3export at [src/AWSHttpClient.cpp:259], error code: 0, message: MethodNotAllowed (405): The specified method is not allowed against this resource.
    2017-06-02 13:08:12.838 Init Session:0x7f6669aec700-a0000003c75556 <ERROR> @v_db_node0002: VP001/5861: Error calling processPartition() in User Function s3export at [src/AWSHttpClient.cpp:259], error code: 0, message: MethodNotAllowed (405): The specified method is not allowed against this resource.
    
  • no, I wasn't able to use s3export for large tables.

  • asaeidiasaeidi - Select Field - Employee

    @ScubaDrew sorry for the delay. 7.2.3 is the first version we introduced the S3 connector. Could you upgrade to a newer version, and see if the problem still exists?

  • Our time with Vertica has come to an end. We will not be doing any further upgrades. My two years dealing Vertica has been filled with issues like this. Incomplete and non-function software for even basic tasks. Thanks for the suggestion @asaeidi -- but off to RedShift we go!

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file