s3export

vcarusi · October 2016

Hi,

Where can I find documentation about this function ? limitations, errors ?

I have a table with 1130 columns/ 13 millions rows and I am trying to export it to s3.

All columns are exported (about 400 files created) but not all rows.

I obtained the following error:

SQL Error [5861] [VP001]: [Vertica][VJDBC](5861) ERROR: Error calling processPartition() in User Function s3export at [src/S3.cpp:540], error code: 0, message: Numeric output buffer too small
[Vertica][VJDBC](5861) ERROR: Error calling processPartition() in User Function s3export at [src/S3.cpp:540], error code: 0, message: Numeric output buffer too small
com.vertica.util.ServerException: [Vertica][VJDBC](5861) ERROR: Error calling processPartition() in User Function s3export at [src/S3.cpp:540], error code: 0, message: Numeric output buffer too small

Thank you,

Veronica

ScubaDrew · May 2017

Hello,

I am having the same problem.

Were you ever able to resolve this?

Thanks,
Drew

ckotsidimos · June 2017

Here:
https://my.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/AdministratorsGuide/BulkLoadCOPY/AWSLibrary/exportAWSlibrary.htm?Highlight=s3export

ScubaDrew · June 2017

Hi @ckotsidimos -- I've read the manual many times. I'm not sure what I'm missing.

Here is the exact error message I get:

17:09:47 [SELECT - 0 rows, 1778.340 secs] [Code: 5861, SQL State: VP001] [Vertica]VJDBC ERROR: Error calling processPartition() in User Function s3export at [src/AWSHttpClient.cpp:259], error code: 0, message: MethodNotAllowed (405): The specified method is not allowed against this resource.
... 4 statement(s) executed, 0 rows affected, exec/fetch time: 1778.856/0.000 sec [3 successful, 1 errors]

asaeidi · June 2017

Hi @ScubaDrew
What version of vertica are you using?
You may encounter that error sometimes when s3export aborts. It tries to clean up by deleting partial uploads, but there is nothing to clean up.

ScubaDrew · June 2017

@asaeidi 07.02.0300

ScubaDrew · June 2017

@asaeidi Is there any way to determine why it aborted?

ScubaDrew · June 2017

From my vertica logs:

2017-06-02 13:00:42.178 Init Session:0x7f6669aec700-a0000003c75556 [UserMessage] <UDx> s3export - Buffer size: 10485760
2017-06-02 13:00:42.241 Init Session:0x7f6669aec700-a0000003c75556 [UserMessage] <UDx> s3export - Buffer size: 10485760
2017-06-02 13:00:42.241 Init Session:0x7f6669aec700-a0000003c75556 [UserMessage] <UDx> s3export - Buffer size: 10485760
2017-06-02 13:00:42.242 Init Session:0x7f6669aec700-a0000003c75556 [UserMessage] <UDx> s3export - Buffer size: 10485760
2017-06-02 13:00:42.317 EEcmdq:0x7f65105e05d0 [UserMessage] <UDx> s3export - Export url: s3://devops.xxx/redshift/dev/import/events7.3f9df5bd.csv
2017-06-02 13:00:42.317 EEcmdq:0x7f6510624790 [UserMessage] <UDx> s3export - Export url: s3://devops.xxx/redshift/dev/import/events7.352b2c07.csv
2017-06-02 13:00:42.319 EEcmdq:0x7f6511671370 [UserMessage] <UDx> s3export - Export url: s3://devops.xxx/redshift/dev/import/events7.80eeb602.csv
2017-06-02 13:00:42.324 EEcmdq:0x7f65126e45c0 [UserMessage] <UDx> s3export - Export url: s3://devops.xxx/redshift/dev/import/events7.64368286.csv
2017-06-02 13:06:37.376 Init Session:0x7f6669aec700-a0000003c75556 <ERROR> @v_db_node0001: VP001/5861: Error calling processPartition() in User Function s3export at [src/AWSHttpClient.cpp:259], error code: 0, message: MethodNotAllowed (405): The specified method is not allowed against this resource.
2017-06-02 13:08:12.838 Init Session:0x7f6669aec700-a0000003c75556 <ERROR> @v_db_node0002: VP001/5861: Error calling processPartition() in User Function s3export at [src/AWSHttpClient.cpp:259], error code: 0, message: MethodNotAllowed (405): The specified method is not allowed against this resource.

vcarusi · June 2017

no, I wasn't able to use s3export for large tables.

asaeidi · June 2017

@ScubaDrew sorry for the delay. 7.2.3 is the first version we introduced the S3 connector. Could you upgrade to a newer version, and see if the problem still exists?

ScubaDrew · June 2017

Our time with Vertica has come to an end. We will not be doing any further upgrades. My two years dealing Vertica has been filled with issues like this. Incomplete and non-function software for even basic tasks. Thanks for the suggestion @asaeidi -- but off to RedShift we go!

s3export

Comments

Leave a Comment