s3export

vcarusi

Hi,

Where can I find documentation about this function ? limitations, errors ?

 

I have a table with 1130 columns/ 13 millions rows and I am trying to export it to s3.

All columns are exported (about 400 files created) but not all rows.

I obtained the following error:

 

SQL Error [5861] [VP001]: [Vertica][VJDBC](5861) ERROR: Error calling processPartition() in User Function s3export at [src/S3.cpp:540], error code: 0, message: Numeric output buffer too small
[Vertica][VJDBC](5861) ERROR: Error calling processPartition() in User Function s3export at [src/S3.cpp:540], error code: 0, message: Numeric output buffer too small
com.vertica.util.ServerException: [Vertica][VJDBC](5861) ERROR: Error calling processPartition() in User Function s3export at [src/S3.cpp:540], error code: 0, message: Numeric output buffer too small

 

Thank you,

Veronica

ScubaDrew

Hello,

I am having the same problem.

Were you ever able to resolve this?

Thanks,
Drew

ckotsidimos

Here:
https://my.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/AdministratorsGuide/BulkLoadCOPY/AWSLibrary/exportAWSlibrary.htm?Highlight=s3export

ScubaDrew

Hi @ckotsidimos -- I've read the manual many times. I'm not sure what I'm missing.

Here is the exact error message I get:

17:09:47 [SELECT - 0 rows, 1778.340 secs] [Code: 5861, SQL State: VP001] [Vertica]VJDBC ERROR: Error calling processPartition() in User Function s3export at [src/AWSHttpClient.cpp:259], error code: 0, message: MethodNotAllowed (405): The specified method is not allowed against this resource.
... 4 statement(s) executed, 0 rows affected, exec/fetch time: 1778.856/0.000 sec [3 successful, 1 errors]

asaeidi

Hi @ScubaDrew
What version of vertica are you using?
You may encounter that error sometimes when s3export aborts. It tries to clean up by deleting partial uploads, but there is nothing to clean up.

ScubaDrew

@asaeidi 07.02.0300

ScubaDrew

@asaeidi Is there any way to determine why it aborted?

ScubaDrew

From my vertica logs:

2017-06-02 13:00:42.178 Init Session:0x7f6669aec700-a0000003c75556 [UserMessage] <UDx> s3export - Buffer size: 10485760
2017-06-02 13:00:42.241 Init Session:0x7f6669aec700-a0000003c75556 [UserMessage] <UDx> s3export - Buffer size: 10485760
2017-06-02 13:00:42.241 Init Session:0x7f6669aec700-a0000003c75556 [UserMessage] <UDx> s3export - Buffer size: 10485760
2017-06-02 13:00:42.242 Init Session:0x7f6669aec700-a0000003c75556 [UserMessage] <UDx> s3export - Buffer size: 10485760
2017-06-02 13:00:42.317 EEcmdq:0x7f65105e05d0 [UserMessage] <UDx> s3export - Export url: s3://devops.xxx/redshift/dev/import/events7.3f9df5bd.csv
2017-06-02 13:00:42.317 EEcmdq:0x7f6510624790 [UserMessage] <UDx> s3export - Export url: s3://devops.xxx/redshift/dev/import/events7.352b2c07.csv
2017-06-02 13:00:42.319 EEcmdq:0x7f6511671370 [UserMessage] <UDx> s3export - Export url: s3://devops.xxx/redshift/dev/import/events7.80eeb602.csv
2017-06-02 13:00:42.324 EEcmdq:0x7f65126e45c0 [UserMessage] <UDx> s3export - Export url: s3://devops.xxx/redshift/dev/import/events7.64368286.csv
2017-06-02 13:06:37.376 Init Session:0x7f6669aec700-a0000003c75556 <ERROR> @v_db_node0001: VP001/5861: Error calling processPartition() in User Function s3export at [src/AWSHttpClient.cpp:259], error code: 0, message: MethodNotAllowed (405): The specified method is not allowed against this resource.
2017-06-02 13:08:12.838 Init Session:0x7f6669aec700-a0000003c75556 <ERROR> @v_db_node0002: VP001/5861: Error calling processPartition() in User Function s3export at [src/AWSHttpClient.cpp:259], error code: 0, message: MethodNotAllowed (405): The specified method is not allowed against this resource.

vcarusi

no, I wasn't able to use s3export for large tables.

asaeidi

@ScubaDrew sorry for the delay. 7.2.3 is the first version we introduced the S3 connector. Could you upgrade to a newer version, and see if the problem still exists?

ScubaDrew

Our time with Vertica has come to an end. We will not be doing any further upgrades. My two years dealing Vertica has been filled with issues like this. Incomplete and non-function software for even basic tasks. Thanks for the suggestion @asaeidi -- but off to RedShift we go!