Importing from Amazon S3 using the Vertica Library for Amazon Web Services

edited June 2018 in General Discussion

Hi,
I have a use case where i am trying to load the data into one of the Vertica table from a .csv file present in AWS S3 bucket.

I would like to know how can i configure the Vertica Library for Amazon Web Services Parameters ?

I did try doing it using the instructions provided on the below link , but it did not work -->

https://my.vertica.com/docs/7.2.x/HTML/index.htm#Authoring/AdministratorsGuide/ConfiguringTheDB/AwsLibraryParameters.htm

AFTER completing the steps above i ran the below command , but getting the error -->

ERROR MESSAGE - ERROR 6085: Unexpected exception calling process() User Function in UDSource

All i need to achieve is load the data to vertica table from s3 . command that i used is as below -

.

<

table name> WITH SOURCE S3(url='s3://idl-sbg-uw2-processed-analytics-prod-prd/<filename.csv>');

If anyone has done this kind of usecase , please let me know . Appreciate your help.

Comments

  • asaeidiasaeidi - Select Field - Employee

    Hi, what's version of Vertica you're using?

  • Vertica Analytic Database v8.1.1-15

  • I tried executing these commands on VSQL first and then copy command as below -- >

    ALTER SESSION SET UDPARAMETER FOR awslib aws_region='us-west-2';
    ALTER SESSION SET UDPARAMETER FOR awslib aws_id='xxx';
    ALTER SESSION SET UDPARAMETER FOR awslib aws_secret='XXX';

    => copy abc.table SOURCE S3(url='s3://s3bucket/xyz.csv') DELIMITER ',';
    ERROR 5861: Error calling process() in User Function UDSource at [src/S3.cpp:137], error code: 0, message: Client Error: The AWS Access Key Id you provided does not exist in our records

  • Primarily i want to know how do i configure my vertica or what kind of configuration needs to be done in order to enable this copy from S3 to vertica.

  • Jim_KnicelyJim_Knicely - Select Field - Administrator

    Looks like you are attempting to configure it properly. The error "The AWS Access Key Id you provided does not exist in our records" is coming from AWS, not Vertica.

    I think you are using an invalid AWS_ID.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file