The Vertica Forum recently got a makeover! Let us know what you think by filling out this short, anonymous survey.
Please take this survey to help us learn more about how you use third party tools. Your input is greatly appreciated!

Specify custom S3 endpoint for backup

vishakha_lallvishakha_lall Vertica Customer
edited May 2020 in General Discussion

I am trying to backup the Vertica cluster to a S3 like data store internal to my enterprise network. We have similar credentials (ACCESS KEY and SECRET KEY).
Here's how my .ini file looks like

s3_backup_path = s3://vertica_backups
s3_backup_file_system_path = []:/vertica/backups
s3_concurrency_backup = 10
s3_concurrency_restore = 10

hardLinkLocal = True

dbName = production
dbUser = dbadmin
dbPromptForPassword = False

snapshotName = fullbak1
restorePointLimit = 3
objectRestoreMode = createOrReplace
passwordFile = pwdfile
enableFreeSpaceCheck = True

Where can I supply my specific endpoint? For instance, my S3 store is available on a.b.c.d:80. I have tried changing s3_backup_path = a.b.c.d:80://wms_vertica_backups but I get the error Error: Error in VBR config: Invalid s3_backup_path. Also, I have the ACCESS KEY and SECRET KEY in ~/.aws/credentials.

After going through more resources I have exported the following ENV variables VBR_BACKUP_STORAGE_ENDPOINT_URL, VBR_BACKUP_STORAGE_ACCESS_KEY_ID, VBR_BACKUP_STORAGE_SECRET_ACCESS_KEY. vbr init throws the error Error: Unable to locate credentials Init FAILED. , I'm guessing it is still trying to connect to the AWS S3 servers. (Now removed credentials from ~/.aws/credentials

I think it's worthy to add that I'm running Vertica Enterprise mode.


Best Answer


  • LenoyJLenoyJ - Select Field - Employee

    What version are you on? The VBR capability for compatible on-prem S3 environments was added in 9.3.1.

  • vishakha_lallvishakha_lall Vertica Customer

    @LenoyJ I'm using Vertica Enterprise mode 8.1.1, is there any work around for the version that I am using?

  • vishakha_lallvishakha_lall Vertica Customer

    @sahil_kumar Yep, that worked like a breeze, the line number for s3_endpoint_url was different for me but it was easy to figure out. Thanks a ton!

  • vishakha_lallvishakha_lall Vertica Customer
    edited May 2020

    This can now be ignored, resolved!

  • sahil_kumarsahil_kumar Vertica Employee Employee

    Thanks for the confirmation @vishakha , but will recommend to get upgrade to vertica 9.3.1 or higher as you might need some other workarounds for other different operations(other than vbr) while using on-prem S3 in vertica 8.1.1.

  • ttbestasttbestas Vertica Customer
    edited February 2021

    @sahil_kumar i have 9.2.1 version, done everything per your instructions but still getting region error:
    File "/opt/vertica/oss/python/lib/python2.7/site-packages/botocore-1.5.65-py2.7.egg/botocore/", line 229, in get_auth_instance
    raise botocore.exceptions.NoRegionError()
    NoRegionError: You must specify a region.

    evi ~/.bashrc
    export DEFAULT_REGION="us-east-1"
    export ACCESS_KEY_ID=x
    export SECRET_ACCESS_KEY=x
    export REGION="us-east-1"
    export AWS_REGION="us-east-1"

    also i get AWSRegion=us-east-1 on all nodes if i check vs_configuration_parameters

  • SruthiASruthiA Vertica Employee Employee

    @ttbestas Can you share me your ini file used for backup?

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file