Specify custom S3 endpoint for backup

vishakha_lallvishakha_lall Vertica Customer
edited May 2020 in General Discussion

I am trying to backup the Vertica cluster to a S3 like data store internal to my enterprise network. We have similar credentials (ACCESS KEY and SECRET KEY).
Here's how my .ini file looks like

[S3]
s3_backup_path = s3://vertica_backups
s3_backup_file_system_path = []:/vertica/backups
s3_concurrency_backup = 10
s3_concurrency_restore = 10

[Transmission]
hardLinkLocal = True

[Database]
dbName = production
dbUser = dbadmin
dbPromptForPassword = False

[Misc]
snapshotName = fullbak1
restorePointLimit = 3
objectRestoreMode = createOrReplace
passwordFile = pwdfile
enableFreeSpaceCheck = True

Where can I supply my specific endpoint? For instance, my S3 store is available on a.b.c.d:80. I have tried changing s3_backup_path = a.b.c.d:80://wms_vertica_backups but I get the error Error: Error in VBR config: Invalid s3_backup_path. Also, I have the ACCESS KEY and SECRET KEY in ~/.aws/credentials.

After going through more resources I have exported the following ENV variables VBR_BACKUP_STORAGE_ENDPOINT_URL, VBR_BACKUP_STORAGE_ACCESS_KEY_ID, VBR_BACKUP_STORAGE_SECRET_ACCESS_KEY. vbr init throws the error Error: Unable to locate credentials Init FAILED. , I'm guessing it is still trying to connect to the AWS S3 servers. (Now removed credentials from ~/.aws/credentials

I think it's worthy to add that I'm running Vertica Enterprise mode.

Tagged:

Best Answer

Answers

  • LenoyJLenoyJ - Select Field - Employee

    What version are you on? The VBR capability for compatible on-prem S3 environments was added in 9.3.1.

  • vishakha_lallvishakha_lall Vertica Customer

    @LenoyJ I'm using Vertica Enterprise mode 8.1.1, is there any work around for the version that I am using?

  • vishakha_lallvishakha_lall Vertica Customer

    @sahil_kumar Yep, that worked like a breeze, the line number for s3_endpoint_url was different for me but it was easy to figure out. Thanks a ton!

  • vishakha_lallvishakha_lall Vertica Customer
    edited May 2020

    This can now be ignored, resolved!

  • sahil_kumarsahil_kumar Vertica Employee Employee

    Thanks for the confirmation @vishakha , but will recommend to get upgrade to vertica 9.3.1 or higher as you might need some other workarounds for other different operations(other than vbr) while using on-prem S3 in vertica 8.1.1.

  • ttbestasttbestas Vertica Customer
    edited February 2021

    @sahil_kumar i have 9.2.1 version, done everything per your instructions but still getting region error:
    File "/opt/vertica/oss/python/lib/python2.7/site-packages/botocore-1.5.65-py2.7.egg/botocore/signers.py", line 229, in get_auth_instance
    raise botocore.exceptions.NoRegionError()
    NoRegionError: You must specify a region.

    evi ~/.bashrc
    export DEFAULT_REGION="us-east-1"
    export ACCESS_KEY_ID=x
    export SECRET_ACCESS_KEY=x
    export REGION="us-east-1"
    export VBR_BACKUP_STORAGE_ACCESS_KEY_ID=x
    export VBR_BACKUP_STORAGE_SECRET_ACCESS_KEY=x
    export AWS_REGION="us-east-1"

    also i get AWSRegion=us-east-1 on all nodes if i check vs_configuration_parameters

  • SruthiASruthiA Vertica Employee Administrator

    @ttbestas Can you share me your ini file used for backup?

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file