Please take this survey to help us learn more about how you use third party tools. Your input is greatly appreciated!

Configure vbr to backup to MinIO

I have successfully configured Vertica 12 to use an on-prem MinIO server in Eon Mode for communal storage.

Now I am trying to configure vbr to backup to another bucket on the same server.

No matter what I do, vbr consistently tries to target the bucket on https://s3.my-minio-region.amazonaws.com.

Any help with pointing me in the right direction is appreciated.

Here's the command:
/opt/vertica/bin/vbr --task init --s3-force-init -c ./backup.ini --debug 3

I have set the following keys in the environment before calling vbr:

  • VBR_BACKUP_STORAGE_ACCESS_KEY_ID="..."
  • VBR_BACKUP_STORAGE_SECRET_ACCESS_KEY="..."
  • VBR_BACKUP_STORAGE_ENDPOINT_URL="https://minio-server:9000"
  • VBR_COMMUNAL_STORAGE_ACCESS_KEY_ID="..."
  • VBR_COMMUNAL_STORAGE_SECRET_ACCESS_KEY="..."
  • VBR_COMMUNAL_STORAGE_ENDPOINT_URL="https://minio-server:9000"

In the backup.ini file I am specifying in the [CloudStorage] section the following keys:

  • cloud_storage_backup_path = s3://acid-backup/backup_and_restore/acid.my-minio.net/dw_backup/db
  • cloud_storage_backup_file_system_path = []:/data/backup/data/
  • cloud_storage_encrypt_transport = True
  • cloud_storage_concurrency_backup = 10
  • cloud_storage_concurrency_backup = 10

Here's the last bits from the log file (--debug 3 but doesn't seem to add much):

`2023-01-05 03:20:05 localhost local_storage exists /.identity
2023-01-05 03:20:05 localhost s3 exists /.identity
2023-01-05 03:21:47 localhost vbr Traceback (most recent call last):
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
conn = connection.create_connection(
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/urllib3/util/connection.py", line 73, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/opt/vertica/oss/python3/lib/python3.9/socket.py", line 953, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/botocore/httpsession.py", line 323, in send
urllib_response = conn.urlopen(
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/urllib3/util/retry.py", line 507, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/urllib3/packages/six.py", line 770, in reraise
raise value
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
conn.connect()
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/urllib3/connection.py", line 353, in connect
conn = self._new_conn()
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/urllib3/connection.py", line 181, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7f3abaeadcd0>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/vertica/bin/vbr", line 4686, in _exists
self._client.head_object(Bucket=self._bucket, Key=self._rmtpath(remote_file))
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/botocore/client.py", line 386, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/botocore/client.py", line 691, in _make_api_call
http, parsed_response = self._make_request(
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/botocore/client.py", line 711, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/botocore/endpoint.py", line 156, in _send_request
raise exception
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/botocore/endpoint.py", line 200, in _do_get_response
http_response = self._send(request)
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/botocore/endpoint.py", line 269, in _send
return self.http_session.send(request)
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/botocore/httpsession.py", line 352, in send
raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://s3.us-my-minio-region.amazonaws.com/acid-backup/backup_and_restore/acid.my-minio.net/dw_backup/db/.identity"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/vertica/bin/vbr", line 12145, in main
if vbr_task.run():
File "/opt/vertica/bin/vbr", line 9656, in run
self._initialize_lock()
File "/opt/vertica/bin/vbr", line 9649, in _initialize_lock
self._backup_locker.check_and_regenerate_identity_files(
File "/opt/vertica/bin/vbr", line 11438, in check_and_regenerate_identity_files
if cloud_storage_access.exists(identity_file):
File "/opt/vertica/bin/vbr", line 4040, in exists
return self._exists(remote_file)
File "/opt/vertica/bin/vbr", line 4694, in _exists
raise VbrS3Error(str(ex))
VbrS3Error: Error accessing remote storage: Could not connect to the endpoint URL: "https://s3.my-minio-region.amazonaws.com/acid-backup/backup_and_restore/acid.my-minio.net/dw_backup/db/.identity": returncode=None

2023-01-05 03:21:47 localhost vbr Error: Error accessing remote storage: Could not connect to the endpoint URL: "https://s3.my-minio-region.amazonaws.com/acid-backup/backup_and_restore/acid.my-minio.net/dw_backup/db/.identity": returncode=None
Init FAILED.

`

Tagged:

Answers

  • HibikiHibiki Employee

    Can you check whether the environment variables have been set correctly?

    $ env | grep VBR
    
  • SruthiASruthiA Employee

    In the description, you have mentioned that it is minio. But why is the URL having s3.amazonaws.com in the error?

    VbrS3Error: Error accessing remote storage: Could not connect to the endpoint URL: "https://s3.my-minio-region.amazonaws.com/acid-backup/backup_and_restore/acid.my-minio.net/dw_backup/db/.identity": returncode=None

  • edited January 5

    Hi @Hibiki , it looks like they are correct:
    [[email protected] ~]$ env | grep VBR
    VBR_COMMUNAL_STORAGE_ACCESS_KEY_ID=2...Z
    VBR_BACKUP_STORAGE_ACCESS_KEY_ID=2...Z
    VBR_COMMUNAL_STORAGE_SECRET_ACCESS_KEY=B...W
    VBR_COMMUNAL_STORAGE_ENDPOINT_URL=https://my-minio-server:9000
    VBR_BACKUP_STORAGE_ENDPOINT_URL=https://my-minio-server:9000
    VBR_BACKUP_STORAGE_SECRET_ACCESS_KEY=B...W

  • Hi @SruthiA , that is the problem... I am not specifying s3.amazonaws.com anywhere.

  • SruthiASruthiA Employee

    can you share me the exact vertica version? Can you set cloud_storage_encrypt_transport to false and try init? based on the docs it looks we dont support that option for minio

    select version();

    cloud_storage_encrypt_transport : Boolean, if set to true uses SSL encryption to encrypt data moving between your Vertica cluster and your cloud storage instance.

    You must set this parameter to true if backing up or restoring from:

    Amazon EC2 cluster
    Google Cloud Storage (GCS)
    Eon Mode on-premises database with communal storage on HDFS, to use wire encryption.

    https://www.vertica.com/docs/12.0.x/HTML/Content/Authoring/AdministratorsGuide/BackupRestore/ConfigFileReference/CloudStorage.htm

  • If it helps, here's the results from vertica --version:
    vertica(v12.0.1-0) built by @re-docker3 from releases/[email protected]c555f47876e8 on 'Wed Aug 24 19:06:17 2022' $BuildId$

  • SruthiASruthiA Employee

    thank you. Based on the docs, cloud_storage_encrypt_transport is not applicable for minio.. since it is s3 bucket, I think URL is getting s3.amazonaws.com automatically as cloud_storage_encrypt_transport set to true... Please try init by setting it to false as mentioned earlier.

  • Hi @SruthiA ,
    I tried modified the setting and re-ran but am getting the exact same results:
    VbrS3Error: Error accessing remote storage: Could not connect to the endpoint URL: "https://s3.my-minio-region.amazonaws.com/acid-backup/backup_and_restore/acid.my-minio.net/dw_backup/db/.identity": returncode=None

    2023-01-05 17:22:21 localhost vbr Error: Error accessing remote storage: Could not connect to the endpoint URL: "https://s3.my-minio-region.amazonaws.com/acid-backup/backup_and_restore/acid.my-minio.net/dw_backup/db/.identity": returncode=None

    It is very confusing that it is not taking the URL I set in the environment.

    Also, just for completeness, here's the select output from vsql:
    default=> select version();

    version

    Vertica Analytic Database v12.0.1-0
    (1 row)

  • SruthiASruthiA Employee

    can you share me the full ini file? also in your EON database, is awsenablehttps set to 1?

  • edited January 5

    Hi @SruthiA ,

    I have attached the ini file but obscured one variable. (also, I had to rename to .ini.txt as the forum would not allow me to attach a .ini fil)

    Also, yes, awsenablehttps is set to 1 as the endpoint is https.
    Note, Vertica is correctly connected to MinIO for communal storage and operating properly from that perspective.

  • Sorry... before was an empty file.

  • SruthiASruthiA Employee
    edited January 5

    I just tested and it works fine for me.. only difference is I don't have tls enabled for accessing minio.. Hence set endpoint URL with http

    export VBR_BACKUP_STORAGE_ACCESS_KEY_ID=
    export VBR_BACKUP_STORAGE_SECRET_ACCESS_KEY=
    export VBR_COMMUNAL_STORAGE_ACCESS_KEY_ID=
    export VBR_COMMUNAL_STORAGE_SECRET_ACCESS_KEY=
    export VBR_BACKUP_STORAGE_ENDPOINT_URL=http://x.x.x.x:9000/
    export VBR_COMMUNAL_STORAGE_ENDPOINT_URL=http://x.x.x.x:9000/

    [[email protected] ~]$ /opt/vertica/bin/vbr --task init --s3-force-init -c backup.ini --debug 3
    Initializing backup locations.
    Backup locations initialized.

    [[email protected] ~]$ cat backup.ini
    [CloudStorage]
    cloud_storage_backup_path = s3://dbbackup/
    cloud_storage_backup_file_system_path = []:/home/dbadmin
    cloud_storage_concurrency_backup = 10
    cloud_storage_concurrency_restore = 10
    cloud_storage_encrypt_transport = False

    [Misc]
    snapshotName = dw_bkp
    tempDir = /tmp/vbr
    restorePointLimit = 10
    enableFreeSpaceCheck = True

    [Database]
    dbName = eonv1202u
    dbPromptForPassword = True

    [[email protected] ~]$ vsql
    Welcome to vsql, the Vertica Analytic Database interactive terminal.

    Type: \h or \? for help with vsql commands
    \g or terminate with semicolon to execute query
    \q to quit

    eonv1202u=> select version();

    version

    Vertica Analytic Database v12.0.2-0
    (1 row)

  • SruthiASruthiA Employee

    since you have tls enabled for minio, try including the ca cert used for communicating with communal for cloud_storage_ca_bundle parameter in the ini file and see if it helps.

  • Hi @SruthiA ,

    Apologies for the delayed response.
    I did try with tls disabled, and getting a different error:
    VbrS3Error: Error accessing remote storage: An error occurred (400) when calling the HeadObject operation: Bad Request: returncode=400

    2023-01-05 21:22:00 localhost vbr Error: Error accessing remote storage: An error occurred (400) when calling the HeadObject operation: Bad Request: returncode=400
    Init FAILED.

    So, I'll try your suggestion in the morning (pulling together the cert bundle) and update the thread.

    Just unclear as to why I did not need to provide the cert bundle for Eon mode usage pattern.

    Thank you!

  • SruthiASruthiA Employee
    edited January 5

    I got that same error when I used wrong port number for minio URL. by mistake I used 9090 instead of 9000. Once I corrected it.. init worked.. so please check port number as well as the ca bundle..

    /opt/vertica/bin/vbr --task init --s3-force-init -c backup.ini --debug 3
    Error: Error accessing remote storage: An error occurred (400) when calling the HeadObject operation: Bad Request: returncode=400
    Init FAILED.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file

Can't find what you're looking for? Search the Vertica Documentation, Knowledge Base, or Blog for more information.