Configure vbr to backup to MinIO

jordzillajordzilla Vertica Customer
in General Discussion

I have successfully configured Vertica 12 to use an on-prem MinIO server in Eon Mode for communal storage.

Now I am trying to configure vbr to backup to another bucket on the same server.

No matter what I do, vbr consistently tries to target the bucket on https://s3.my-minio-region.amazonaws.com.

Any help with pointing me in the right direction is appreciated.

Here's the command:
/opt/vertica/bin/vbr --task init --s3-force-init -c ./backup.ini --debug 3

I have set the following keys in the environment before calling vbr:

  • VBR_BACKUP_STORAGE_ACCESS_KEY_ID="..."
  • VBR_BACKUP_STORAGE_SECRET_ACCESS_KEY="..."
  • VBR_BACKUP_STORAGE_ENDPOINT_URL="https://minio-server:9000"
  • VBR_COMMUNAL_STORAGE_ACCESS_KEY_ID="..."
  • VBR_COMMUNAL_STORAGE_SECRET_ACCESS_KEY="..."
  • VBR_COMMUNAL_STORAGE_ENDPOINT_URL="https://minio-server:9000"

In the backup.ini file I am specifying in the [CloudStorage] section the following keys:

  • cloud_storage_backup_path = s3://acid-backup/backup_and_restore/acid.my-minio.net/dw_backup/db
  • cloud_storage_backup_file_system_path = []:/data/backup/data/
  • cloud_storage_encrypt_transport = True
  • cloud_storage_concurrency_backup = 10
  • cloud_storage_concurrency_backup = 10

Here's the last bits from the log file (--debug 3 but doesn't seem to add much):

`2023-01-05 03:20:05 localhost local_storage exists /.identity
2023-01-05 03:20:05 localhost s3 exists /.identity
2023-01-05 03:21:47 localhost vbr Traceback (most recent call last):
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
conn = connection.create_connection(
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/urllib3/util/connection.py", line 73, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/opt/vertica/oss/python3/lib/python3.9/socket.py", line 953, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/botocore/httpsession.py", line 323, in send
urllib_response = conn.urlopen(
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/urllib3/util/retry.py", line 507, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/urllib3/packages/six.py", line 770, in reraise
raise value
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
conn.connect()
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/urllib3/connection.py", line 353, in connect
conn = self._new_conn()
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/urllib3/connection.py", line 181, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7f3abaeadcd0>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/vertica/bin/vbr", line 4686, in _exists
self._client.head_object(Bucket=self._bucket, Key=self._rmtpath(remote_file))
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/botocore/client.py", line 386, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/botocore/client.py", line 691, in _make_api_call
http, parsed_response = self._make_request(
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/botocore/client.py", line 711, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/botocore/endpoint.py", line 156, in _send_request
raise exception
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/botocore/endpoint.py", line 200, in _do_get_response
http_response = self._send(request)
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/botocore/endpoint.py", line 269, in _send
return self.http_session.send(request)
File "/opt/vertica/oss/python3/lib/python3.9/site-packages/botocore/httpsession.py", line 352, in send
raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://s3.us-my-minio-region.amazonaws.com/acid-backup/backup_and_restore/acid.my-minio.net/dw_backup/db/.identity"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/vertica/bin/vbr", line 12145, in main
if vbr_task.run():
File "/opt/vertica/bin/vbr", line 9656, in run
self._initialize_lock()
File "/opt/vertica/bin/vbr", line 9649, in _initialize_lock
self._backup_locker.check_and_regenerate_identity_files(
File "/opt/vertica/bin/vbr", line 11438, in check_and_regenerate_identity_files
if cloud_storage_access.exists(identity_file):
File "/opt/vertica/bin/vbr", line 4040, in exists
return self._exists(remote_file)
File "/opt/vertica/bin/vbr", line 4694, in _exists
raise VbrS3Error(str(ex))
VbrS3Error: Error accessing remote storage: Could not connect to the endpoint URL: "https://s3.my-minio-region.amazonaws.com/acid-backup/backup_and_restore/acid.my-minio.net/dw_backup/db/.identity": returncode=None

2023-01-05 03:21:47 localhost vbr Error: Error accessing remote storage: Could not connect to the endpoint URL: "https://s3.my-minio-region.amazonaws.com/acid-backup/backup_and_restore/acid.my-minio.net/dw_backup/db/.identity": returncode=None
Init FAILED.

`

Tagged:

Answers

  • HibikiHibiki Vertica Employee Employee

    Can you check whether the environment variables have been set correctly?

    $ env | grep VBR
  • SruthiASruthiA Administrator

    In the description, you have mentioned that it is minio. But why is the URL having s3.amazonaws.com in the error?

    VbrS3Error: Error accessing remote storage: Could not connect to the endpoint URL: "https://s3.my-minio-region.amazonaws.com/acid-backup/backup_and_restore/acid.my-minio.net/dw_backup/db/.identity": returncode=None

  • jordzillajordzilla Vertica Customer
    edited January 2023

    Hi @Hibiki , it looks like they are correct:
    [dbadmin@acid-db ~]$ env | grep VBR
    VBR_COMMUNAL_STORAGE_ACCESS_KEY_ID=2...Z
    VBR_BACKUP_STORAGE_ACCESS_KEY_ID=2...Z
    VBR_COMMUNAL_STORAGE_SECRET_ACCESS_KEY=B...W
    VBR_COMMUNAL_STORAGE_ENDPOINT_URL=https://my-minio-server:9000
    VBR_BACKUP_STORAGE_ENDPOINT_URL=https://my-minio-server:9000
    VBR_BACKUP_STORAGE_SECRET_ACCESS_KEY=B...W

  • jordzillajordzilla Vertica Customer

    Hi @SruthiA , that is the problem... I am not specifying s3.amazonaws.com anywhere.

  • SruthiASruthiA Administrator

    can you share me the exact vertica version? Can you set cloud_storage_encrypt_transport to false and try init? based on the docs it looks we dont support that option for minio

    select version();

    cloud_storage_encrypt_transport : Boolean, if set to true uses SSL encryption to encrypt data moving between your Vertica cluster and your cloud storage instance.

    You must set this parameter to true if backing up or restoring from:

    Amazon EC2 cluster
    Google Cloud Storage (GCS)
    Eon Mode on-premises database with communal storage on HDFS, to use wire encryption.

    https://www.vertica.com/docs/12.0.x/HTML/Content/Authoring/AdministratorsGuide/BackupRestore/ConfigFileReference/CloudStorage.htm

  • jordzillajordzilla Vertica Customer

    If it helps, here's the results from vertica --version:
    vertica(v12.0.1-0) built by @re-docker3 from releases/VER_12_0_RELEASE_BUILD_1_0_20220824@8fd47a18e3db04a072e8e24ccb07c555f47876e8 on 'Wed Aug 24 19:06:17 2022' $BuildId$

  • SruthiASruthiA Administrator

    thank you. Based on the docs, cloud_storage_encrypt_transport is not applicable for minio.. since it is s3 bucket, I think URL is getting s3.amazonaws.com automatically as cloud_storage_encrypt_transport set to true... Please try init by setting it to false as mentioned earlier.

  • jordzillajordzilla Vertica Customer

    Hi @SruthiA ,
    I tried modified the setting and re-ran but am getting the exact same results:
    VbrS3Error: Error accessing remote storage: Could not connect to the endpoint URL: "https://s3.my-minio-region.amazonaws.com/acid-backup/backup_and_restore/acid.my-minio.net/dw_backup/db/.identity": returncode=None

    2023-01-05 17:22:21 localhost vbr Error: Error accessing remote storage: Could not connect to the endpoint URL: "https://s3.my-minio-region.amazonaws.com/acid-backup/backup_and_restore/acid.my-minio.net/dw_backup/db/.identity": returncode=None

    It is very confusing that it is not taking the URL I set in the environment.

    Also, just for completeness, here's the select output from vsql:
    default=> select version();

    version

    Vertica Analytic Database v12.0.1-0
    (1 row)

  • SruthiASruthiA Administrator

    can you share me the full ini file? also in your EON database, is awsenablehttps set to 1?

  • jordzillajordzilla Vertica Customer
    edited January 2023

    Hi @SruthiA ,

    I have attached the ini file but obscured one variable. (also, I had to rename to .ini.txt as the forum would not allow me to attach a .ini fil)

    Also, yes, awsenablehttps is set to 1 as the endpoint is https.
    Note, Vertica is correctly connected to MinIO for communal storage and operating properly from that perspective.

  • jordzillajordzilla Vertica Customer

    Sorry... before was an empty file.

  • SruthiASruthiA Administrator
    edited January 2023

    I just tested and it works fine for me.. only difference is I don't have tls enabled for accessing minio.. Hence set endpoint URL with http

    export VBR_BACKUP_STORAGE_ACCESS_KEY_ID=
    export VBR_BACKUP_STORAGE_SECRET_ACCESS_KEY=
    export VBR_COMMUNAL_STORAGE_ACCESS_KEY_ID=
    export VBR_COMMUNAL_STORAGE_SECRET_ACCESS_KEY=
    export VBR_BACKUP_STORAGE_ENDPOINT_URL=http://x.x.x.x:9000/
    export VBR_COMMUNAL_STORAGE_ENDPOINT_URL=http://x.x.x.x:9000/

    [dbadmin@node1 ~]$ /opt/vertica/bin/vbr --task init --s3-force-init -c backup.ini --debug 3
    Initializing backup locations.
    Backup locations initialized.

    [dbadmin@node1 ~]$ cat backup.ini
    [CloudStorage]
    cloud_storage_backup_path = s3://dbbackup/
    cloud_storage_backup_file_system_path = []:/home/dbadmin
    cloud_storage_concurrency_backup = 10
    cloud_storage_concurrency_restore = 10
    cloud_storage_encrypt_transport = False

    [Misc]
    snapshotName = dw_bkp
    tempDir = /tmp/vbr
    restorePointLimit = 10
    enableFreeSpaceCheck = True

    [Database]
    dbName = eonv1202u
    dbPromptForPassword = True

    [dbadmin@node1 ~]$ vsql
    Welcome to vsql, the Vertica Analytic Database interactive terminal.

    Type: \h or \? for help with vsql commands
    \g or terminate with semicolon to execute query
    \q to quit

    eonv1202u=> select version();

    version

    Vertica Analytic Database v12.0.2-0
    (1 row)

  • SruthiASruthiA Administrator

    since you have tls enabled for minio, try including the ca cert used for communicating with communal for cloud_storage_ca_bundle parameter in the ini file and see if it helps.

  • jordzillajordzilla Vertica Customer

    Hi @SruthiA ,

    Apologies for the delayed response.
    I did try with tls disabled, and getting a different error:
    VbrS3Error: Error accessing remote storage: An error occurred (400) when calling the HeadObject operation: Bad Request: returncode=400

    2023-01-05 21:22:00 localhost vbr Error: Error accessing remote storage: An error occurred (400) when calling the HeadObject operation: Bad Request: returncode=400
    Init FAILED.

    So, I'll try your suggestion in the morning (pulling together the cert bundle) and update the thread.

    Just unclear as to why I did not need to provide the cert bundle for Eon mode usage pattern.

    Thank you!

  • SruthiASruthiA Administrator
    edited January 2023

    I got that same error when I used wrong port number for minio URL. by mistake I used 9090 instead of 9000. Once I corrected it.. init worked.. so please check port number as well as the ca bundle..

    /opt/vertica/bin/vbr --task init --s3-force-init -c backup.ini --debug 3
    Error: Error accessing remote storage: An error occurred (400) when calling the HeadObject operation: Bad Request: returncode=400
    Init FAILED.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Format
SpoilerCodeQuote
Heading 2Heading 1
Emoji
:):D:(;):/:o:s:p:'(:|B):#:*<3o:)>:)
Url
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file
Home • General DiscussionComment As ...