vbr remove from s3 fails

ramki

Hi, We have added a new node over the weekend and vbr is failing to remove the backup taken prior to adding the new node, tried to remove it manually it fails with S3 error, I have never seen this error before.
[dbadmin@analytics-gold-prod-vertica-1 ~]$ vbr -t remove -c /home/dbadmin/backups/vertica_backup.ini --archive 20210319_170128
Removing restore points: 20210319_170128
Error: Error accessing remote storage: When deleting files from S3. An error occurred (SlowDown) when calling the DeleteObjects operation (reached max retries: 10): Please reduce your request rate. : returncode=None
Remove FAILED.
After this quick-check fails, have to run quick-repair to fix the manifest file, I am exactly sure what is going on, I could access the S3 bucket without any issues from the vertica host.
We are running vertica 9.2.1-18
As you can see below that one I am trying to remove with 4 nodes is the backup taken prior to adding the 5th node.
[dbadmin@analytics-gold-prod-vertica-1 ~]$ vbr -t quick-check -c /home/dbadmin/backups/vertica_backup.ini
Checking backup consistency.
List all snapshots in backup location:
Snapshot name and restore point: fullBackup_20210323_033003, nodes:[u'v_carezen_node0001', u'v_carezen_node0002', u'v_carezen_node0003', u'v_carezen_node0004', u'v_carezen_node0005'].
Snapshot name and restore point: fullBackup_20210319_170128, nodes:[u'v_carezen_node0001', u'v_carezen_node0002', u'v_carezen_node0003', u'v_carezen_node0004'].
Snapshot name and restore point: fullBackup_20210322_033004, nodes:[u'v_carezen_node0001', u'v_carezen_node0002', u'v_carezen_node0003', u'v_carezen_node0004', u'v_carezen_node0005'].
Snapshot name and restore point: fullBackup_20210321_033003, nodes:[u'v_carezen_node0001', u'v_carezen_node0002', u'v_carezen_node0003', u'v_carezen_node0004', u'v_carezen_node0005'].
Backup consistency check complete.
Regards,
Ramki

SruthiA

@ramki It is a known issue. limitation from AWS. What did you set s3_concurrency_backup and s3_concurrency_restore parameter to in your ini file?
https://aws.amazon.com/premiumsupport/knowledge-center/s3-resolve-503-slowdown-throttling/

SruthiA

@ramki please open a support case. we can work with engineering and get the backups removed. VER-55270 is bug we have it for this internally.

ramki

Thank you both, here are my concurrency settings.
s3_concurrency_backup = 10
s3_concurrency_restore = 10

ramki

Going to open a support ticket now... Thanks

SruthiA

@ramki can you try remove task by setting s3_concurrency_backup and s3_concurrency_restore to 1 ?