vbr backups deletion issue - vertica 9.3

p_kovalenko

Hello.
I am doing daily backups via vbr tool, with restorePointLimit=7. However, sometimes old backups can not be deleted (but sometimes it can delete it). For example, now I have 13th and 15th October backups remaining, but 14th was deleted.

When I try to remove backup manually via vbr, it shows me error:

vbr -t remove -c /u01/vertica_backup/vbr_full_backup.ini --archive 20201030_230106  --debug 3
Removing restore points: 20201030_230106
Error: Trying to delete untracked object: 029645f15d46e69ab9a75d6b80db404d00a00000066fa8e9.  Run backup validation.
Remove FAILED.

And when I try to do garbadge collection, it also ends up with error:

vbr -t collect-garbage -c /u01/vertica_backup/vbr_full_backup.ini --debug 3
Deleting unreferenced objects
Regenerating backup manifest for location [10.178.33.193]:/u01/vertica_backup/full_backup
Error: Error accessing remote storage: failed accessing remote storage on <IP_ADDR>: ssh_exchange_identification: read: Connection reset by peer
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(601) [Receiver=3.0.7]
: returncode=None
Collect-garbage FAILED.

And it is interesting, that in log file I can see rsync operations performing successfully, and after some time - rsync error:

File "/opt/vertica/bin/vbr", line 3788, in _run_rsync
 "failed accessing remote storage on %s: %s" % (remote_addr, err), proc.returncode)
VbrRsyncError: Error accessing remote storage: failed accessing remote storage on <IP_ADDR>: ssh_exchange_identification: read: Connection reset by     peer^M
 rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
  rsync error: unexplained error (code 255) at io.c(601) [Receiver=3.0.7]
   : returncode=None

So I have no idea what to do to fix this issue, or even how to remove it manually/using bash script. Maybe you can help me.

Nimmi_gupta

@p_kovalenko
What's the vertica version?
As per the first error message you have shared "Error: Trying to delete untracked object" looks like a known bug and fix is available in 10.0SP1
vbr -t remove -c /u01/vertica_backup/vbr_full_backup.ini --archive 20201030_230106 --debug 3
Removing restore points: 20201030_230106
Error: Trying to delete untracked object: 029645f15d46e69ab9a75d6b80db404d00a00000066fa8e9. Run backup validation.
Remove FAILED.
Check the release notes:
https://www.vertica.com/docs/ReleaseNotes/10.0.x/Vertica_10.0.x_Release_Notes.htm
VER-71997 : Backup/DR Vertica backups were failing to delete old restore points when the number of backup objects exceeded 10,000 objects. This issue has been fixed.

p_kovalenko

@Nimmi_gupta
Thank you for the reply.
I am counting restore point objects by checking corresponding snapshot manifest file and counting lines of this file. Am I conting right?
If i am understand counting objects right, yes, I usually have more than 10 000 objects in one restore point.

SruthiA

Is SSH working fine? Can you share me the contents of /etc/ssh/sshd_config?

VbrRsyncError: Error accessing remote storage: failed accessing remote storage on : ssh_exchange_identification: read: Connection reset by peer^M