Backup failing: vbr reports "Error: Missing critical file: [IP]:/path//Snapshots/.../[file].manifest
Vertica Backup Failing
Hi
I have inherited a database from a DBA that is no longer with the company, and one of the known issues is the lack of backup.
Node 1 has a NAS location (RAID 0+1) for the backup destination, which it serves as an NFS mount to the rest of the nodes. Performance has been checked using vioperf and it is excellent from all nodes in the cluster.
However, after successfully initializing the snapshot using "-t init", when I run the object level backup, it returns the following error:
$ vbr -t backup -c obj_bkp_husvm00-0.ini
Warning: config mapping is missing entries for nodes: (list of STANDBY nodes being ignored)
Starting backup of database PROD.
Objects: ['SCHEMA.tablename']
Participating nodes: (list of PERMANENT nodes to be backed up)
Enter vertica password:
Snapshotting database.
Snapshot complete.
Error: Missing critical file: [IP-address]:/path//Snapshots/obj_bkp_001_(timestamp)/v_PROD_node0018/obj_bkp_001.manifest
Backup FAILED.
If I go to the NAS mount point path on node 1, I don't see any directory named "Snapshots" - ignoring the weird "//" in the location.
Going to the node referenced by the IP-address (node 18), I also do not see any directory named Snapshots.
I assume this indicates that perhaps there must have been a failed backup in the past, which damaged the manifest, which for whatever reason is hidden from me?
How do I resolve/fix this?
Thanks in advance,
TH
Comments
Hi
What version are running in? If your cluster version is greater than 7.2.x version I beleive you can run below full-check command to find if your backup doesn't have any unreferenced/missing objects left from any previous partial/failed backup.
$ /opt/vertica/bin/vbr.py -t full-check -c <config_file_name.ini> --report-file=/tmp/backupquickcheck.json
Request you to kindly run below two new switches/checks (quick-repair and collect-garbage) that have been newly introduced from 7.2.2-2 version and perform below task that can help you resolve this issue.
The quick-repair task rebuilds the backup manifest based on the manifests contained in the backup location.
$ vbr -t quick-repair -c <config_file_name.ini>
Another command after that you need to run is mentioned as below which is with collect-garbage switch. The task of the same is to rebuild your backup manifest and deletes any backup objects that do not appear in the manifest.
Use the following form to perform a garbage collection task:
$vbr -t collect-garbage -c <config_file_name.ini> --report-file=/tmp/collect-garbage.json
Please use below documentation link for detailed information on the same.
https://my.vertica.com/docs/7.2.x/HTML/index.htm#Authoring/AdministratorsGuide/BackupRestore/RepairingBackups.htm
Than
Rahul
Unfortunately, we are running version 7.2.1-4, which apparently does not have any of those options available to us.
Is it normal to have two slashes in a path name? How does the OS interpret a file reference like that?
Was the back up successful in past? Can you share me your ini file
Sruthi
Hi,
Apparently there was some kind of corruption in the target storage.
After I deleted everything in the destination directories, and re-initialized (vbr.py -t init --config-file file.ini), the backup ran successfully, and we have not had any issues since then.
Thank you to all who chimed in!
Thor