Backup failing: vbr reports "Error: Missing critical file: [IP]:/path//Snapshots/.../[file].manifest

Vertica Backup Failing

 

Hi

 

I have inherited a database from a DBA that is no longer with the company, and one of the known issues is the lack of backup.

 

Node 1 has a NAS location (RAID 0+1) for the backup destination, which it serves as an NFS mount to the rest of the nodes.  Performance has been checked using vioperf and it is excellent from all nodes in the cluster.

 

However, after successfully initializing the snapshot using "-t init", when I run the object level backup, it returns the following error:

 

$ vbr -t backup -c obj_bkp_husvm00-0.ini

Warning: config mapping is missing entries for nodes: (list of STANDBY nodes being ignored)

Starting backup of database PROD.

Objects: ['SCHEMA.tablename']

Participating nodes: (list of PERMANENT nodes to be backed up)

Enter vertica password:

Snapshotting database.

Snapshot complete.

Error: Missing critical file: [IP-address]:/path//Snapshots/obj_bkp_001_(timestamp)/v_PROD_node0018/obj_bkp_001.manifest

Backup FAILED.

 

If I go to the NAS mount point path on node 1, I don't see any directory named "Snapshots" - ignoring the weird "//" in the location.

 

Going to the node referenced by the IP-address (node 18), I also do not see any directory named Snapshots.

 

I assume this indicates that perhaps there must have been a failed backup in the past, which damaged the manifest, which for whatever reason is hidden from me?

 

How do I resolve/fix this?

 

Thanks in advance,

 

TH

Comments

  • Hi 

     

    What version are running in? If your cluster version is greater than 7.2.x version I beleive you can run below full-check command to find if your backup doesn't have any unreferenced/missing objects left from any previous partial/failed backup.

     

    $ /opt/vertica/bin/vbr.py -t full-check -c <config_file_name.ini> --report-file=/tmp/backupquickcheck.json

    Request you to kindly run below two new switches/checks (quick-repair and collect-garbage) that have been newly introduced from 7.2.2-2 version and perform below task that can help you resolve this issue.

    The quick-repair task rebuilds the backup manifest based on the manifests contained in the backup location.

    $ vbr -t quick-repair -c <config_file_name.ini>


    Another command after that you need to run is mentioned as below which is with collect-garbage switch. The task of the same is to rebuild your backup manifest and deletes any backup objects that do not appear in the manifest.

    Use the following form to perform a garbage collection task:

    $vbr -t collect-garbage -c <config_file_name.ini> --report-file=/tmp/collect-garbage.json


    Please use below documentation link for detailed information on the same.

    https://my.vertica.com/docs/7.2.x/HTML/index.htm#Authoring/AdministratorsGuide/BackupRestore/RepairingBackups.htm

     

    Than

    Rahul

  • Unfortunately, we are running version 7.2.1-4, which apparently does not have any of those options available to us.

     

    Is it normal to have two slashes in a path name?  How does the OS interpret a file reference like that?

  • SruthiASruthiA Vertica Employee Administrator

    Was the back up successful in past? Can you share me your ini file

     

     

    Sruthi

  • Hi,

     

    Apparently there was some kind of corruption in the target storage.

     

    After I deleted everything in the destination directories, and re-initialized (vbr.py -t init --config-file file.ini), the backup ran successfully, and we have not had any issues since then.

     

    Thank you to all who chimed in!

     

    Thor

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file