The Vertica Forum recently got a makeover! Let us know what you think by filling out this short, anonymous survey.

Vertica is failing to start

/vertica/data/data/unified_db/v_unified_db_node0001_catalog/Epoch.log

Last good epoch: 0x2c25d ended at '2017-01-16 13:05:48.039817-05'

Last good catalog version: 0x32139

 

Vertical is failing to start:

 

we tried to start with last good epoch:

 

/opt/vertica/bin/admintools --tool restart_db  -d unified_db -p XXX -e last

 

Info: Using default last good epoch

*** Restarting database unified_db at '0' epoch 0 ***

Unable to read database catalogs - cannot start database.

Database startup at '0' (epoch 0) failed. Contact Vertica Technical Support

Comments

  • Let me explain what's going on with single node Vertica DB example for simplicity.

    Single node Vertica DB can run into this situation, if a node had some kind of issue with a ROS containers saved on disk. Vertica checks for validity of each ROS container at startup and may find one ROS container to be missing or corrupted. If Vertica find ROS container missing , it will mark checkpoint epoch of the projection having missing ROS container to epoch one less than start epoch of missing ROS container. LGE epoch of a node is min of checkpoint epoch of projections on that node and above situation will pull LGE of node back too.

    Vertica does not allow recovery of to epoch prior to AHM(ancient history mark). Vertica does not keep map of epoch and timestamps for epochs prior to AHM epoch. If node recovery is suggesting LGE to be older than AHM epoch , vertica has not way to map this to timestamp and for that reason , you are seeing recovery epoch to 0.

    How to fix this situation in database after Version 7.2.x:

    1. You can start database in unsafe mode using following command. Unsafe mode lets you query vertica tables but no DDL and DML.

    admintools -t start_db -d -U

    1. Query projection_checkpoint_epochs table to find projection that has CPE lower than AHM epoch. You can use get_ahm_epoch() api to find AHM epoch.

    2. Disable recovery for table/projection having issue by running following command
      SELECT do_tm_task('abortrecovery','

    <

    table name>');

    1. Shutdown database and restart db without -U flag.

    IMPORTANT: You need to know that with above steps you have told vertica to not recover a specific projection and table. This table will have data consistency issues and it is recomended to recreate this table by loading data from source files or by copying data from corrupted table.

    Hope this helps ..

  • Thank you for this instruction. It help us today.

    Today part of our servers were randomly shutdown while cluster recovery. Due to electricity. Catalog became inconsistent and each node had each own LGE that was less than AHM. Cluster didn't start nor by force option, nor by rollbacking to last LGE.

    It would be great if such technical points would be reflected in the documentation.

  • How to disable abortrecovery so that on next restart vertica properly recovers these projections/tables ?

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file

Can't find what you're looking for? Search the Vertica Documentation, Knowledge Base, or Blog for more information.