Error starting database, no nodes are up
i have three node cluster. now unable to start database i am getting the error
Error starting database, no nodes are up
how we can resolve this?
Error starting database, no nodes are up
how we can resolve this?
0
Comments
1- vertica.log that you will find in the <<database>>/v_<<database>>_catalog/
2- check the dblog that you can find in the database folder.
Hope this help,
Eugenia
I am posting last part of the log files
Vertica.log
2014-04-11 10:25:28.859 Main:0x4b92680 [SAL] <INFO> Queueing directory [/catalog/ubossdb/v_ubossdb_node0001_data/371/45035997392014371.del.del.del.del.del.del.del.del.del.del] for removal2014-04-11 10:25:28.863 Main:0x4b92680 [SAL] <INFO> Queueing directory [/catalog/ubossdb/v_ubossdb_node0001_data/087/45035997392014087.del.del.del.del.del.del.del.del.del.del] for removal
2014-04-11 10:25:28.864 Main:0x4b92680 [SAL] <INFO> Queueing directory [/catalog/ubossdb/v_ubossdb_node0001_data/087/45035997392012087.del.del.del.del.del.del.del.del.del.del] for removal
2014-04-11 10:25:28.992 Main:0x4b92680 [Catalog] <INFO> Checking 23947 files in 2 threads
2014-04-11 10:25:41.957 Main:0x4b92680 [Recover] <INFO> Loading UDx libraries
2014-04-11 10:25:41.957 Main:0x4b92680 [Recover] <INFO> Setting up UDx pointers
2014-04-11 10:25:41.961 Main:0x4b92680 <PANIC> @v_ubossdb_node0001: VX001/2973: Data consistency problems found; startup aborted
HINT: Check that all file systems are properly mounted. Also, the --force option can be used to delete corrupted data and recover from the cluster
LOCATION: mainEntryPoint, /scratch_a/release/vbuild/vertica/Basics/vertica.cpp:1166
2014-04-11 10:25:42.092 Main:0x4b92680 [Main] <PANIC> Wrote backtrace to ErrorReport.txt
dbLog
04/11/14 10:24:51 SP_connect: DEBUG: Auth list is: NULL
Let me know how many nodes do you have.
Eugenia
node2
Data consistency problems found; startup aborted
HINT: Check that all file systems are properly mounted. Also, the --force option can be used to delete corrupted data and recover from the cluster
LOCATION: mainEntryPoint, /scratch_a/release/vbuild/vertica/Basics/vertica.cpp:1166
2014-04-11 10:25:48.808 Main:0x66eb680 [Main] <PANIC> Wrote backtrace to ErrorReport.txt
node3
2014-04-11 10:25:43.466 Main:0x5851680 [Recover] <INFO> Loading UDx libraries2014-04-11 10:25:43.466 Main:0x5851680 [Recover] <INFO> Setting up UDx pointers
2014-04-11 10:25:43.471 Main:0x5851680 <PANIC> @v_ubossdb_node0003: VX001/2973: Data consistency problems found; startup aborted
HINT: Check that all file systems are properly mounted. Also, the --force option can be used to delete corrupted data and recover from the cluster
LOCATION: mainEntryPoint, /scratch_a/release/vbuild/vertica/Basics/vertica.cpp:1166
2014-04-11 10:25:43.593 Main:0x5851680 [Main] <PANIC> Wrote backtrace to ErrorReport.txt
Try to restart using admintools the last good epoch. In admintools go to advance menu/rollback to the last epoch, see what number it gives you.
Let me know.
Eugenia
i can see the files data directory.
Last epoch number is 0.
Najeeb
Anybody can help me?
Thanks,
Najeeb
Unfortunately, it sounds like you have lost your data. Vertica is unable to recover a consistent copy of your database; the newest complete, not-corrupt version of the database that is available is from epoch 0, ie., a new database.
Running three separate nodes on VMware, especially if they're on the same underlying physical hardware (?), is a risky proposition. If your disk stops working, or if the underlying operating system is ignoring requests in the VM to flush data to disk, etc., then you can easily lose data; sometimes a lot of data.
It may be possible to extract some data from the internal data files; or to drop corrupt tables and enable the system to restore to a consistent view for just some of your tables. Unfortunately, I don't think that this process is currently documented, nor is it automated; it can end up involving a bunch of low-level hacking of the internals of the files inside the database. Not something (in the general case) that would be amenable to a simple walk-through. For folks with a support contract, this is a case where our support folks could walk you through the process. Short of that, maybe there are third-party consultants out there who have figured out the process and would be willing to help you through it, to see how much data is recoverable? The alternative would be to start over and restore from a backup.
Adam
Hacking on the internals of Vertica's data store is very much a "under-Support-guidance-only" type of thing. If you do something wrong, you can easily make things worse rather than better. That said, if you don't have access to Support, and you have nothing to lose / can't get at your data anyway, and someone has some ideas...