Vertica recovery "RollBack Database to Last Good Epoch " failed

Felix_Huang

Hello, 

 

Our Vertica 7.2.0 has 6 nodes, 2 nodes were down, and restart the DB the other 4 nodes are always INITIALIZING. 

I run admintools -> 7.Advanced Menu -> 1.RollBack Database to Last Good Epoch,  met error: 

 

*** Restarting database Future_State_POC at '2015-12-22 08:38:35.257259+00' epoch 38337 ***

..

adminTools Last Chance Error Handler running...
raised error: <type 'exceptions.TypeError'>
error message: 'NoneType' object has no attribute '__getitem__'
trace file: /opt/vertica/log/adminTools.errors
REPORT THIS INFORMATION TO TECHNICAL SUPPORT
AND INCLUDE CONTENTS OF THE TRACE FILE IN YOUR REPORT

 

/opt/vertica/log/adminTools.errors  content: 

 

2015-12-23 07:56:53.178 admintools/8645:0x7fefe68ad700 [__main__.<module>] <CRITICAL> --------------------------------
2015-12-23 07:56:53.179 admintools/8645:0x7fefe68ad700 [__main__.<module>] <CRITICAL> raisedError
2015-12-23 07:56:53.180 admintools/8645:0x7fefe68ad700 [__main__.<module>] <CRITICAL> Traceback (most recent call last):
File "/opt/vertica/bin/admintools", line 405, in <module>
ac.runApp()
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/ui/adminCtrl.py", line 3161, in runApp
self.__ui.runFromNavigation("main")
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/ui/uiMgr.py", line 165, in a_wrapper_accepting_args
fn(self, *args, **kwargs)
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/ui/uiMgr.py", line 450, in runFromNavigation
self.__navigator.navigate(uimanager=self, startingPoint=runFrom)
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/ui/Navigator.py", line 336, in navigate
((self.__menus[ nextMenu ][ self.NAV_MACTIONS ])[ offset ])()
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/ui/adminCtrl.py", line 322, in restartDBFromEpoch
result, theMessage = self.__exec.dbStart(DBName=results[0], DBpassword=DBpassword, epoch=lastEpoch)
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/engine/adminExec.py", line 800, in dbStart
log=r.content["db_log_path"]))
TypeError: 'NoneType' object has no attribute '__getitem__'

 

Wish help to recover the Vertica DB. Thanks a lot.

 

Felix_Huang

Problem solved, due to unusual status of 2 nodes, can't write files caused DB can't start, just reboot them to recover. 

 

*** Restarting hosts for database Future_State_POC ***

        restart host 15.48.44.136 with catalog v_future_state_poc_node0005_catalog

A system-related error has occurred.  For more information

see the error number returned below. Exiting.

[Errno 30] Read-only file system: '/tmp/incoming-5e45d610-7901-43f6-844a-fa8a6cb67969'

[dbadmin@c4t17224 log]$ cd /tmp

[dbadmin@c4t17224 tmp]$ touch x

touch: cannot touch `x': Read-only file system