Database Startup issue
Hello everyone,
I have 3 nodes DB on Vertica 9.2 on two ESXi Servers,
Few days ago i found two of nodes was gone (It was due to disk issue)
So i restart those nodes (VM) but vertica didn't start
So i tried to do rollback to LGE, but i didn't help me:
Also i tired to rollback from admintools gui, and i got strange message (first epoch greater than 2nd):
so I Started DB in unsafe mode:
and then i did what @skamat said in this topic:
https://forum.vertica.com/discussion/comment/238515/#Comment_238515
but i don't have any projection that has CPE lower than AHM epoch now.
My DB is only start in unsafe mode now.
and in this case i can't do MAKE_AHM_NOW:
Please help me to fix it.
Leave a Comment
Can't find what you're looking for? Search the Vertica Documentation, Knowledge Base, or Blog for more information.
Answers
From the recovery cleark node can you provide the ASR detail information.
Please let me know how can i provide it?
Is it possible to attach the vertica.log from all the 3 nodes? I can get that info once we have the log.
Thanks @Nimmi_gupta
Please find the attached log
1) As per vertica.log from Node3 (you have shared), there is a warning for LGE < AHM.
LGE=0x18a90ae --> 25858222
AHM=0x18a90b4 --> 25858228
From vertica.log
2019-11-16 10:39:10.849 DistCall Dispatch:7fb0c4d06700 [Recover] LGE=0x18a90ae, AHM=0x18a90b4, LGE < AHM. Data may be corrupted.
2019-11-16 10:39:10.849 DistCall Dispatch:7fb0c4d06700 [Main] Writing epoch=0x18a90ae, ending at '2019-11-10 10:35:03.523396+03:30', catalog version=0x40453eb, K-safety=0, AHM=0x18a90b4, ending at '2019-11-10 10:35:03.523396+03:30', to epoch log file [/vertica/DB/v_yyyy_node0003_catalog/Epoch.log]
Can you start the database in unsafe mode and run the below query
select projection_name, is_up_to_date,checkpoint_epoch,would_recover,is_behind_ahm from v_catalog.projection_checkpoint_epochs where is_behind_ahm;
select get_current_epoch(), get_last_good_epoch(), get_ahm_epoch(), get_ahm_time(), sysdate;
Thanks Nimmi and sorry for the late reply,
First query didn't returned any result and second one was:
In recent days i exported all the tables and schema manually (in unsafe mode) and rebuild database (because i needed to fix it asap).
I faced this issue for 3rd time in last month, I think VERTICA should workaround it.