Database Startup issue

verbanverban Vertica Customer

Hello everyone,

I have 3 nodes DB on Vertica 9.2 on two ESXi Servers,
Few days ago i found two of nodes was gone (It was due to disk issue)
So i restart those nodes (VM) but vertica didn't start

So i tried to do rollback to LGE, but i didn't help me:

Also i tired to rollback from admintools gui, and i got strange message (first epoch greater than 2nd):

so I Started DB in unsafe mode:

and then i did what @skamat said in this topic:
https://forum.vertica.com/discussion/comment/238515/#Comment_238515
but i don't have any projection that has CPE lower than AHM epoch now.
My DB is only start in unsafe mode now.
and in this case i can't do MAKE_AHM_NOW:

Please help me to fix it.

Tagged:

Answers

  • Nimmi_guptaNimmi_gupta - Select Field - Employee

    From the recovery cleark node can you provide the ASR detail information.

  • verbanverban Vertica Customer

    Please let me know how can i provide it?

  • Nimmi_guptaNimmi_gupta - Select Field - Employee

    Is it possible to attach the vertica.log from all the 3 nodes? I can get that info once we have the log.

  • verbanverban Vertica Customer

    Thanks @Nimmi_gupta
    Please find the attached log

  • Nimmi_guptaNimmi_gupta - Select Field - Employee

    1) As per vertica.log from Node3 (you have shared), there is a warning for LGE < AHM.
    LGE=0x18a90ae --> 25858222
    AHM=0x18a90b4 --> 25858228
    From vertica.log
    2019-11-16 10:39:10.849 DistCall Dispatch:7fb0c4d06700 [Recover] LGE=0x18a90ae, AHM=0x18a90b4, LGE < AHM. Data may be corrupted.
    2019-11-16 10:39:10.849 DistCall Dispatch:7fb0c4d06700 [Main] Writing epoch=0x18a90ae, ending at '2019-11-10 10:35:03.523396+03:30', catalog version=0x40453eb, K-safety=0, AHM=0x18a90b4, ending at '2019-11-10 10:35:03.523396+03:30', to epoch log file [/vertica/DB/v_yyyy_node0003_catalog/Epoch.log]
    Can you start the database in unsafe mode and run the below query
    select projection_name, is_up_to_date,checkpoint_epoch,would_recover,is_behind_ahm from v_catalog.projection_checkpoint_epochs where is_behind_ahm;
    select get_current_epoch(), get_last_good_epoch(), get_ahm_epoch(), get_ahm_time(), sysdate;

  • verbanverban Vertica Customer

    Thanks Nimmi and sorry for the late reply,
    First query didn't returned any result and second one was:

    Name               |Value                                             |
    -------------------|--------------------------------------------------|
    get_current_epoch  |25978958                                          |
    get_last_good_epoch|25924401                                          |
    get_ahm_epoch      |25858228                                          |
    get_ahm_time       |Current AHM Time: 2019-11-10 07:35:03.523396|
    sysdate            |2019-11-19 08:26:27
    

    In recent days i exported all the tables and schema manually (in unsafe mode) and rebuild database (because i needed to fix it asap).
    I faced this issue for 3rd time in last month, I think VERTICA should workaround it.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file