how to start database when .gt file deleted

edited April 2018 in General Discussion

hello vertica team!!
By mistakenly deleted .gt file in the data folder, what I have to do?

Comments

  • Jim_KnicelyJim_Knicely - Select Field - Administrator
    edited April 2018

    Hi,

    Are you on a multi-node K-Safe cluster? If so Vertica can recover for you.

    Example:

    I'll delete a .gt file:

    [dbadmin@vertica01 681]$ rm -f 0297d0a17c34fe9261c352d8f2c551a400a0000000002dc9_0.gt
    
    [dbadmin@vertica01 681]$ vsql
    Password:
    Welcome to vsql, the Vertica Analytic Database interactive terminal.
    
    Type:  \h or \? for help with vsql commands
           \g or terminate with semicolon to execute query
           \q to quit
    
    dbadmin=> select * from test;
    ERROR 6407:  StorageBundleReader: Error opening file [/home/dbadmin/rename_me/v_rename_me_node0001_data/681/0297d0a17c34fe9261c352d8f2c551a400a0000000002dc9_0.gt] for read: No such file or directory
    
    dbadmin=>\q
    

    That's not good!

    Now I'll stop the DB to get to where you are ...

    [dbadmin@vertica01 681]$ admintools -t stop_db -d rename_me -p ****** -F
            Connecting to database
            Issuing shutdown command to database
    Database rename_me stopped successfully
    

    Next I will try to restart:

    [dbadmin@vertica01 681]$ admintools -t start_db -d rename_me -p vertica8
            Starting nodes:
                    v_rename_me_node0001 (192.168.2.200)
                    v_rename_me_node0002 (192.168.2.201)
                    v_rename_me_node0003 (192.168.2.202)
            Starting Vertica on all nodes. Please wait, databases with a large catalog may take a while to initialize.
            Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
            Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
            Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
            Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
            Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
            Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
            Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
            Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (INITIALIZING) v_rename_me_node0003: (INITIALIZING)
            Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (UP) v_rename_me_node0003: (UP)
    Nodes UP: v_rename_me_node0003, v_rename_me_node0002
    Nodes DOWN: v_rename_me_node0001 (may be still initializing).
            Found these errors in startup.logs on hosts:
    host ['192.168.2.200'] report: @v_rename_me_node0001: VX001/2973: Data consistency problems found; startup aborted
            HINT:  Check that all file systems are properly mounted.  Also, the --force option can be used to delete corrupted data and recover from the cluster
            LOCATION:  mainEntryPoint, /scratch_a/release/svrtar28000/vbuild/vertica/Basics/vertica.cpp:1696
    Do you want to continue waiting? (yes/no) [yes] no
            Server startup was successful on some nodes, but not complete
    
    Press RETURN to continue
    Database rename_me did not start successfully: No better than partial startup
    

    Not a full start :'( Note the hint about using the --force option to recover.

    Which node is down?

    [dbadmin@vertica01 681]$ admintools -t list_allnodes | grep rename_me
     v_rename_me_node0001 | 192.168.2.200 | DOWN  | vertica-9.0.1.7 | rename_me
     v_rename_me_node0002 | 192.168.2.201 | UP    | vertica-9.0.1.7 | rename_me
     v_rename_me_node0003 | 192.168.2.202 | UP    | vertica-9.0.1.7 | rename_me
    

    Restart the node using the --force option:

    [dbadmin@vertica01 681]$ admintools -t restart_node -s 192.168.2.200 -d rename_me --force
    Info: no password specified, using none
    *** Restarting nodes for database rename_me ***
            restart host 192.168.2.200 with catalog v_rename_me_node0001_catalog
            issuing multi-node restart
            Starting nodes:
                    v_rename_me_node0001 (192.168.2.200)
            Starting Vertica on all nodes. Please wait, databases with a large catalog may take a while to initialize.
            Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0003: (UP)
            Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0003: (UP)
            Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0003: (UP)
            Node Status: v_rename_me_node0001: (UP) v_rename_me_node0003: (UP)
    Restart Nodes result:  1
    
    [dbadmin@vertica01 681]$ admintools -t list_allnodes | grep rename_me
     v_rename_me_node0001 | 192.168.2.200 | UP    | vertica-9.0.1.7 | rename_me
     v_rename_me_node0002 | 192.168.2.201 | UP    | vertica-9.0.1.7 | rename_me
     v_rename_me_node0003 | 192.168.2.202 | UP    | vertica-9.0.1.7 | rename_me
    

    Is my data ok? Yup!

    [dbadmin@vertica01 681]$ vsql
    Password:
    Welcome to vsql, the Vertica Analytic Database interactive terminal.
    
    Type:  \h or \? for help with vsql commands
           \g or terminate with semicolon to execute query
           \q to quit
    
    dbadmin=> select * from test;
     c1
    ----
      5
      1
      2
      3
      4
    (5 rows)
    

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file