Are you on a multi-node K-Safe cluster? If so Vertica can recover for you.
Example:
I'll delete a .gt file:
[dbadmin@vertica01 681]$ rm -f 0297d0a17c34fe9261c352d8f2c551a400a0000000002dc9_0.gt
[dbadmin@vertica01 681]$ vsql
Password:
Welcome to vsql, the Vertica Analytic Database interactive terminal.
Type: \h or \? for help with vsql commands
\g or terminate with semicolon to execute query
\q to quit
dbadmin=> select * from test;
ERROR 6407: StorageBundleReader: Error opening file [/home/dbadmin/rename_me/v_rename_me_node0001_data/681/0297d0a17c34fe9261c352d8f2c551a400a0000000002dc9_0.gt] for read: No such file or directory
dbadmin=>\q
[dbadmin@vertica01 681]$ admintools -t start_db -d rename_me -p vertica8
Starting nodes:
v_rename_me_node0001 (192.168.2.200)
v_rename_me_node0002 (192.168.2.201)
v_rename_me_node0003 (192.168.2.202)
Starting Vertica on all nodes. Please wait, databases with a large catalog may take a while to initialize.
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (INITIALIZING) v_rename_me_node0003: (INITIALIZING)
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (UP) v_rename_me_node0003: (UP)
Nodes UP: v_rename_me_node0003, v_rename_me_node0002
Nodes DOWN: v_rename_me_node0001 (may be still initializing).
Found these errors in startup.logs on hosts:
host ['192.168.2.200'] report: @v_rename_me_node0001: VX001/2973: Data consistency problems found; startup aborted
HINT: Check that all file systems are properly mounted. Also, the --force option can be used to delete corrupted data and recover from the cluster
LOCATION: mainEntryPoint, /scratch_a/release/svrtar28000/vbuild/vertica/Basics/vertica.cpp:1696
Do you want to continue waiting? (yes/no) [yes] no
Server startup was successful on some nodes, but not complete
Press RETURN to continue
Database rename_me did not start successfully: No better than partial startup
Not a full start :'( Note the hint about using the --force option to recover.
[dbadmin@vertica01 681]$ admintools -t restart_node -s 192.168.2.200 -d rename_me --force
Info: no password specified, using none
*** Restarting nodes for database rename_me ***
restart host 192.168.2.200 with catalog v_rename_me_node0001_catalog
issuing multi-node restart
Starting nodes:
v_rename_me_node0001 (192.168.2.200)
Starting Vertica on all nodes. Please wait, databases with a large catalog may take a while to initialize.
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0003: (UP)
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0003: (UP)
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0003: (UP)
Node Status: v_rename_me_node0001: (UP) v_rename_me_node0003: (UP)
Restart Nodes result: 1
[dbadmin@vertica01 681]$ admintools -t list_allnodes | grep rename_me
v_rename_me_node0001 | 192.168.2.200 | UP | vertica-9.0.1.7 | rename_me
v_rename_me_node0002 | 192.168.2.201 | UP | vertica-9.0.1.7 | rename_me
v_rename_me_node0003 | 192.168.2.202 | UP | vertica-9.0.1.7 | rename_me
Is my data ok? Yup!
[dbadmin@vertica01 681]$ vsql
Password:
Welcome to vsql, the Vertica Analytic Database interactive terminal.
Type: \h or \? for help with vsql commands
\g or terminate with semicolon to execute query
\q to quit
dbadmin=> select * from test;
c1
----
5
1
2
3
4
(5 rows)
Comments
Hi,
Are you on a multi-node K-Safe cluster? If so Vertica can recover for you.
Example:
I'll delete a .gt file:
[dbadmin@vertica01 681]$ rm -f 0297d0a17c34fe9261c352d8f2c551a400a0000000002dc9_0.gt [dbadmin@vertica01 681]$ vsql Password: Welcome to vsql, the Vertica Analytic Database interactive terminal. Type: \h or \? for help with vsql commands \g or terminate with semicolon to execute query \q to quit dbadmin=> select * from test; ERROR 6407: StorageBundleReader: Error opening file [/home/dbadmin/rename_me/v_rename_me_node0001_data/681/0297d0a17c34fe9261c352d8f2c551a400a0000000002dc9_0.gt] for read: No such file or directory dbadmin=>\qThat's not good!
Now I'll stop the DB to get to where you are ...
[dbadmin@vertica01 681]$ admintools -t stop_db -d rename_me -p ****** -F Connecting to database Issuing shutdown command to database Database rename_me stopped successfullyNext I will try to restart:
[dbadmin@vertica01 681]$ admintools -t start_db -d rename_me -p vertica8 Starting nodes: v_rename_me_node0001 (192.168.2.200) v_rename_me_node0002 (192.168.2.201) v_rename_me_node0003 (192.168.2.202) Starting Vertica on all nodes. Please wait, databases with a large catalog may take a while to initialize. Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN) Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN) Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN) Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN) Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN) Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN) Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN) Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (INITIALIZING) v_rename_me_node0003: (INITIALIZING) Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (UP) v_rename_me_node0003: (UP) Nodes UP: v_rename_me_node0003, v_rename_me_node0002 Nodes DOWN: v_rename_me_node0001 (may be still initializing). Found these errors in startup.logs on hosts: host ['192.168.2.200'] report: @v_rename_me_node0001: VX001/2973: Data consistency problems found; startup aborted HINT: Check that all file systems are properly mounted. Also, the --force option can be used to delete corrupted data and recover from the cluster LOCATION: mainEntryPoint, /scratch_a/release/svrtar28000/vbuild/vertica/Basics/vertica.cpp:1696 Do you want to continue waiting? (yes/no) [yes] no Server startup was successful on some nodes, but not complete Press RETURN to continue Database rename_me did not start successfully: No better than partial startupNot a full start :'( Note the hint about using the --force option to recover.
Which node is down?
Restart the node using the --force option:
[dbadmin@vertica01 681]$ admintools -t restart_node -s 192.168.2.200 -d rename_me --force Info: no password specified, using none *** Restarting nodes for database rename_me *** restart host 192.168.2.200 with catalog v_rename_me_node0001_catalog issuing multi-node restart Starting nodes: v_rename_me_node0001 (192.168.2.200) Starting Vertica on all nodes. Please wait, databases with a large catalog may take a while to initialize. Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0003: (UP) Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0003: (UP) Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0003: (UP) Node Status: v_rename_me_node0001: (UP) v_rename_me_node0003: (UP) Restart Nodes result: 1 [dbadmin@vertica01 681]$ admintools -t list_allnodes | grep rename_me v_rename_me_node0001 | 192.168.2.200 | UP | vertica-9.0.1.7 | rename_me v_rename_me_node0002 | 192.168.2.201 | UP | vertica-9.0.1.7 | rename_me v_rename_me_node0003 | 192.168.2.202 | UP | vertica-9.0.1.7 | rename_meIs my data ok? Yup!
[dbadmin@vertica01 681]$ vsql Password: Welcome to vsql, the Vertica Analytic Database interactive terminal. Type: \h or \? for help with vsql commands \g or terminate with semicolon to execute query \q to quit dbadmin=> select * from test; c1 ---- 5 1 2 3 4 (5 rows)Thank u @Jim_Knicely