Are you on a multi-node K-Safe cluster? If so Vertica can recover for you.
Example:
I'll delete a .gt file:
[dbadmin@vertica01 681]$ rm -f 0297d0a17c34fe9261c352d8f2c551a400a0000000002dc9_0.gt
[dbadmin@vertica01 681]$ vsql
Password:
Welcome to vsql, the Vertica Analytic Database interactive terminal.
Type: \h or \? for help with vsql commands
\g or terminate with semicolon to execute query
\q to quit
dbadmin=> select * from test;
ERROR 6407: StorageBundleReader: Error opening file [/home/dbadmin/rename_me/v_rename_me_node0001_data/681/0297d0a17c34fe9261c352d8f2c551a400a0000000002dc9_0.gt] for read: No such file or directory
dbadmin=>\q
[dbadmin@vertica01 681]$ admintools -t start_db -d rename_me -p vertica8
Starting nodes:
v_rename_me_node0001 (192.168.2.200)
v_rename_me_node0002 (192.168.2.201)
v_rename_me_node0003 (192.168.2.202)
Starting Vertica on all nodes. Please wait, databases with a large catalog may take a while to initialize.
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (INITIALIZING) v_rename_me_node0003: (INITIALIZING)
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (UP) v_rename_me_node0003: (UP)
Nodes UP: v_rename_me_node0003, v_rename_me_node0002
Nodes DOWN: v_rename_me_node0001 (may be still initializing).
Found these errors in startup.logs on hosts:
host ['192.168.2.200'] report: @v_rename_me_node0001: VX001/2973: Data consistency problems found; startup aborted
HINT: Check that all file systems are properly mounted. Also, the --force option can be used to delete corrupted data and recover from the cluster
LOCATION: mainEntryPoint, /scratch_a/release/svrtar28000/vbuild/vertica/Basics/vertica.cpp:1696
Do you want to continue waiting? (yes/no) [yes] no
Server startup was successful on some nodes, but not complete
Press RETURN to continue
Database rename_me did not start successfully: No better than partial startup
Not a full start Note the hint about using the --force option to recover.
[dbadmin@vertica01 681]$ admintools -t restart_node -s 192.168.2.200 -d rename_me --force
Info: no password specified, using none
*** Restarting nodes for database rename_me ***
restart host 192.168.2.200 with catalog v_rename_me_node0001_catalog
issuing multi-node restart
Starting nodes:
v_rename_me_node0001 (192.168.2.200)
Starting Vertica on all nodes. Please wait, databases with a large catalog may take a while to initialize.
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0003: (UP)
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0003: (UP)
Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0003: (UP)
Node Status: v_rename_me_node0001: (UP) v_rename_me_node0003: (UP)
Restart Nodes result: 1
[dbadmin@vertica01 681]$ admintools -t list_allnodes | grep rename_me
v_rename_me_node0001 | 192.168.2.200 | UP | vertica-9.0.1.7 | rename_me
v_rename_me_node0002 | 192.168.2.201 | UP | vertica-9.0.1.7 | rename_me
v_rename_me_node0003 | 192.168.2.202 | UP | vertica-9.0.1.7 | rename_me
Is my data ok? Yup!
[dbadmin@vertica01 681]$ vsql
Password:
Welcome to vsql, the Vertica Analytic Database interactive terminal.
Type: \h or \? for help with vsql commands
\g or terminate with semicolon to execute query
\q to quit
dbadmin=> select * from test;
c1
----
5
1
2
3
4
(5 rows)
Comments
Hi,
Are you on a multi-node K-Safe cluster? If so Vertica can recover for you.
Example:
I'll delete a .gt file:
That's not good!
Now I'll stop the DB to get to where you are ...
Next I will try to restart:
Not a full start Note the hint about using the --force option to recover.
Which node is down?
Restart the node using the --force option:
Is my data ok? Yup!
Thank u @Jim_Knicely