Could n't able to restart vertica database

sprusty

Facing some issues during restarting vertica database by the following command:
admintools -t restart_node -F -s -d

The error was saying "Node was not down. Can't restart"

Jim_Knicely

Are you trying to restart the entire DB or just a node in the DB?

sprusty

I am trying to restart the entire vertica database

Jim_Knicely

Then try using the start_db admintool.

admintools -t start_db -d <<dbname>>

sprusty

It was giving me the below issue "Error: Shutdown: aborting shutdown"

Jim_Knicely

What do you get from this command?

admintools -t list_allnodes

sprusty

Node | Host | State | Version | DB
-----------------+---------------+-------+-------------------+------
v_pmdb_node0001 | 172.27.21.249 | UP | vertica-7.1.2.000 | pmdb

Jim_Knicely

Do you have a single node cluster? If so, looks like it is up. Can you connect to the pmdb database using vsql?

sprusty

Yes We have a single node cluster. We can also connect using vsql

Jim_Knicely

I'm confused. So you want to restart the database that is up? Why?

Anyway, if that's what you want to do, do this:

admintools -t stop_db -d pmdb
admintools -t start_db -d pmdb

sprusty

I have already tried to stop and then start it. Still facing the above issue which saying Error: Shutdown.

We are trying to restart it as there was an issue with the report extraction in OBR.
The issue is as below:
"The following database error occured: [Vertica]VJDBC ERROR FileColumnReader block /vertica/datafiles/pmdb/v_pmdb_node0001_data/129/45035999921377129/45035999921377129_0.fdb @ 0's CRC 35ae46f1f9e0c65a doesn't match record 2203c0ee4b544bc2."

Jim_Knicely

Oh. Ok.

See:
https://forum.vertica.com/discussion/207463/strange-error-message-during-query

sprusty

We are not able to start the vertica databse now with the above command. Can you please help.

sprusty

Even after restoring the vertica database, still not able to start the database. Can you please help ?

Jim_Knicely

What does this command return?

admintools -t return_epoch -d pmdb

sprusty

LastEpoch|0

sprusty

Hi Jim, Can we discuss over a call if you have time ?

sprusty

Hi Jim, We have used the below command to start the vertica db.
admintools -t start_db -d pmdb -F

However It's not getting completely started ie in the list 2 status is INITIALIZING and rest of the node status is DOWN.

Can you please help.

sprusty

Jim_Knicely

@sprusty - Did you get the node started?

If not, try the attached script to manually start the node. I had to name the file with the .txt extension to upload it. Rename it to manual_start_up.sh when you DL it.

Copy the file to your node and make sure it is executable. Then run it like this:

./manual_start_up.sh pmdb

Jim_Knicely

I'd also try renaming the corrupt file to something else:

mv /vertica/datafiles/pmdb/v_pmdb_node0001_data/129/45035999921377129/45035999921377129_0.fdb /vertica/datafiles/pmdb/v_pmdb_node0001_data/129/45035999921377129/45035999921377129_0.fdb_old

Then try to start the DB:

admintools -t start_db -d pmdb -F

Normaly Vertica will recover the file, but since you only have one node that's not going to happen. So you might lose some data.

sprusty

Hi Jim, During running the script it was giving me an error saying "/bin/bash^M: bad interpreter: No such file or directory". So I ran a command sed -i -e 's/\r$//' scriptname.sh and then executed the same. During executing the script it said, database is up, however when checking it was showing DOWN.

INFO: Database pmdb is not running

Kindly help

Jim_Knicely

@sprusty - Did you try renaming that file?

sprusty

Thanks Jim for your continuous response on this.
Currently the DB is up after setting the projections to 0.

What should be our next course of action.
Regards
Suryakanta