Vertica node is down and not able to bring it up
Hello , i have a vertica setup cluster with 7 nodes and all of them went down. i followed this link that i found online https://www.vertica.com/blog/what-should-i-do-when-the-database-node-is-down/ and did all the steps as mentioned and no luck. Could someone reach out if they have gone through the same situation. This is one of of development setup and currently proceeding with a fresh install so that i can initialize the cluster again. I am worried if the same happens on a production environment.
Tagged:
0
Answers
What is the error message in logs?
Which logs would you like to see ?
please share me vertica.log and startup.log
I was checking the spread and it showed me this.
I was checking the spread and it showed me this
It might be possible that spread is down. please share me vertica.log and startup.log.
{
"goal" : 2599,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Reading Catalog",
"text" : "Reading Checkpoint (bytes)",
"timestamp" : "2019-09-10 01:12:15.460"
}
{
"goal" : 2599,
"node" : "v_insights_data_node0001",
"progress" : 273,
"stage" : "Reading Catalog",
"text" : "Reading Checkpoint (bytes)",
"timestamp" : "2019-09-10 01:12:15.461"
}
{
"goal" : 2599,
"node" : "v_insights_data_node0001",
"progress" : 2872,
"stage" : "Reading Catalog",
"text" : "Reading Checkpoint (bytes)",
"timestamp" : "2019-09-10 01:12:15.461"
}
{
"goal" : 4341580,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Reading Catalog",
"text" : "Applying transaction log (bytes)",
"timestamp" : "2019-09-10 01:12:15.461"
}
{
"goal" : 4341580,
"node" : "v_insights_data_node0001",
"progress" : 4341580,
"stage" : "Reading Catalog",
"text" : "Applying transaction log (bytes)",
"timestamp" : "2019-09-10 01:12:15.642"
}
{
"goal" : 3003,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Reading Catalog",
"text" : "Indexing Objects",
"timestamp" : "2019-09-10 01:12:15.642"
}
{
"goal" : 3003,
"node" : "v_insights_data_node0001",
"progress" : 2743,
"stage" : "Reading Catalog",
"text" : "Indexing Objects",
"timestamp" : "2019-09-10 01:12:15.647"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Connecting to Spread",
"text" : "Connecting to spread /opt/vertica/spread/tmp/4803",
"timestamp" : "2019-09-10 01:12:15.660"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.661"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 5565650,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.677"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 13333375,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.699"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 19271376,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.718"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 29272721,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.740"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 35040684,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.757"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 48870052,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.797"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 54867564,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.815"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Check Storage",
"text" : "Removing unnecessary storage files",
"timestamp" : "2019-09-10 01:12:15.857"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.349"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 22,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.350"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 44,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.350"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 66,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.351"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 88,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.351"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 110,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.351"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 132,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.352"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 154,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.352"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 172,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.352"
}
{
"goal" : 6,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.352"
}
{
"goal" : 6,
"node" : "v_insights_data_node0001",
"progress" : 6,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.352"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Waiting for Cluster Invite",
"text" : "Prepare to be invited",
"timestamp" : "2019-09-10 01:12:22.000"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Waiting for Cluster Invite",
"text" : "Prepare to be invited",
"timestamp" : "2019-09-10 01:12:24.000"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Waiting for Cluster Invite",
"text" : "Ready to be invited",
"timestamp" : "2019-09-10 01:12:24.661"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Waiting for Cluster Invite",
"text" : "Invited",
"timestamp" : "2019-09-10 01:12:31.993"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Startup Complete",
"text" : "Node is UP",
"timestamp" : "2019-09-10 01:12:32.415"
}
Startup.log shows node is UP right? What is the output of
admintools -t list_allnodes
population stopped on 11 th October and today is 16 th October .
Please share vertica.log from Oct 11 to review root cause
{
"goal" : 2599,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Reading Catalog",
"text" : "Reading Checkpoint (bytes)",
"timestamp" : "2019-09-10 01:12:15.460"
}
{
"goal" : 2599,
"node" : "v_insights_data_node0001",
"progress" : 273,
"stage" : "Reading Catalog",
"text" : "Reading Checkpoint (bytes)",
"timestamp" : "2019-09-10 01:12:15.461"
}
{
"goal" : 2599,
"node" : "v_insights_data_node0001",
"progress" : 2872,
"stage" : "Reading Catalog",
"text" : "Reading Checkpoint (bytes)",
"timestamp" : "2019-09-10 01:12:15.461"
}
{
"goal" : 4341580,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Reading Catalog",
"text" : "Applying transaction log (bytes)",
"timestamp" : "2019-09-10 01:12:15.461"
}
{
"goal" : 4341580,
"node" : "v_insights_data_node0001",
"progress" : 4341580,
"stage" : "Reading Catalog",
"text" : "Applying transaction log (bytes)",
"timestamp" : "2019-09-10 01:12:15.642"
}
{
"goal" : 3003,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Reading Catalog",
"text" : "Indexing Objects",
"timestamp" : "2019-09-10 01:12:15.642"
}
{
"goal" : 3003,
"node" : "v_insights_data_node0001",
"progress" : 2743,
"stage" : "Reading Catalog",
"text" : "Indexing Objects",
"timestamp" : "2019-09-10 01:12:15.647"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Connecting to Spread",
"text" : "Connecting to spread /opt/vertica/spread/tmp/4803",
"timestamp" : "2019-09-10 01:12:15.660"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.661"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 5565650,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.677"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 13333375,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.699"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 19271376,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.718"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 29272721,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.740"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 35040684,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.757"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 48870052,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.797"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 54867564,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.815"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Check Storage",
"text" : "Removing unnecessary storage files",
"timestamp" : "2019-09-10 01:12:15.857"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.349"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 22,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.350"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 44,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.350"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 66,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.351"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 88,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.351"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 110,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.351"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 132,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.352"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 154,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.352"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 172,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.352"
}
{
"goal" : 6,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.352"
}
{
"goal" : 6,
"node" : "v_insights_data_node0001",
"progress" : 6,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.352"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Waiting for Cluster Invite",
"text" : "Prepare to be invited",
"timestamp" : "2019-09-10 01:12:22.000"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Waiting for Cluster Invite",
"text" : "Prepare to be invited",
"timestamp" : "2019-09-10 01:12:24.000"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Waiting for Cluster Invite",
"text" : "Ready to be invited",
"timestamp" : "2019-09-10 01:12:24.661"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Waiting for Cluster Invite",
"text" : "Invited",
"timestamp" : "2019-09-10 01:12:31.993"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Startup Complete",
"text" : "Node is UP",
"timestamp" : "2019-09-10 01:12:32.415"
}
dbadmin@pl2:/$ admintools -t list_allnodes
Node | Host | State | Version | DB
--------------------------+---------+-------+------------------+---------------
v_insights_data_node0001 | 3.0.0.1 | DOWN | unavailable | insights_data
v_insights_data_node0002 | 3.0.0.2 | DOWN | unavailable | insights_data
v_insights_data_node0003 | 3.0.0.3 | DOWN | vertica-8.1.1.10 | insights_data
v_insights_data_node0004 | 3.0.0.4 | DOWN | vertica-8.1.1.10 | insights_data
v_insights_data_node0005 | 3.0.0.5 | DOWN | vertica-8.1.1.10 | insights_data
v_insights_data_node0006 | 3.0.0.6 | DOWN | vertica-8.1.1.10 | insights_data
v_insights_data_node0007 | 3.0.0.7 | DOWN | vertica-8.1.1.10 | insights_data
Based on the below output, I recommend you to start node3 till node7. Once those are UP, you can try bringing node1 and node2 UP
dbadmin@pl2:/$ admintools -t list_allnodes
Node | Host | State | Version | DB
--------------------------+---------+-------+------------------+---------------
v_insights_data_node0001 | 3.0.0.1 | DOWN | unavailable | insights_data
v_insights_data_node0002 | 3.0.0.2 | DOWN | unavailable | insights_data
v_insights_data_node0003 | 3.0.0.3 | DOWN | vertica-8.1.1.10 | insights_data
v_insights_data_node0004 | 3.0.0.4 | DOWN | vertica-8.1.1.10 | insights_data
v_insights_data_node0005 | 3.0.0.5 | DOWN | vertica-8.1.1.10 | insights_data
v_insights_data_node0006 | 3.0.0.6 | DOWN | vertica-8.1.1.10 | insights_data
v_insights_data_node0007 | 3.0.0.7 | DOWN | vertica-8.1.1.10 | insights_data