The Vertica Forum recently got a makeover! Let us know what you think by filling out this short, anonymous survey.

Vertica node is down and not able to bring it up

Hello , i have a vertica setup cluster with 7 nodes and all of them went down. i followed this link that i found online https://www.vertica.com/blog/what-should-i-do-when-the-database-node-is-down/ and did all the steps as mentioned and no luck. Could someone reach out if they have gone through the same situation. This is one of of development setup and currently proceeding with a fresh install so that i can initialize the cluster again. I am worried if the same happens on a production environment.

Tagged:

Answers

  • SruthiASruthiA Employee

    What is the error message in logs?

  • Which logs would you like to see ?

  • SruthiASruthiA Employee

    please share me vertica.log and startup.log

  • I was checking the spread and it showed me this.

  • I was checking the spread and it showed me this

  • SruthiASruthiA Employee

    It might be possible that spread is down. please share me vertica.log and startup.log.

  • {
    "goal" : 2599,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Reading Catalog",
    "text" : "Reading Checkpoint (bytes)",
    "timestamp" : "2019-09-10 01:12:15.460"
    }
    {
    "goal" : 2599,
    "node" : "v_insights_data_node0001",
    "progress" : 273,
    "stage" : "Reading Catalog",
    "text" : "Reading Checkpoint (bytes)",
    "timestamp" : "2019-09-10 01:12:15.461"
    }
    {
    "goal" : 2599,
    "node" : "v_insights_data_node0001",
    "progress" : 2872,
    "stage" : "Reading Catalog",
    "text" : "Reading Checkpoint (bytes)",
    "timestamp" : "2019-09-10 01:12:15.461"
    }
    {
    "goal" : 4341580,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Reading Catalog",
    "text" : "Applying transaction log (bytes)",
    "timestamp" : "2019-09-10 01:12:15.461"
    }
    {
    "goal" : 4341580,
    "node" : "v_insights_data_node0001",
    "progress" : 4341580,
    "stage" : "Reading Catalog",
    "text" : "Applying transaction log (bytes)",
    "timestamp" : "2019-09-10 01:12:15.642"
    }
    {
    "goal" : 3003,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Reading Catalog",
    "text" : "Indexing Objects",
    "timestamp" : "2019-09-10 01:12:15.642"
    }
    {
    "goal" : 3003,
    "node" : "v_insights_data_node0001",
    "progress" : 2743,
    "stage" : "Reading Catalog",
    "text" : "Indexing Objects",
    "timestamp" : "2019-09-10 01:12:15.647"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Connecting to Spread",
    "text" : "Connecting to spread /opt/vertica/spread/tmp/4803",
    "timestamp" : "2019-09-10 01:12:15.660"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.661"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 5565650,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.677"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 13333375,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.699"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 19271376,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.718"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 29272721,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.740"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 35040684,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.757"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 48870052,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.797"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 54867564,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.815"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Check Storage",
    "text" : "Removing unnecessary storage files",
    "timestamp" : "2019-09-10 01:12:15.857"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.349"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 22,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.350"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 44,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.350"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 66,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.351"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 88,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.351"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 110,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.351"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 132,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.352"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 154,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.352"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 172,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.352"
    }
    {
    "goal" : 6,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.352"
    }
    {
    "goal" : 6,
    "node" : "v_insights_data_node0001",
    "progress" : 6,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.352"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Waiting for Cluster Invite",
    "text" : "Prepare to be invited",
    "timestamp" : "2019-09-10 01:12:22.000"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Waiting for Cluster Invite",
    "text" : "Prepare to be invited",
    "timestamp" : "2019-09-10 01:12:24.000"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Waiting for Cluster Invite",
    "text" : "Ready to be invited",
    "timestamp" : "2019-09-10 01:12:24.661"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Waiting for Cluster Invite",
    "text" : "Invited",
    "timestamp" : "2019-09-10 01:12:31.993"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Startup Complete",
    "text" : "Node is UP",
    "timestamp" : "2019-09-10 01:12:32.415"
    }

  • SruthiASruthiA Employee
    edited October 16

    Startup.log shows node is UP right? What is the output of

    admintools -t list_allnodes

  • via Email
    Yes , but could you check the timestamp on the log file ? The data
    population stopped on 11 th October and today is 16 th October .
  • SruthiASruthiA Employee

    Please share vertica.log from Oct 11 to review root cause

  • {
    "goal" : 2599,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Reading Catalog",
    "text" : "Reading Checkpoint (bytes)",
    "timestamp" : "2019-09-10 01:12:15.460"
    }
    {
    "goal" : 2599,
    "node" : "v_insights_data_node0001",
    "progress" : 273,
    "stage" : "Reading Catalog",
    "text" : "Reading Checkpoint (bytes)",
    "timestamp" : "2019-09-10 01:12:15.461"
    }
    {
    "goal" : 2599,
    "node" : "v_insights_data_node0001",
    "progress" : 2872,
    "stage" : "Reading Catalog",
    "text" : "Reading Checkpoint (bytes)",
    "timestamp" : "2019-09-10 01:12:15.461"
    }
    {
    "goal" : 4341580,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Reading Catalog",
    "text" : "Applying transaction log (bytes)",
    "timestamp" : "2019-09-10 01:12:15.461"
    }
    {
    "goal" : 4341580,
    "node" : "v_insights_data_node0001",
    "progress" : 4341580,
    "stage" : "Reading Catalog",
    "text" : "Applying transaction log (bytes)",
    "timestamp" : "2019-09-10 01:12:15.642"
    }
    {
    "goal" : 3003,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Reading Catalog",
    "text" : "Indexing Objects",
    "timestamp" : "2019-09-10 01:12:15.642"
    }
    {
    "goal" : 3003,
    "node" : "v_insights_data_node0001",
    "progress" : 2743,
    "stage" : "Reading Catalog",
    "text" : "Indexing Objects",
    "timestamp" : "2019-09-10 01:12:15.647"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Connecting to Spread",
    "text" : "Connecting to spread /opt/vertica/spread/tmp/4803",
    "timestamp" : "2019-09-10 01:12:15.660"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.661"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 5565650,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.677"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 13333375,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.699"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 19271376,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.718"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 29272721,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.740"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 35040684,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.757"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 48870052,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.797"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 54867564,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.815"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Check Storage",
    "text" : "Removing unnecessary storage files",
    "timestamp" : "2019-09-10 01:12:15.857"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.349"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 22,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.350"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 44,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.350"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 66,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.351"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 88,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.351"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 110,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.351"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 132,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.352"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 154,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.352"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 172,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.352"
    }
    {
    "goal" : 6,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.352"
    }
    {
    "goal" : 6,
    "node" : "v_insights_data_node0001",
    "progress" : 6,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.352"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Waiting for Cluster Invite",
    "text" : "Prepare to be invited",
    "timestamp" : "2019-09-10 01:12:22.000"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Waiting for Cluster Invite",
    "text" : "Prepare to be invited",
    "timestamp" : "2019-09-10 01:12:24.000"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Waiting for Cluster Invite",
    "text" : "Ready to be invited",
    "timestamp" : "2019-09-10 01:12:24.661"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Waiting for Cluster Invite",
    "text" : "Invited",
    "timestamp" : "2019-09-10 01:12:31.993"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Startup Complete",
    "text" : "Node is UP",
    "timestamp" : "2019-09-10 01:12:32.415"
    }

    @SruthiA said:
    Startup.log shows node is UP right? What is the output of

    admintools -t list_allnodes

    [email protected]:/$ admintools -t list_allnodes
    Node | Host | State | Version | DB
    --------------------------+---------+-------+------------------+---------------
    v_insights_data_node0001 | 3.0.0.1 | DOWN | unavailable | insights_data
    v_insights_data_node0002 | 3.0.0.2 | DOWN | unavailable | insights_data
    v_insights_data_node0003 | 3.0.0.3 | DOWN | vertica-8.1.1.10 | insights_data
    v_insights_data_node0004 | 3.0.0.4 | DOWN | vertica-8.1.1.10 | insights_data
    v_insights_data_node0005 | 3.0.0.5 | DOWN | vertica-8.1.1.10 | insights_data
    v_insights_data_node0006 | 3.0.0.6 | DOWN | vertica-8.1.1.10 | insights_data
    v_insights_data_node0007 | 3.0.0.7 | DOWN | vertica-8.1.1.10 | insights_data

  • SruthiASruthiA Employee

    Based on the below output, I recommend you to start node3 till node7. Once those are UP, you can try bringing node1 and node2 UP

    [email protected]:/$ admintools -t list_allnodes
    Node | Host | State | Version | DB
    --------------------------+---------+-------+------------------+---------------
    v_insights_data_node0001 | 3.0.0.1 | DOWN | unavailable | insights_data
    v_insights_data_node0002 | 3.0.0.2 | DOWN | unavailable | insights_data
    v_insights_data_node0003 | 3.0.0.3 | DOWN | vertica-8.1.1.10 | insights_data
    v_insights_data_node0004 | 3.0.0.4 | DOWN | vertica-8.1.1.10 | insights_data
    v_insights_data_node0005 | 3.0.0.5 | DOWN | vertica-8.1.1.10 | insights_data
    v_insights_data_node0006 | 3.0.0.6 | DOWN | vertica-8.1.1.10 | insights_data
    v_insights_data_node0007 | 3.0.0.7 | DOWN | vertica-8.1.1.10 | insights_data

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file

Can't find what you're looking for? Search the Vertica Documentation, Knowledge Base, or Blog for more information.