Vertica node is down and not able to bring it up

Hello , i have a vertica setup cluster with 7 nodes and all of them went down. i followed this link that i found online https://www.vertica.com/blog/what-should-i-do-when-the-database-node-is-down/ and did all the steps as mentioned and no luck. Could someone reach out if they have gone through the same situation. This is one of of development setup and currently proceeding with a fresh install so that i can initialize the cluster again. I am worried if the same happens on a production environment.

Tagged:

Answers

  • SruthiASruthiA Administrator

    What is the error message in logs?

  • Which logs would you like to see ?

  • SruthiASruthiA Administrator

    please share me vertica.log and startup.log

  • I was checking the spread and it showed me this.

  • I was checking the spread and it showed me this

  • SruthiASruthiA Administrator

    It might be possible that spread is down. please share me vertica.log and startup.log.

  • {
    "goal" : 2599,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Reading Catalog",
    "text" : "Reading Checkpoint (bytes)",
    "timestamp" : "2019-09-10 01:12:15.460"
    }
    {
    "goal" : 2599,
    "node" : "v_insights_data_node0001",
    "progress" : 273,
    "stage" : "Reading Catalog",
    "text" : "Reading Checkpoint (bytes)",
    "timestamp" : "2019-09-10 01:12:15.461"
    }
    {
    "goal" : 2599,
    "node" : "v_insights_data_node0001",
    "progress" : 2872,
    "stage" : "Reading Catalog",
    "text" : "Reading Checkpoint (bytes)",
    "timestamp" : "2019-09-10 01:12:15.461"
    }
    {
    "goal" : 4341580,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Reading Catalog",
    "text" : "Applying transaction log (bytes)",
    "timestamp" : "2019-09-10 01:12:15.461"
    }
    {
    "goal" : 4341580,
    "node" : "v_insights_data_node0001",
    "progress" : 4341580,
    "stage" : "Reading Catalog",
    "text" : "Applying transaction log (bytes)",
    "timestamp" : "2019-09-10 01:12:15.642"
    }
    {
    "goal" : 3003,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Reading Catalog",
    "text" : "Indexing Objects",
    "timestamp" : "2019-09-10 01:12:15.642"
    }
    {
    "goal" : 3003,
    "node" : "v_insights_data_node0001",
    "progress" : 2743,
    "stage" : "Reading Catalog",
    "text" : "Indexing Objects",
    "timestamp" : "2019-09-10 01:12:15.647"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Connecting to Spread",
    "text" : "Connecting to spread /opt/vertica/spread/tmp/4803",
    "timestamp" : "2019-09-10 01:12:15.660"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.661"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 5565650,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.677"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 13333375,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.699"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 19271376,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.718"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 29272721,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.740"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 35040684,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.757"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 48870052,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.797"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 54867564,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.815"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Check Storage",
    "text" : "Removing unnecessary storage files",
    "timestamp" : "2019-09-10 01:12:15.857"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.349"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 22,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.350"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 44,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.350"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 66,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.351"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 88,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.351"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 110,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.351"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 132,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.352"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 154,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.352"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 172,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.352"
    }
    {
    "goal" : 6,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.352"
    }
    {
    "goal" : 6,
    "node" : "v_insights_data_node0001",
    "progress" : 6,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.352"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Waiting for Cluster Invite",
    "text" : "Prepare to be invited",
    "timestamp" : "2019-09-10 01:12:22.000"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Waiting for Cluster Invite",
    "text" : "Prepare to be invited",
    "timestamp" : "2019-09-10 01:12:24.000"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Waiting for Cluster Invite",
    "text" : "Ready to be invited",
    "timestamp" : "2019-09-10 01:12:24.661"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Waiting for Cluster Invite",
    "text" : "Invited",
    "timestamp" : "2019-09-10 01:12:31.993"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Startup Complete",
    "text" : "Node is UP",
    "timestamp" : "2019-09-10 01:12:32.415"
    }

  • SruthiASruthiA Administrator
    edited October 2019

    Startup.log shows node is UP right? What is the output of

    admintools -t list_allnodes

  • Yes , but could you check the timestamp on the log file ? The data
    population stopped on 11 th October and today is 16 th October .
  • SruthiASruthiA Administrator

    Please share vertica.log from Oct 11 to review root cause

  • {
    "goal" : 2599,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Reading Catalog",
    "text" : "Reading Checkpoint (bytes)",
    "timestamp" : "2019-09-10 01:12:15.460"
    }
    {
    "goal" : 2599,
    "node" : "v_insights_data_node0001",
    "progress" : 273,
    "stage" : "Reading Catalog",
    "text" : "Reading Checkpoint (bytes)",
    "timestamp" : "2019-09-10 01:12:15.461"
    }
    {
    "goal" : 2599,
    "node" : "v_insights_data_node0001",
    "progress" : 2872,
    "stage" : "Reading Catalog",
    "text" : "Reading Checkpoint (bytes)",
    "timestamp" : "2019-09-10 01:12:15.461"
    }
    {
    "goal" : 4341580,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Reading Catalog",
    "text" : "Applying transaction log (bytes)",
    "timestamp" : "2019-09-10 01:12:15.461"
    }
    {
    "goal" : 4341580,
    "node" : "v_insights_data_node0001",
    "progress" : 4341580,
    "stage" : "Reading Catalog",
    "text" : "Applying transaction log (bytes)",
    "timestamp" : "2019-09-10 01:12:15.642"
    }
    {
    "goal" : 3003,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Reading Catalog",
    "text" : "Indexing Objects",
    "timestamp" : "2019-09-10 01:12:15.642"
    }
    {
    "goal" : 3003,
    "node" : "v_insights_data_node0001",
    "progress" : 2743,
    "stage" : "Reading Catalog",
    "text" : "Indexing Objects",
    "timestamp" : "2019-09-10 01:12:15.647"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Connecting to Spread",
    "text" : "Connecting to spread /opt/vertica/spread/tmp/4803",
    "timestamp" : "2019-09-10 01:12:15.660"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.661"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 5565650,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.677"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 13333375,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.699"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 19271376,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.718"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 29272721,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.740"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 35040684,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.757"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 48870052,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.797"
    }
    {
    "goal" : 55301669,
    "node" : "v_insights_data_node0001",
    "progress" : 54867564,
    "stage" : "Read DataCollector",
    "text" : "Inventory files (bytes)",
    "timestamp" : "2019-09-10 01:12:15.815"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Check Storage",
    "text" : "Removing unnecessary storage files",
    "timestamp" : "2019-09-10 01:12:15.857"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.349"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 22,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.350"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 44,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.350"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 66,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.351"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 88,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.351"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 110,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.351"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 132,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.352"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 154,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.352"
    }
    {
    "goal" : 172,
    "node" : "v_insights_data_node0001",
    "progress" : 172,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.352"
    }
    {
    "goal" : 6,
    "node" : "v_insights_data_node0001",
    "progress" : 0,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.352"
    }
    {
    "goal" : 6,
    "node" : "v_insights_data_node0001",
    "progress" : 6,
    "stage" : "Check Storage",
    "text" : "Confirming storage matches catalog (files)",
    "timestamp" : "2019-09-10 01:12:16.352"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Waiting for Cluster Invite",
    "text" : "Prepare to be invited",
    "timestamp" : "2019-09-10 01:12:22.000"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Waiting for Cluster Invite",
    "text" : "Prepare to be invited",
    "timestamp" : "2019-09-10 01:12:24.000"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Waiting for Cluster Invite",
    "text" : "Ready to be invited",
    "timestamp" : "2019-09-10 01:12:24.661"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Waiting for Cluster Invite",
    "text" : "Invited",
    "timestamp" : "2019-09-10 01:12:31.993"
    }
    {
    "node" : "v_insights_data_node0001",
    "stage" : "Startup Complete",
    "text" : "Node is UP",
    "timestamp" : "2019-09-10 01:12:32.415"
    }

    @SruthiA said:
    Startup.log shows node is UP right? What is the output of

    admintools -t list_allnodes

    dbadmin@pl2:/$ admintools -t list_allnodes
    Node | Host | State | Version | DB
    --------------------------+---------+-------+------------------+---------------
    v_insights_data_node0001 | 3.0.0.1 | DOWN | unavailable | insights_data
    v_insights_data_node0002 | 3.0.0.2 | DOWN | unavailable | insights_data
    v_insights_data_node0003 | 3.0.0.3 | DOWN | vertica-8.1.1.10 | insights_data
    v_insights_data_node0004 | 3.0.0.4 | DOWN | vertica-8.1.1.10 | insights_data
    v_insights_data_node0005 | 3.0.0.5 | DOWN | vertica-8.1.1.10 | insights_data
    v_insights_data_node0006 | 3.0.0.6 | DOWN | vertica-8.1.1.10 | insights_data
    v_insights_data_node0007 | 3.0.0.7 | DOWN | vertica-8.1.1.10 | insights_data

  • SruthiASruthiA Administrator

    Based on the below output, I recommend you to start node3 till node7. Once those are UP, you can try bringing node1 and node2 UP

    dbadmin@pl2:/$ admintools -t list_allnodes
    Node | Host | State | Version | DB
    --------------------------+---------+-------+------------------+---------------
    v_insights_data_node0001 | 3.0.0.1 | DOWN | unavailable | insights_data
    v_insights_data_node0002 | 3.0.0.2 | DOWN | unavailable | insights_data
    v_insights_data_node0003 | 3.0.0.3 | DOWN | vertica-8.1.1.10 | insights_data
    v_insights_data_node0004 | 3.0.0.4 | DOWN | vertica-8.1.1.10 | insights_data
    v_insights_data_node0005 | 3.0.0.5 | DOWN | vertica-8.1.1.10 | insights_data
    v_insights_data_node0006 | 3.0.0.6 | DOWN | vertica-8.1.1.10 | insights_data
    v_insights_data_node0007 | 3.0.0.7 | DOWN | vertica-8.1.1.10 | insights_data

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file