Options

one vertica Node is down in cluster

MonikaMonika Community Edition User

Initially, our Vertica cluster consisted of three nodes, but unfortunately, one of them became permanently unavailable. To rectify the issue, we replaced the nonfunctional node with a new one (v_node0003). However, upon adding the new node to the cluster, we encountered a problem where the database was unable to operate with all three nodes. Specifically, the third node kept going down every ten minutes. Despite having already enabled Passwordless ssh across all three nodes, the issue persists.

Answers

  • Options
    moshegmosheg Vertica Employee Administrator
    edited April 2023

    Please open a support case to investigate why the same node goes down.

    In general, you can see messages in the file vertica.log
    To find where vertica.log file is located run the command: admintools -t list_db -d YourDatabaseName

    In vertica.log a [word] in square brackets indicates the source of the log entry.
    A <word> in angle brackets indicates the severity of the message.

    In the following list <LOG> reflects the lowest severity and <PANIC> the highest.
    <LOG>, <INFO>, <NOTICE>, <WARNING>, <ERROR>, <ROLLBACK>, <INTERNAL>, <FATAL> , <PANIC>

    To start investigate why a node goes down scan vertica.log for messages, on all nodes, with high severities.
    For example: <INTERNAL>, <FATAL> and <PANIC> severities.
    In addition, strings like “Shutdown” and “shutting down” will point you towards parts of the log you are interested in.

    For example:
    grep -n -i -e 'FATAL' -e '<INTERNAL>' -e '<PANIC>' -e 'signal' -e 'select shutdown' -e 'NETWORK change with 1 VS sets' vertica.log

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file