How to delay Vertica node shutdown when k-safety assessment fails?
We're using a 3 nodes Vertica cluster. The network connection between the nodes sometimes fails for a short amount of time (ex : 30 seconds).
When this happens, all nodes quickly shut down as soon as they detect that other nodes are unreachable (because k-safety cannot be satisfied). For example, the following sequence is recorded in the vertica log by the node0003 :
00:04:30.633 node v_feedback_node0001 left the cluster ... 00:04:30.670 Node left cluster, reassessing k-safety... ... 00:04:32.389 node v_feedback_node0002 left the cluster ... 00:04:32.414 Changing node v_feedback_node0003 startup state from UP to UNSAFE ... 00:04:33.425 Shutting down this node ... 00:04:38.547 node v_feedback_node0003 left the cluster
Any of you knows if it's possible to configure a delay after which each node will try to reconnect to others before giving up and shutting down ?