[Vertica][VJDBC](4236) ROLLBACK: One or more nodes did not open a data connection to this node.

verbanverban Vertica Customer

Hi,
I have 3 nodes cluster DB (v9.2.0) and yesterday i found there is NTP issue.
i fixed times manually (there was 1-10 minutes time shift between nodes),
few minutes later i got most queries failed.
now:

  • Each node can ping others (same as before)
  • There is no networking issue (vnetperf result is fine and spread is ok!)
  • All nodes are UP (i stop/start recently but it not help)
  • v_monitor.error_messages has too many error such as:

One or more nodes did not open a data connection to this node. This may indicate a network configuration problem. Check that the private interfaces used for communication among the cluster hosts reside in the same subnet and are returned first by host address lookup
Recceive on v_db_node0001: query has been canceled
RecvFiles on v_db_node0002: Open failed on node [v_db_node0003] ()
DataTargetProxy on v_db_node0002: handle is canceled

Please let me know what's doing wrong?

Tagged:

Best Answer

Answers

  • Bryan_HBryan_H Vertica Employee Administrator

    Is SSL enabled? Clock drift will cause issues with SSL and may require reboot of all hosts to synchronize clocks.
    Were there any messages in vertica.log on the initiator node, or messages in dmesg or /var/log/messages on any node indicating a system issue?
    It's recommended to open a support case if you can.

  • verbanverban Vertica Customer

    Thanks Bryan,
    No, SSL is not enabled,
    After 2 days i found the issue. as mentioned in error log it was a networking issue. But i don't know what exactly it was!
    I deleted all routes on nodes and redefined all of them.
    Now I'm curious about it, because as least ping was OK and also spread (traffic) was working!!
    How to t-shoot it next time?

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file