Please take this survey to help us learn more about how you use third party tools. Your input is greatly appreciated!

[Vertica][VJDBC](4236) ROLLBACK: One or more nodes did not open a data connection to this node.

Hi,
I have 3 nodes cluster DB (v9.2.0) and yesterday i found there is NTP issue.
i fixed times manually (there was 1-10 minutes time shift between nodes),
few minutes later i got most queries failed.
now:

  • Each node can ping others (same as before)
  • There is no networking issue (vnetperf result is fine and spread is ok!)
  • All nodes are UP (i stop/start recently but it not help)
  • v_monitor.error_messages has too many error such as:

One or more nodes did not open a data connection to this node. This may indicate a network configuration problem. Check that the private interfaces used for communication among the cluster hosts reside in the same subnet and are returned first by host address lookup
Recceive on v_db_node0001: query has been canceled
RecvFiles on v_db_node0002: Open failed on node [v_db_node0003] ()
DataTargetProxy on v_db_node0002: handle is canceled

Please let me know what's doing wrong?

Tagged:

Best Answer

Answers

  • Is SSL enabled? Clock drift will cause issues with SSL and may require reboot of all hosts to synchronize clocks.
    Were there any messages in vertica.log on the initiator node, or messages in dmesg or /var/log/messages on any node indicating a system issue?
    It's recommended to open a support case if you can.

  • Thanks Bryan,
    No, SSL is not enabled,
    After 2 days i found the issue. as mentioned in error log it was a networking issue. But i don't know what exactly it was!
    I deleted all routes on nodes and redefined all of them.
    Now I'm curious about it, because as least ping was OK and also spread (traffic) was working!!
    How to t-shoot it next time?

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file

Can't find what you're looking for? Search the Vertica Documentation, Knowledge Base, or Blog for more information.