Queries being routed to node which is down

Mark_1

We run a 6 node cluster V7.2.3-19. Node 6 unexpectedly crashed, the server itself rebooted and Vertica on that node stayed down for 2hrs until it was force restarted. The Vertica cluster stayed up throughout on the remaining 5 nodes, we run k-safe 1.

We have a F5 load balancer through which the application connects via. Although we can see that no connections got routed to node 6 while it was down, we still received the following errors from sessions that connected to other nodes (node 1 in this example) and attempted to run queries:

@v_marketdata_node0001: 08006/4539: Received no response from v_marketdata_node0006 in session reset

From looking through the logs of the 5 active nodes during this time, this is the pattern:

@v_marketdata_node0001: 00000/3316: Executing statement: '--^M

Comes:
[PQuery] TX:a00000214e173c(_marketdata_node0001-15601:0x461ce0) select reset_session();
Init Session:0x7f4a440ae880-a00000214e173c [Session] [Query] TX:a00000214e173c(_marketdata_node0001-15601:0x461ce0) select reset_session();

Followed by:
Rollback Txn: a00000214e173c

Init Session:0x7f4a440ae880 @v_marketdata_node0001: 08006/4539: Received no response from v_marketdata_node0006 in session reset

Any ideas as to why queries were still trying to fan out across node 6? Client is ado.net version 07.02.0312.

Thanks

Jim_Knicely

Hi,

The ado.net driver used to issue the reset_session() function on every node in a Vertica cluster, even those that were down... If you upgrade your ado.net driver to 7.2, you'll actually get an exception and the connection will fail! So don't do that!

This issue is fixed in Vertica 9.0:

9.0.0-0 (10/16/2017) - Client Drivers - ADO: The RESET_SESSION session management function now functions properly when run against a cluster with a DOWN node.

Mark_1

Thanks Jim - very useful