Unable to close session "shutdown error: Transport endpoint is not connected"

LucasLedesma

Guys,

 

We are having troubles to kill some long running Vertica queries.

 

This is what we see in the logs when trying to cancel the session:

 

Initiator Node:

 

2016-11-22 05:59:37.014 Spread Client:0xae17920 [VMPI] <INFO> Removing 45036012405448168 from list of initialized nodes for session v_cnamx_node0024-333361:0x3014b
2016-11-22 07:01:34.772 Spread Client:0xae17920 [VMPI] <INFO> Removing 45036012405394718 from list of initialized nodes for session v_cnamx_node0024-333361:0x3014b
2016-11-22 09:22:45.243 Init Session:0x7f7da0004420 [Session] <INFO> [Query] TX:0(v_cnamx_node0024-333361:0x3b0a2) select close_session('v_cnamx_node0024-333361:0x3014b');
2016-11-22 09:23:48.785 Init Session:0x7f7da0004420 [Session] <INFO> [Query] TX:0(v_cnamx_node0024-333361:0x3b0a2) select * from sessions where session_id = 'v_cnamx_node0024-333361:0x3014b';
2016-11-22 09:23:48.840 Init Session:0x7f7da0004420-21000000022a8bc [Txn] <INFO> Begin Txn: 21000000022a8bc 'select * from sessions where session_id = 'v_cnamx_node0024-333361:0x3014b';'
2016-11-22 09:23:55.496 Init Session:0x7f7da0004420-21000000022a8bc [Session] <INFO> [Query] TX:21000000022a8bc(v_cnamx_node0024-333361:0x3b0a2) select close_session('v_cnamx_node0024-333361:0x3014b');
2016-11-22 09:23:55.892 Init Session:0x7f7da0004420-21000000022a8bc [Session] <WARNING> session v_cnamx_node0024-333361:0x3014b socket shutdown error: Transport endpoint is not connected

 

OTHER NODES:

 

2016-11-22 09:22:46.473 DistCall Dispatch:0x7fcad005c3f0 [Session] <WARNING> setClose: no session v_cnamx_node0024-333361:0x3014b to close
2016-11-22 09:22:46.525 Spread Client:0xbbd16f0 [Session] <WARNING> Trying to cancel Session v_cnamx_node0024-333361:0x3014b but have not found any cancel handle.
2016-11-22 09:23:55.895 DistCall Dispatch:0x7fcad005c3f0-21000000022a8bc [Session] <WARNING> setClose: no session v_cnamx_node0024-333361:0x3014b to close
2016-11-22 09:23:56.313 Spread Client:0xbbd16f0 [Session] <WARNING> Trying to cancel Session v_cnamx_node0024-333361:0x3014b but have not found any cancel handle.

 

The only way we have to cancel this transactions is by restarting the initiator node.

 

Any idea how we can solve this issue? It is becoming frequent (at least once per day).

 

Thanks,

Lucas Ledesma.-

 

 

Adrian_Oprea_1

 Hi , 

track you running sessions by using this SQL:

 

The output will provide you with the close sesison syntax as well, it work every time

SELECT
     node_name
    ,user_name
    ,'SELECT CLOSE_SESSION(''' || session_id || ''');'  AS CloseSession
    ,statement_start
    ,(GETDATE() - statement_start)::INTERVAL  AS current_statement_duration  
    ,REGEXP_REPLACE(current_statement,'[\r\n\t]',' ') AS current_statement 
    ,session_id
    ,transaction_id
    ,statement_id
    ,client_hostname
	,client_os
    ,login_timestamp
    ,runtime_priority
    ,ssl_state
    ,authentication_method
    ,transaction_start
    ,GETDATE() AS Today
FROM v_monitor.sessions
ORDER BY current_statement_duration DESC 
;

LucasLedesma

The session is not reponding to the attempt to close it or cancel-interrupt the query.

The only way to stop it is to stop the vertica process on the initiator node and then restart it. The query seems to be hung.

 

ashendra

We're you able to resolve this without restarting the indicator node. We're you able to identify why session goes into a hung state.