Failure in UDx RPC call InvokeDestroyUDL(): timed out in receiving a UDx message

dcruise

I have been using this UDx for a while now and have had great success with it. I have used it on 2 Vertica instances before but I recently have developed my production cluster. I am utilizing this again and on some of the scripts it is working great, but on my mission critical scripts (of course) it has just been hanging. It will sit there endlessly until I kill the process (CTRL+C). Some works with the COPY tablename FROM LOCAL, and some work just with COPY tablename FROM. I am copying from a mounted drive, but again some of the scripts work just fine. 

Script I am running:
copy config from '/home/dbadmin/Files/*.xml' on v_test_node0001 with filter ExternalFilter(cmd='/home/dbadmin/VScripts/t.py');

Just hangs until I terminate:


I am pretty sure it is not the script because I have used them on other clusters and they work fine.

Scripts that completes successfully:
Works on a single file
Copy config from '/mnt/FCHadoop/Extraction/SN/sn.xml' WITH FILTER ExternalFilter(cmd= 'python /home/dbadmin/VScripts/ParseConfig.py');

Works on mulitple files with a different script


After digging through all of the logs I can find here are some of the highlighted anomalies that I see.

Vertica.log
"failure in UDx RPC call InvokeDestroyUDL(): timed out in receiving a UDx message"

Full Error:  Failure in UDx RPC call InvokeDestroyUDL(): timed out in receiving a UDx message
LOCATION:  makeUDxRemoteProcedureCallHandlingErrors, /scratch_a/release/30493/vbuild/vertica/EE/EEUtil/UDxFenceSupport.cpp:390
(this shows for each file that it is iterating through)

THIS ALSO HAS SHOWN UP SOME TIMES, EVEN ON SUCCESS
"Cannot getting Non-SelectPath ID because the Input Path of the Select Path is NULL"

UDxFencedProcesses.log
UDx side process started 18:58:48.001 [C++-VerticaNode1-46320:0x512-58107]  0x7fc35eb2f740 Has set to receive signal 9 when parent process dies
 18:58:48.001 [C++-VerticaNode1-46320:0x512-58107]  0x7fc35eb2f740 Finished setting up signal handlers.
 18:58:48.001 [C++-VerticaNode1-46320:0x512-58107]  0x7fc35eb2f740 My port: 50407
 18:58:48.001 [C++-VerticaNode1-46320:0x512-58107]  0x7fc35eb2f740 My address: 0.0.0.0
 18:58:48.001 [C++-VerticaNode1-46320:0x512-58107]  0x7fc35eb2f740 Vertica port: 34417
 18:58:48.001 [C++-VerticaNode1-46320:0x512-58107]  0x7fc35eb2f740 Vertica address: 127.0.0.1
 18:58:48.001 [C++-VerticaNode1-46320:0x512-58107]  0x7fc35eb2f740 Vertica address family: 2
 19:00:14.996 [C++-VerticaNode1-46320:0x512-58107]  0x7fc35d2e3700 Error in recv(): Vertica process has exited abnormally
 19:00:15.048 [C++-VerticaNode1-46320:0x512-58107]  0x7fc313fff700 Error in recv(): Vertica process has exited abnormally
 19:00:15.048 [C++-VerticaNode1-46320:0x512-58107]  0x7fc330ff9700 Error in recv(): Vertica process has exited abnormally
 19:00:15.048 [C++-VerticaNode1-46320:0x512-58107]  0x7fc331ffb700 Error in recv(): Vertica process has exited abnormally
 19:00:15.048 [C++-VerticaNode1-46320:0x512-58107]  0x7fc3327fc700 Error in recv(): Vertica process has exited abnormally
 19:00:15.048 [C++-VerticaNode1-46320:0x512-58107]  0x7fc3557fa700 Error in recv(): Vertica process has exited abnormally
 19:00:15.048 [C++-VerticaNode1-46320:0x512-58107]  0x7fc3317fa700 Error in recv(): Vertica process has exited abnormally
 19:00:16.001 [C++-VerticaNode1-46320:0x512-58107]  0x7fc35eb2f740 Received SIGTERM, exiting
 19:00:16.001 [C++-VerticaNode1-46320:0x512-58107]  0x7fc35c8a4700 Received SIGTERM, exiting

CopyErrorLogs
shows a file for each one of the files I am iterating through, but these files are blank.