UDx crashed in production due to pytorch dependency

karthigaimuthu

We have recently upgraded our vertica from 10.0.1-2 to 11.0.1-2 , we had created on udx with pytorch library for one of the ML use case.

But , in the recent times udx is getting crashed whenever we call it. Please find below the error traces we aren't sure what's needs to be done over here to solve it.

Error Trace:

QueryError: Severity: ERROR, Message: Failure in UDx RPC call InvokeGetParameterType(): Error calling setupExecContext() in User Defined Object [pytorch_predict] at [/data/jenkins/workspace/RE-ReleaseBuilds/RE-Jackhammer_2/server/vertica/OSS/UDxFence/PythonInterface.cpp:176], error code: 0, message: Error [/data/jenkins/workspace/RE-ReleaseBuilds/RE-Jackhammer_2/server/vertica/OSS/UDxFence/PythonInterface.cpp:212] function ['import']
(Python error type [])
Traceback (most recent call last):
File "/vertica/data/smartech/v_smartech_node0007_catalog/Libraries/02265b6c4cc535e2d115c38a5c5d342a01000011ca9d4e0c/pytorch_predict_02265b6c4cc535e2d115c38a5c5d342a01000011ca9d4e0c.py", line 1, in
import torch
File "/vertica/data/smartech/v_smartech_node0007_catalog/Libraries/02265b6c4cc535e2d115c38a5c5d342a01000011ca9d4e0c/site-packages/torch/init.py", line 207, in
raise ImportError(textwrap.dedent('''
ImportError: Failed to load PyTorch C extensions:
It appears that PyTorch has loaded the torch/_C folder
of the PyTorch repository rather than the C extensions which
are expected in the torch._C namespace. This can occur when
using the install workflow. e.g.
$ python setup.py install && python -c "import torch"

This error can generally be solved using the `develop` workflow
    $ python setup.py develop && python -c "import torch"  # This should succeed
or by running Python from a different directory.

, Sqlstate: VP001, Routine: performUDxCall, File: /data/jenkins/workspace/RE-ReleaseBuilds/RE-Jackhammer_2/server/vertica/EE/EEUtil/UDxFenceSupport.cpp, Line: 561, Error Code: 3399, SQL: "SELECT pytorch_predict(* USING PARAMETERS input_folder = '/efs/PREDICTIVE_SEGMENTS/mothersparshcee/LIVE', output_folder = '/efs/PREDICTIVE_SEGMENTS/mothersparshcee/PS_7/LIVE_PROBS', weights_file = '/efs/PREDICTIVE_SEGMENTS/mothersparshcee/PS_7/model_weights/model.pth', segment = 'PS', output_columns = 0 ) OVER() FROM (SELECT 1) AS FOO;"

SergeB

Starting with 11.0.1 Vertica moved to Python 3.9 (was 3.7 in previous versions), it's possible Pytorch might have to be reinstalled and your UDX library with Pytortch dependency dropped and reloaded.

melonygunn

@SergeB Wordle said:
Starting with 11.0.1 Vertica moved to Python 3.9 (was 3.7 in previous versions), it's possible Pytorch might have to be reinstalled and your UDX library with Pytortch dependency dropped and reloaded.

Thanks for the information. It really helped me!