How can I update Protol Buffer runtime library

JamesLeslie

I am trying to execute a Python UDx which has a few dependencies.

I have created a conda environment on the same server where Vertica is installed, which is running Redhat x86 OS. For this conda environment, all dependencies are stored in a folder called site-packages inside the conda env folder.

The definition for my conda env is as follows:

name: ner
channels:
  - defaults
dependencies:
  - python=3.9.6
  - pip
  - numpy
  - protobuf
  - spacy

When creating the UDx library I included the DEPENDS parameter as follows:

CREATE OR REPLACE LIBRARY textprocess AS '/opt/apps/DataScience/perform_NER/perform_NER.py' DEPENDS '/opt/apps/DataScience/perform_NER/site-packages' LANGUAGE 'Python';

CREATE OR REPLACE TRANSFORM FUNCTION perform_NER AS LANGUAGE 'Python' NAME 'PerformNERFactory' LIBRARY textprocess fenced;

It installs successfully, but when I try to invoke the UDx I get the following error message:

[Code: 3399, SQL State: VP001]  [Vertica][VJDBC](3399) ERROR: Failure in UDx RPC call InvokeProcessPartition(): Exception calling processPartition() in User Defined Object [perform_NER], message: This program requires version 3.9.0 of the Protocol Buffer runtime library, but the installed version is 3.6.1.  Please update your library.  If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library.  (Version verification failed in "bazel-out/k8-opt/bin/tensorflow/core/framework/tensor_shape.pb.cc".)

So it looks like I need to update the Protocol Buffer runtime library. Please could someone show me how to update the version of this library that is used by Vertica.

We are using Vertica 12.0

JamesLeslie

Sorry, I forgot to mention: we are using Vertica 12.0

Bryan_H

The UDx process should pick up the protobuf library in the DEPENDS path. Could you verify the protobuf version in your conda virtualenv with: python3 -m pip list

JamesLeslie

Sure, the output when I run conda list shows:

protobuf                  4.21.5                   pypi_0    pypi

I tried to delete and recreate the environment where I specified to use pip for protobuf and spacy, as the PyPi versions of these packages are more recent:

name: ner
channels:
  - defaults
dependencies:
  - python=3.9.6
  - pip
  - numpy
  pip:
    - protobuf
    - spacy

SruthiA

@JamesLeslie : This is a known issue. Could you please open a support case for it?

JamesLeslie

Yes, sorry for the silly question but how do I open a support case?

SruthiA

Please click on cases in the below link and open a support case after you login.

https://portal.microfocus.com/s/?language=en_US

JamesLeslie

I create an account there but it says I do not have permission to access resources:

SruthiA

Please check with your Manager. If you have a valid support contract with vertica, you will be able to create support cases.

JamesLeslie

Thanks, working on that now. Is there any workaround I can do in the meantime? I am trying to figure out how to specifically use a version of SpaCy that uses Protocol Buffer runtime library v3.6.1. (the version that the error message tells me is installed)

Bryan_H

In your virtualenv, you can install specific protobuf version with "python3 -m pip install protobuf==3.6.1"

JamesLeslie

Thanks Bryan, the version of protobuf in my conda environment isn't the issue, as all code runs inside my own virtual environment without any issues.
Furthermore, the version of protobuf (the Python package) seems to be a different thing to the Protocol Buffer runtime library. I have tried with various version of protobuf (currently using 4.21.5) and yet the error message is always the same when I try to run the UDx.

It doesn't seem to make a difference which version of protobuf I install in my conda environment, it will always say that I have Protocol Buffer runtime library v3.6.1

Bryan_H

I haven't worked with protobuf directly in a while, but I note in the docs that the compiler "protoc" and C++ API version need to match the installed library, so it may be picking up the version from a C++ library (notably the Python UDX wrapper binary "vertica-udx-Python" links to protobuf C++ v3.6.1). This is a case where it might be necessary to build and install v3.6.1 protoc and Python package from source though I'm checking with our internal Python team whether they have any ideas for this.

JamesLeslie

Hi @Bryan_H , have you managed to check with the Python team about this issue?
I logged a case through support, and my reference number is 02380459