How can I update Protol Buffer runtime library

JamesLeslieJamesLeslie Community Edition User
edited August 2022 in General Discussion

I am trying to execute a Python UDx which has a few dependencies.

I have created a conda environment on the same server where Vertica is installed, which is running Redhat x86 OS. For this conda environment, all dependencies are stored in a folder called site-packages inside the conda env folder.

The definition for my conda env is as follows:

name: ner
channels:
  - defaults
dependencies:
  - python=3.9.6
  - pip
  - numpy
  - protobuf
  - spacy

When creating the UDx library I included the DEPENDS parameter as follows:

CREATE OR REPLACE LIBRARY textprocess AS '/opt/apps/DataScience/perform_NER/perform_NER.py' DEPENDS '/opt/apps/DataScience/perform_NER/site-packages' LANGUAGE 'Python';

CREATE OR REPLACE TRANSFORM FUNCTION perform_NER AS LANGUAGE 'Python' NAME 'PerformNERFactory' LIBRARY textprocess fenced;

It installs successfully, but when I try to invoke the UDx I get the following error message:

[Code: 3399, SQL State: VP001]  [Vertica][VJDBC](3399) ERROR: Failure in UDx RPC call InvokeProcessPartition(): Exception calling processPartition() in User Defined Object [perform_NER], message: This program requires version 3.9.0 of the Protocol Buffer runtime library, but the installed version is 3.6.1.  Please update your library.  If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library.  (Version verification failed in "bazel-out/k8-opt/bin/tensorflow/core/framework/tensor_shape.pb.cc".)

So it looks like I need to update the Protocol Buffer runtime library. Please could someone show me how to update the version of this library that is used by Vertica.

We are using Vertica 12.0

Answers

  • JamesLeslieJamesLeslie Community Edition User

    Sorry, I forgot to mention: we are using Vertica 12.0

  • Bryan_HBryan_H Vertica Employee Administrator

    The UDx process should pick up the protobuf library in the DEPENDS path. Could you verify the protobuf version in your conda virtualenv with: python3 -m pip list

  • JamesLeslieJamesLeslie Community Edition User

    Sure, the output when I run conda list shows:

    protobuf                  4.21.5                   pypi_0    pypi
    

    I tried to delete and recreate the environment where I specified to use pip for protobuf and spacy, as the PyPi versions of these packages are more recent:

    name: ner
    channels:
      - defaults
    dependencies:
      - python=3.9.6
      - pip
      - numpy
      pip:
        - protobuf
        - spacy
    
  • SruthiASruthiA Vertica Employee Administrator

    @JamesLeslie : This is a known issue. Could you please open a support case for it?

  • JamesLeslieJamesLeslie Community Edition User

    Yes, sorry for the silly question but how do I open a support case?

  • SruthiASruthiA Vertica Employee Administrator

    Please click on cases in the below link and open a support case after you login.

    https://portal.microfocus.com/s/?language=en_US

  • JamesLeslieJamesLeslie Community Edition User

    I create an account there but it says I do not have permission to access resources:

  • SruthiASruthiA Vertica Employee Administrator

    Please check with your Manager. If you have a valid support contract with vertica, you will be able to create support cases.

  • JamesLeslieJamesLeslie Community Edition User

    Thanks, working on that now. Is there any workaround I can do in the meantime? I am trying to figure out how to specifically use a version of SpaCy that uses Protocol Buffer runtime library v3.6.1. (the version that the error message tells me is installed)

  • Bryan_HBryan_H Vertica Employee Administrator

    In your virtualenv, you can install specific protobuf version with "python3 -m pip install protobuf==3.6.1"

  • JamesLeslieJamesLeslie Community Edition User

    Thanks Bryan, the version of protobuf in my conda environment isn't the issue, as all code runs inside my own virtual environment without any issues.
    Furthermore, the version of protobuf (the Python package) seems to be a different thing to the Protocol Buffer runtime library. I have tried with various version of protobuf (currently using 4.21.5) and yet the error message is always the same when I try to run the UDx.

    It doesn't seem to make a difference which version of protobuf I install in my conda environment, it will always say that I have Protocol Buffer runtime library v3.6.1

  • Bryan_HBryan_H Vertica Employee Administrator

    I haven't worked with protobuf directly in a while, but I note in the docs that the compiler "protoc" and C++ API version need to match the installed library, so it may be picking up the version from a C++ library (notably the Python UDX wrapper binary "vertica-udx-Python" links to protobuf C++ v3.6.1). This is a case where it might be necessary to build and install v3.6.1 protoc and Python package from source though I'm checking with our internal Python team whether they have any ideas for this.

  • JamesLeslieJamesLeslie Community Edition User
    edited September 2022

    Hi @Bryan_H , have you managed to check with the Python team about this issue?
    I logged a case through support, and my reference number is 02380459

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file