Building FilterFunctions in Vertica 9.1.0-1 Fails

Terry_Caudill

Environment:
Vertica BYOL, Amazon Linux 2.0 Version: 9.1.0-1

I was able to build the filter functions in Vertica 9.0 (though it was on CentOS), but when I attempt in Vertica 9.1.0-1 I am getting the following:

cd /opt/vertica/sdk/examples
make

g++ -I /opt/vertica/sdk/include -I HelperLibraries -g -Wall -Wno-unused-value -shared -fPIC -o /opt/vertica/sdk/examples/build/ScalarFunctions.so ScalarFunctions/.cpp /opt/vertica/sdk/include/Vertica.cpp
In file included from /opt/vertica/sdk/include/Vertica.h:75:0,
from ScalarFunctions/Add2Ints.cpp:7:
/opt/vertica/sdk/include/VerticaUDx.h: In member function ‘void Vertica::UDXObject::cancelUDX(Vertica::ServerInterface&)’:
/opt/vertica/sdk/include/VerticaUDx.h:2305:57: error: operand type ‘bool’ is incompatible with argument 1 of ‘__sync_fetch_and_or’
if (!__sync_fetch_and_or(&(this->canceled), true)) {
^
In file included from /opt/vertica/sdk/include/Vertica.h:75:0,
from ScalarFunctions/AddAnyInts.cpp:9:
/opt/vertica/sdk/include/VerticaUDx.h: In member function ‘void Vertica::UDXObject::cancelUDX(Vertica::ServerInterface&)’:
/opt/vertica/sdk/include/VerticaUDx.h:2305:57: error: operand type ‘bool*’ is incompatible with argument 1 of ‘__sync_fetch_and_or’
if (!__sync_fetch_and_or(&(this->canceled), true)) {
^

Terry_Caudill

So its complaining about the first argument type for __sync_fetch_and_or(). This code is in the VerticaUDx.h:

void cancelUDX(ServerInterface &srvInterface) {
     if (!__sync_fetch_and_or(&(this->canceled), true)) {                                                                                                                          
     cancel(srvInterface);
  }
}

I convert the first arg to an integer and it compiles:

void cancelUDX(ServerInterface &srvInterface) {
  int local_canceled = this->canceled !=0 ? 1 : 0;
  if (!__sync_fetch_and_or(&local_canceled, true)) {
     cancel(srvInterface);
  }
}

Don't know if this is what is expected at runtime.

Sergey_Cherepan_1

That would be a very dumb fix (sorry for words)
Goal of __sync_fetch_and_or is to extract atomic value of cancel and set it to true. Use of atomic allow not to acquire lock.
Fix circumvent atomic operation, canceled is not being set etc. Effectively, fix is breaking code.

gcc __sync_fetch_and_or is an extension of Intel API and accept 1,2,4 and 8 bytes integer parameter. Version of gcc >= 7 (???) do not convert bool into single-byte integral by default, and code error out on compilation.
Correct fix would be

    if (!__sync_fetch_and_or(reinterpret_cast<unsigned char*>(&(this->canceled)), reinterpret_cast<unsigned char*>(true))) {

Not to mention, Vertica developers are not testing their code against newer compilers. Vertica docs 9.2 say "need g++" to compile UDx, but then it says "some distribution have not compatible g++" and requiring to install g++ v4.8.
Vertica is still on very old and now looking odd g++ 4.8. That is more than 10 years old.

AnitaM77

Hi Sergey, Tried to use your fix it worked for overcoming the above error. We have upgraded to 9.2 and installing extension packages.

But running into this error -

vsql:/opt/vertica/packages/Vertica-Extension-Packages-master/strings_package/ddl/install.sql:7: ROLLBACK 6850: Library [StringsLib] built with incompatible SDK version [8.1.1]. Please rebuild with SDK version [9.2.0] and recreate the library
vsql:/opt/vertica/packages/Vertica-Extension-Packages-master/strings_package/ddl/install.sql:8: ROLLBACK 6850: Library [StringsLib] built with incompatible SDK version [8.1.1]. Please rebuild with SDK version [9.2.0] and recreate the library
vsql:/opt/vertica/packages/Vertica-Extension-Packages-master/strings_package/ddl/install.sql:9: ROLLBACK 6850: Library [StringsLib] built with incompatible SDK version [8.1.1]. Please rebuild with SDK

Any pointers? Thanks!

AnitaM77

Looking forward for some responses on my question above. Thanks!

skeswani

I think the right way to avoid this error is to use gcc 4.8 as shown here
if (!__sync_fetch_and_or(&(this->canceled), true)) {

https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/ExtendingVertica/UDx/DevEnvironment.htm

instead of editing the code, you can use something like this where you select the correct version of g++
! /usr/bin/g++-4.8 -D HAVE_LONG_LONG_INT_64 -g -Wall -Wno-unused-value -shared -fPIC -I /opt/vertica/sdk/include/ -o /tmp/Reader.so ./Reader.cpp /opt/vertica/sdk/include/Vertica.cpp

smithclarkson01

hink the right way to avoid this error is to use gcc 4.8 as shown here
if (!__sync_fetch_and_or(&(this->canceled), true)) {

https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/ExtendingVertica/UDx/DevEnvironment.htm

instead of editing the code, you can use something like this where you select the correct version of g++
! /usr/bin/g++-4.8 -D HAVE_LONG_LONG_INT_64 -g -Wall -Wno-unused-value -shared -fPIC -I /opt/vertica/sdk/include/ -o /tmp/Reader.so ./Reader.cpp /opt/vertica/sdk/include/Vertica.cpp

that's true indeed... worked for me !

smithclarkson01

Hi Sergey, Tried to use your fix it worked for overcoming the above error. We have upgraded to 9.2 and installing extension packages.

But running into this error -

vsql:/opt/vertica/packages/Vertica-Extension-Packages-master/strings_package/ddl/install.sql:7: ROLLBACK 6850: Library [StringsLib] built with incompatible SDK version [8.1.1]. Please rebuild with SDK version [9.2.0] and recreate the library
vsql:/opt/vertica/packages/Vertica-Extension-Packages-master/strings_package/ddl/install.sql:8: ROLLBACK 6850: Library built with incompatible SDK version [8.1.1]. Please rebuild with SDK version [9.2.0] and recreate the library
vsql:/opt/vertica/packages/Vertica-Extension-Packages-master/strings_package/ddl/install.sql:9: ROLLBACK 6850: Library [StringsLib] built with incompatible SDK version [8.1.1]. Please rebuild with SDK

Any pointers?
it makes sense, doesn't it ?

SruthiA

please reinstall packages using the following command

admintools -t install_package -d db-name -p password -P all --force-reinstall

and then re create all your existing libraries.

lovelash

I think the right way to avoid this error is to use gcc 4.8 as shown here
if (!__sync_fetch_and_or(&(this->canceled), true)) {

https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/ExtendingVertica/UDx/DevEnvironment.htm

instead of editing the code, you can use something like this where you select the correct version of g++
! /usr/bin/g++-4.8 -D HAVE_LONG_LONG_INT_64 -g -Wall -Wno-unused-value -shared -fPIC -I /opt/vertica/sdk/include/ -o /tmp/Reader.so ./Reader.cpp /opt/vertica/sdk/include/Vertica.cpp

LenoyJ

@smithclarkson01 you'll need to recompile your UDx's (including Vertica Extension Package) every time you upgrade major versions. See here: https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/ExtendingVertica/UDx/UpdatingUDFsAfterAServerUpgrade.htm

rohan123

@lovelash said:
I think the right way to avoid this error is to use gcc 4.8 as shown here
if (!__sync_fetch_and_or(&(this->canceled), true)) {

https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/ExtendingVertica/UDx/DevEnvironment.htm

instead of editing the code, you can use something like this where you select the correct version of g++
! /usr/bin/g++-4.8 -D HAVE_LONG_LONG_INT_64 -g -Wall -Wno-unused-value -shared -fPIC -I /opt/vertica/sdk/include/ -o /tmp/Reader.so ./Reader.cpp /opt/vertica/sdk/include/Vertica.cpp

Thankyou

scottsen11

Vertica COPY command expects an uncompressed CSV input by default. GZIP option is available when source data files are located on the node, but not available when using the S3 function since it is based on the User-Defined Load (UDL) feature.