The Vertica Forum recently got a makeover! Let us know what you think by filling out this short, anonymous survey.

segfault in libverticaodbc through pyodbc

Here's an obnoxious problem that we've been trying to diagnose at my company: sometimes Python will segfault when using Vertica (via libverticaodbc) under pyodbc. The segfault is purely a function of the complexity of the Python program and seems to have nothing to do with what is being done in Vertica.

Here is a minimal reproduction case:

import pyodbc

def nest(depth, kallable, *args, **kwargs):
    """build a bunch of useless stack frames"""
    if depth <= 0:
        print(kallable, args, kwargs)
        return kallable(*args, **kwargs)
    return nest(depth - 1, kallable, *args, **kwargs)

def test_raw():
    conn = nest(50, pyodbc.connect, 'DRIVER={Vertica};SERVERNAME=localhost')
    print(conn)

if __name__ == '__main__':
    test_raw()

If you change the value of the first argument to nest, this program will eventually segfault when run on a Linux system with unixodbc (tested version 2.2.14 and 2.3.7), any Python version (tested 2.7, 3.5, 3.6, and 3.7), any pyodbc version (tested... a whole bunch of them) and any Vertica Client driver (tested 7.2.3, 8.1.1, and 9.1.1). The number of garbage stack frames you need to create varies (from a low of 13 to a high of 90) depending on the above parameters in a way I don't understand. It doesn't seem to be related to the value of ulimit -s

Top of a backtrace after a segfault:

#0  0x000014f4967fd672 in __gnu_cxx::__exchange_and_add (__mem=0x4801e88348018b40, __val=-1) at atomicity.cc:38
#1  0x000014f4944301bd in Simba::Support::DMCharacteristics::CheckAddress(unsigned long long, int, void*) () from /opt/vertica/lib64/libverticaodbc.so
#2  0x000014f494430379 in Simba::Support::DMCharacteristics::DetectDriverManager() () from /opt/vertica/lib64/libverticaodbc.so
#3  0x000014f4a2b124d3 in _PyEval_EvalCodeWithName (_co=0x14f4a2b41835, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=140733259890096, kwnames=0x4008e9, kwargs=0x20121fd70, kwcount=23040918399772, kwstep=90, defs=0x7fff03f6be40, defcount=19004784, kwdefs=0x59, closure=0x121fd00, name=0x14f4a1c0f2f3, qualname=0x7ffffbad8000)
    at Python/ceval.c:4159

After this, the backtrace starts to turn into gibberish.

As far as I can tell, Simba is part of the Vertica ODBC implementation.

Has anyone else seen anything like this? If so, anyone else figured out a way around it (short of "don't use that much stack before calling an ODBC function")? Am I going to need to spend a support incident to see if I can get an un-stripped copy of libverticaodbc.so and try to debug this?

Comments

  • Were you able to figure out a workaround to this? We just started running into something very similar where the segvault only presents in larger applications, even though the vertica connection works fine from a python shell on the same host.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file

Can't find what you're looking for? Search the Vertica Documentation, Knowledge Base, or Blog for more information.