segfault in libverticaodbc through pyodbc

jbrown

Here's an obnoxious problem that we've been trying to diagnose at my company: sometimes Python will segfault when using Vertica (via libverticaodbc) under pyodbc. The segfault is purely a function of the complexity of the Python program and seems to have nothing to do with what is being done in Vertica.

Here is a minimal reproduction case:

import pyodbc

def nest(depth, kallable, *args, **kwargs):
    """build a bunch of useless stack frames"""
    if depth <= 0:
        print(kallable, args, kwargs)
        return kallable(*args, **kwargs)
    return nest(depth - 1, kallable, *args, **kwargs)

def test_raw():
    conn = nest(50, pyodbc.connect, 'DRIVER={Vertica};SERVERNAME=localhost')
    print(conn)

if __name__ == '__main__':
    test_raw()

If you change the value of the first argument to nest, this program will eventually segfault when run on a Linux system with unixodbc (tested version 2.2.14 and 2.3.7), any Python version (tested 2.7, 3.5, 3.6, and 3.7), any pyodbc version (tested... a whole bunch of them) and any Vertica Client driver (tested 7.2.3, 8.1.1, and 9.1.1). The number of garbage stack frames you need to create varies (from a low of 13 to a high of 90) depending on the above parameters in a way I don't understand. It doesn't seem to be related to the value of ulimit -s

Top of a backtrace after a segfault:

#0  0x000014f4967fd672 in __gnu_cxx::__exchange_and_add (__mem=0x4801e88348018b40, __val=-1) at atomicity.cc:38
#1  0x000014f4944301bd in Simba::Support::DMCharacteristics::CheckAddress(unsigned long long, int, void*) () from /opt/vertica/lib64/libverticaodbc.so
#2  0x000014f494430379 in Simba::Support::DMCharacteristics::DetectDriverManager() () from /opt/vertica/lib64/libverticaodbc.so
#3  0x000014f4a2b124d3 in _PyEval_EvalCodeWithName (_co=0x14f4a2b41835, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=140733259890096, kwnames=0x4008e9, kwargs=0x20121fd70, kwcount=23040918399772, kwstep=90, defs=0x7fff03f6be40, defcount=19004784, kwdefs=0x59, closure=0x121fd00, name=0x14f4a1c0f2f3, qualname=0x7ffffbad8000)
    at Python/ceval.c:4159

After this, the backtrace starts to turn into gibberish.

As far as I can tell, Simba is part of the Vertica ODBC implementation.

Has anyone else seen anything like this? If so, anyone else figured out a way around it (short of "don't use that much stack before calling an ODBC function")? Am I going to need to spend a support incident to see if I can get an un-stripped copy of libverticaodbc.so and try to debug this?

pcichons

Were you able to figure out a workaround to this? We just started running into something very similar where the segvault only presents in larger applications, even though the vertica connection works fine from a python shell on the same host.