python3 + rpy + pyodb + vertica 7 = Linux segfault

matthewcornell · January 2014

Hi Folks,

I've discovered a python-vertica stack configuration that causes a segmentation fault on our Linux cluster. I'd really appreciate your help in deciding how to proceed with a solution. I've attached as much information as I can to help figure this out.

The summary: When running in a virtualenv, if I import rpy2.objects then pyodb and then try to execute certain queries, I get the segfault. Interestingly, if I swap the import order, it seems to work OK (!) Testing on previous versions of pyodb have the segfault happening at connect() instead of execute().

Thanks very much,

matt

** commands to install the stack via virtualenv

/share/apps/python3/bin/python3.3 ~/bin/virtualenv.py --no-site-packages ~/virtualenvs/vertica-test
source ~/virtualenvs/vertica-test/bin/activate

pip install http://pyodbc.googlecode.com/files/pyodbc-3.0.7.zip
-> Successfully installed pyodbc

pip install rpy2
-> Successfully installed rpy2

** commands in interpreter to reproduce the segfault

# NB: bad username or password -> segfault at connect()
# NB: if the import order is switched -> NO segfault

[cornell@wright ~]$ source ~/virtualenvs/vertica-test/bin/activate

(vertica-test)[cornell@wright ~]$ python3.3
Python 3.3.2 (default, May 16 2013, 11:31:48)
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import rpy2.robjects
>>> import pyodbc
>>> conn = pyodbc.connect('DSN=vertica_kdl_dsn;UID=xx;PWD=xx')
>>> curs = conn.cursor()
>>> curs.execute('DROP TABLE IF EXISTS foo')
Segmentation fault

(vertica-test)[cornell@wright ~]$ python3.3
Python 3.3.2 (default, May 16 2013, 11:31:48)
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyodbc
>>> import rpy2.robjects
>>> conn = pyodbc.connect('DSN=vertica_kdl_dsn;UID=xx;PWD=xx')
>>> curs = conn.cursor()
>>> curs.execute('DROP TABLE IF EXISTS foo')
<pyodbc.Cursor object at 0x7f8508b5fe10>
>>>

** versions

Python 3.3.2 (default, May 16 2013, 11:31:48)
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux

>>> rpy2.__version__
'2.3.8'

>>> pyodbc.version
'3.0.7'
>>> pyodbc.SQL_DBMS_VER
18
>>> pyodbc.SQL_DM_VER
171
>>> pyodbc.SQL_DRIVER_ODBC_VER
77
>>> pyodbc.SQL_DRIVER_VER
7
>>> pyodbc.SQL_ODBC_VER
10

$ isql --version
unixODBC 2.2.14

/opt/vertica/lib64:
lrwxrwxrwx 1 root root       23 Dec 18 09:43 libverticaodbc.so.7.0 -> libverticaodbc.so.7.0.0

[dbadmin@compute-0-0 ~]$ /opt/vertica/bin/adminTools
Vertica Analytic Database 7.0.0-0 Administration Tools

[cornell@wright ~]$ uname -a
Linux wright.cs.umass.edu 2.6.32-279.14.1.el6.x86_64 #1 SMP Tue Nov 6 23:43:09 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
[cornell@wright ~]$ cat /proc/version
Linux version 2.6.32-279.14.1.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Tue Nov 6 23:43:09 UTC 2012
[cornell@wright ~]$

** odbc configuration files

*** /etc/odbcinst.ini

[PostgreSQL]
...

[MySQL]
...

[vertica_driver]
Description     = Vertica 7 ODBC driver
Driver          = /opt/vertica/lib/libverticaodbc.so
Driver64        = /opt/vertica/lib64/libverticaodbc.so
FileUsage       = 1

[ODBC]
Threading       = 1
Trace           = 1
TraceFile       = /tmp/odbctrace.log
Debug           = 1
DebugFile       = /tmp/odbcdebug.log

*** /etc/odbc.ini

[ODBC Data Sources]
vertica_kdl_dsn = kdl database on HP Vertica 7

[vertica_kdl_dsn]
Driver = vertica_driver
Description = ODBC Driver DSN for kdl master database
Servername = compute-0-0
Port = 5433
Database = kdl
Locale = en_US

*** /etc/vertica.ini

[Driver]
DriverManagerEncoding = UTF-16
ODBCInstLib = /usr/lib64/libodbcinst.so
ErrorMessagesPath = /opt/vertica/lib64
LogLevel = 4
LogPath = /tmp

*** odbc trace files: see attached (odbc-trace-2014-01-17)

/tmp/odbctrace.log
/tmp/vertica_odbc_conn_1.log
/tmp/vertica_driver.log

matthewcornell · January 2014

Here are the three trace files I was able to find:

* /tmp/odbctrace.log

[ODBC][18732][1389975908.696629][__handles.c][450]
        Exit:[SQL_SUCCESS]
            Environment = 0x26ba190
[ODBC][18732][1389975908.696678][SQLSetEnvAttr.c][182]
        Entry:
            Environment = 0x26ba190
            Attribute = SQL_ATTR_ODBC_VERSION
            Value = 0x3
            StrLen = 4
[ODBC][18732][1389975908.696704][SQLSetEnvAttr.c][349]
        Exit:[SQL_SUCCESS]
[ODBC][18732][1389975908.696730][SQLAllocHandle.c][364]
        Entry:
            Handle Type = 2
            Input Handle = 0x26ba190
[ODBC][18732][1389975908.696753][SQLAllocHandle.c][482]
        Exit:[SQL_SUCCESS]
            Output Handle = 0x2671220
[ODBC][18732][1389975908.696789][SQLDriverConnectW.c][286]
        Entry:
            Connection = 0x2671220
            Window Hdl = (nil)
            Str In = [DSN=vertica_kdl_dsn;xx=test;Password=xx][length = 51]
            Str Out = (nil)
            Str Out Max = 0
            Str Out Ptr = (nil)
            Completion = 0
        UNICODE Using encoding ASCII 'ISO8859-1' and UNICODE 'UCS-2LE'

[ODBC][18732][1389975908.727246][SQLDriverConnectW.c][842]
        Exit:[SQL_SUCCESS_WITH_INFO]
            Connection Out [[NULL]]
[ODBC][18732][1389975908.727304][SQLSetConnectAttr.c][321]
        Entry:
            Connection = 0x2671220
            Attribute = SQL_ATTR_AUTOCOMMIT
            Value = (nil)
            StrLen = -5
[ODBC][18732][1389975908.727626][SQLSetConnectAttr.c][675]
        Exit:[SQL_SUCCESS]
[ODBC][18732][1389975908.727700][SQLGetInfo.c][546]
        Entry:
            Connection = 0x2671220
            Info Type = SQL_DRIVER_ODBC_VER (77)
            Info Value = 0x7fffb87dc190
            Buffer Length = 20
            StrLen = 0x7fffb87dc1de
[ODBC][18732][1389975908.727745][SQLGetInfo.c][608]
        Exit:[SQL_SUCCESS]
[ODBC][18732][1389975908.727770][SQLGetInfo.c][546]
        Entry:
            Connection = 0x2671220
            Info Type = SQL_DESCRIBE_PARAMETER (10002)
            Info Value = 0x7fffb87dc1c0
            Buffer Length = 2
            StrLen = 0x7fffb87dc1de
[ODBC][18732][1389975908.727809][SQLGetInfo.c][608]
        Exit:[SQL_SUCCESS]
[ODBC][18732][1389975908.727832][SQLGetInfo.c][546]
        Entry:
            Connection = 0x2671220
            Info Type = SQL_NEED_LONG_DATA_LEN (111)
            Info Value = 0x7fffb87dc1c0
            Buffer Length = 2
            StrLen = 0x7fffb87dc1de
[ODBC][18732][1389975908.727870][SQLGetInfo.c][608]
        Exit:[SQL_SUCCESS]
[ODBC][18732][1389975908.727892][SQLAllocHandle.c][529]
        Entry:
            Handle Type = 3
            Input Handle = 0x2671220
[ODBC][18732][1389975908.728118][SQLAllocHandle.c][1064]
        Exit:[SQL_SUCCESS]
            Output Handle = 0x28779d0
[ODBC][18732][1389975908.728148][SQLGetTypeInfo.c][164]
        Entry:
            Statement = 0x28779d0
            Data Type = SQL_TYPE_TIMESTAMP
[ODBC][18732][1389975908.747035][SQLGetTypeInfo.c][314]
        Exit:[SQL_SUCCESS]
[ODBC][18732][1389975908.747079][SQLFetch.c][158]
        Entry:
            Statement = 0x28779d0
[ODBC][18732][1389975908.747124][SQLFetch.c][340]
        Exit:[SQL_SUCCESS]
[ODBC][18732][1389975908.747151][SQLGetData.c][233]
        Entry:
            Statement = 0x28779d0
            Column Number = 3
            Target Type = 4 SQL_INTEGER
            Buffer Length = 4
            Target Value = 0x7fffb87dc1d8
            StrLen Or Ind = (nil)
[ODBC][18732][1389975908.747205][SQLGetData.c][497]
        Exit:[SQL_SUCCESS]
            Buffer = [26]
            Strlen Or Ind = NULLPTR
[ODBC][18732][1389975908.747227][SQLGetTypeInfo.c][164]
        Entry:
            Statement = 0x28779d0
            Data Type = SQL_VARCHAR
[ODBC][18732][1389975908.747246][SQLGetTypeInfo.c][186]Error: 24000
[ODBC][18732][1389975908.747280][SQLGetTypeInfo.c][164]
        Entry:
            Statement = 0x28779d0
            Data Type = Unknown(-9)
[ODBC][18732][1389975908.747300][SQLGetTypeInfo.c][186]Error: 24000
[ODBC][18732][1389975908.747327][SQLGetTypeInfo.c][164]
        Entry:
            Statement = 0x28779d0
            Data Type = SQL_BINARY
[ODBC][18732][1389975908.747346][SQLGetTypeInfo.c][186]Error: 24000
[ODBC][18732][1389975908.747384][SQLFreeStmt.c][140]
        Entry:
            Statement = 0x28779d0
            Option = 0
[ODBC][18732][1389975908.747447][SQLFreeStmt.c][246]
        Exit:[SQL_SUCCESS]
[ODBC][18732][1389975908.747884][SQLAllocHandle.c][529]
        Entry:
            Handle Type = 3
            Input Handle = 0x2671220
[ODBC][18732][1389975908.748075][SQLAllocHandle.c][1064]
        Exit:[SQL_SUCCESS]
            Output Handle = 0x2879850
[ODBC][18732][1389975913.528709][SQLFreeStmt.c][140]
        Entry:
            Statement = 0x2879850
            Option = 0
[ODBC][18732][1389975913.528780][SQLFreeStmt.c][246]
        Exit:[SQL_SUCCESS]
[ODBC][18732][1389975913.528816][SQLExecDirectW.c][173]
        Entry:
            Statement = 0x2879850
            SQL = [DROP TABLE IF EXISTS foo][length = 24 (SQL_NTS)]
[ODBC][18732][1389975913.537378][SQLExecDirectW.c][432]
        Exit:[SQL_SUCCESS_WITH_INFO]
[ODBC][18732][1389975913.537422][SQLRowCount.c][169]
        Entry:
            Statement = 0x2879850
            Row Count = 0x7fffb87dc318
[ODBC][18732][1389975913.537453][SQLRowCount.c][240]
        Exit:[SQL_SUCCESS]
            Row Count = 0x7fffb87dc318 -> -1
[ODBC][18732][1389975913.537479][SQLNumResultCols.c][152]
        Entry:
            Statement = 0x2879850
            Column Count = 0x7fffb87dc32e

* /tmp/vertica_odbc_conn_1.log

Jan 17 11:25:08 INFO 3457042176 Connection::SQLSetConnectAttr: Attribute: 115
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: in: Database:kdl, type 0
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: in: Description:ODBC Driver DSN for kdl master database, type 0
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: in: Driver:vertica_driver, type 0
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: in: DSN:vertica_kdl_dsn, type 0
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: in: Locale:en_US, type 0
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: in: PASSWORD:********, type 0
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: in: Port:5433, type 0
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: in: Servername:compute-0-0, type 0
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: in: USERNAME:test, type 0
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: out ColumnsAsChar
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: type 0, value: ?
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: out ConnSettings
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: type 0, value: ?
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: out DirectBatchInsert
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: type 0, value: ?
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: out DriverStringConversions
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: type 0, value: ?
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: out Label
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: type 0, value: ?
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: out ResultBufferSize
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: type 0, value: ?
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: out SSLCertFile
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: type 0, value: ?
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: out SSLKeyFile
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: type 0, value: ?
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: out SSLMode
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: type 0, value: ?
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: out TransactionIsolation
Jan 17 11:25:08 INFO 3457042176 VConnection::UpdateConnectionSettings: type 0, value: ?
Jan 17 11:25:08 INFO 3457042176 VConnection::Connect: Connect to host compute-0-0:5433, db kdl
Jan 17 11:25:08 INFO 3457042176 Connection::SQLGetInfoW: InfoType: 77
Jan 17 11:25:08 INFO 3457042176 Connection::SQLSetConnectAttr: Attribute: 102
Jan 17 11:25:08 INFO 3457042176 Connection::SQLGetInfoW: InfoType: 77
Jan 17 11:25:08 INFO 3457042176 Connection::SQLGetInfoW: InfoType: 10002
Jan 17 11:25:08 INFO 3457042176 Connection::SQLGetInfoW: InfoType: 111
Jan 17 11:25:08 INFO 3457042176 Statement::SQLGetStmtAttrW: Attribute: 10010
Jan 17 11:25:08 INFO 3457042176 Statement::SQLGetStmtAttrW: Attribute: 10011
Jan 17 11:25:08 INFO 3457042176 Statement::SQLGetStmtAttrW: Attribute: 10012
Jan 17 11:25:08 INFO 3457042176 Statement::SQLGetStmtAttrW: Attribute: 10013
Jan 17 11:25:08 INFO 3457042176 Connection::ExecuteCatalogFunction: SQLGetTypeInfo
Jan 17 11:25:08 INFO 3457042176 VMetadataSource::Execute: out_metadataQuery: select * from (select CASE WHEN (UPPER(type_name) = 'CHAR') THEN 'CHAR' WHEN (UPPER(type_name) = 'VARCHAR') THEN 'VARCHAR' ELSE type_name END as data_type_name, CASE WHEN (odbc_type = 1) THEN -8 WHEN (odbc_type = 12) THEN -9 WHEN ( odbc_subtype != 0 ) THEN odbc_subtype ELSE odbc_type END as data_type, column_size, case when type_id not in (5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 114, 115, 116, 117) then null when odbc_type in (-5, 2, 8) then null else E'\'' end as literal_prefix, case when type_id not in (5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 114, 115, 116, 117) then null when odbc_type in (-5, 2, 8) then null else E'\'' end as literal_suffix, creation_parameters as create_param, 1 as nullable, 1 as case_sensitive, 3 as searchable, 0 as unsigned_attribute, 0 as fixed_prec_scale, 0 as auto_unique, CASE WHEN (UPPER(type_name) = 'CHAR') THEN 'CHAR' WHEN (UPPER(type_name) = 'VARCHAR') THEN 'VARCHAR' ELSE type_name END as local_type_name, min_scale as minimum_scale, max_scale as maximum_scale, CASE WHEN (odbc_type = 1) THEN -8 WHEN (odbc_type = 12) THEN -9 ELSE odbc_type END as sql_data_type, CASE WHEN (odbc_type = 10) THEN (odbc_subtype-100) WHEN (odbc_type = 9) THEN (odbc_subtype-90) ELSE 0 END as sql_datetime_sub, 10 as num_prec_radix, NULL as interval_precision from v_catalog.types order by odbc_type, type_name ) as vmd where data_type = E'93'
Jan 17 11:25:08 INFO 3457042176 VMetadataSource::Execute: Command status: SELECT, 2 tuples 19 fields
Jan 17 11:25:08 INFO 3457042176 Statement::SQLGetStmtAttrW: Attribute: 10010
Jan 17 11:25:08 INFO 3457042176 Statement::SQLGetStmtAttrW: Attribute: 10011
Jan 17 11:25:08 INFO 3457042176 Statement::SQLGetStmtAttrW: Attribute: 10012
Jan 17 11:25:08 INFO 3457042176 Statement::SQLGetStmtAttrW: Attribute: 10013
Jan 17 11:25:13 INFO 3457042176 VDataEngine::Prepare: Original Query: DROP TABLE IF EXISTS foo
Jan 17 11:25:13 INFO 3457042176 VDataEngine::Prepare: Query is issued as ExecDirect
Jan 17 11:25:13 INFO 3457042176 VDataEngine::Prepare: Query: DROP TABLE IF EXISTS foo
Jan 17 11:25:13 INFO 3457042176 VDataEngine::Prepare: Initializing prepared statement: _PLAN0xebe360_0

* /tmp/vertica_driver.log

Jan 17 11:25:08 INFO 3457042176 Environment::SQLSetEnvAttr: Attribute: 200
Jan 17 11:25:08 INFO 3457042176 Environment::SQLGetEnvAttr: Attribute: 200
Jan 17 11:25:08 INFO 3457042176 VEnvironment::CreateConnection: ODBC Version is: 3

matthewcornell · January 2014

Here's some connection information, if that helps:

 >>> import rpy2.robjects >>> import pyodbc >>> conn = pyodbc.connect('DSN=vertica_kdl_dsn;UserName=xx;Password=xx') >>> for attr in vars(pyodbc): ... try: ... value = conn.getinfo(getattr(pyodbc, attr)) ... print('{:<40s} | {}'.format(attr, value)) ... except: ... pass ... SQL_SQL_CONFORMANCE | 1 SQL_TXN_ISOLATION_OPTION | 10 SQL_PROCEDURE_TERM | function SQL_DROP_TRANSLATION | 0 SQL_CATALOG_USAGE | 0 SQL_SCHEMA_USAGE | 21 SQL_NULLABLE | 0 SQL_STATIC_CURSOR_ATTRIBUTES2 | 0 SQL_STATIC_CURSOR_ATTRIBUTES1 | 0 SQL_MAX_COLUMNS_IN_TABLE | 1600 SQL_GETDATA_EXTENSIONS | 15 SQL_INTERVAL_MONTH | 0 SQL_LIKE_ESCAPE_CLAUSE | True SQL_TIMEDATE_DIFF_INTERVALS | 511 SQL_ROW_UPDATES | False SQL_MAX_ROW_SIZE | 32000000 SQL_INTERVAL_YEAR | 1600 SQL_XOPEN_CLI_YEAR | 1995 SQL_INTERVAL_SECOND | 0 SQL_INTERVAL_MINUTE | 0 SQL_ACTIVE_ENVIRONMENTS | 0 SQL_MAX_INDEX_SIZE | 0 SQL_TIMEDATE_FUNCTIONS | 1998847 SQL_ALTER_TABLE | 62312 SQL_CREATE_TABLE | 1045 SQL_INTERVAL_DAY_TO_MINUTE | 511 SQL_SCOPE_SESSION | vertica_kdl_dsn SQL_SQL92_REVOKE | 13744 SQL_CREATE_TRANSLATION | 0 SQL_MAX_BINARY_LITERAL_LEN | 0 SQL_PARAM_INPUT | 0 SQL_PARAM_ARRAY_ROW_COUNTS | 1 SQL_COLUMN_ALIAS | True SQL_SQL92_ROW_VALUE_CONSTRUCTOR | 3 SQL_SCROLL_OPTIONS | 1 SQL_SCOPE_TRANSACTION | 0 lowercase | 0 SQL_BATCH_ROW_COUNT | 2 SQL_DYNAMIC_CURSOR_ATTRIBUTES2 | 0 SQL_DYNAMIC_CURSOR_ATTRIBUTES1 | 0 SQL_CATALOG_NAME | False SQL_SUBQUERIES | 31 SQL_SCHEMA_TERM | schema SQL_UNION | 3 SQL_NEED_LONG_DATA_LEN | False SQL_DATA_SOURCE_READ_ONLY | False SQL_DDL_INDEX | 0 SQL_NON_NULLABLE_COLUMNS | 1 SQL_MAX_IDENTIFIER_LEN | 128 SQL_QUOTED_IDENTIFIER_CASE | 4 SQL_SERVER_NAME | compute-0-0 SQL_UNKNOWN_TYPE | 0 SQL_NUMERIC | vertica_kdl_dsn SQLWCHAR_SIZE | vertica_kdl_dsn SQL_CORRELATION_NAME | 2 SQL_DESCRIBE_PARAMETER | False SQL_INTERVAL_HOUR_TO_MINUTE | False SQL_EXPRESSIONS_IN_ORDERBY | True SQL_OJ_CAPABILITIES | 127 SQL_TABLE_TERM | table SQL_SQL92_FOREIGN_KEY_DELETE_RULE | 0 SQL_CATALOG_LOCATION | 1 SQL_BOOKMARK_PERSISTENCE | 0 SQL_MAX_COLUMNS_IN_GROUP_BY | 0 SQL_CREATE_DOMAIN | 0 SQL_INTEGRITY | False SQL_DRIVER_NAME | verticaodbcw.so SQL_TYPE_TIME | 0 SQL_DEFAULT_TXN_ISOLATION | 2 SQL_INFO_SCHEMA_VIEWS | 0 SQL_TYPE_TIMESTAMP | 4 SQL_SCOPE_CURROW | 0 SQL_SQL92_NUMERIC_VALUE_FUNCTIONS | 46 SQL_ORDER_BY_COLUMNS_IN_SELECT | True SQL_TYPE_DATE | 21 SQL_CREATE_ASSERTION | 0 SQL_INTERVAL_MINUTE_TO_SECOND | True SQL_NULLABLE_UNKNOWN | vertica_kdl_dsn SQL_MULTIPLE_ACTIVE_TXN | True SQL_STRING_FUNCTIONS | 9732063 SQL_CONCAT_NULL_BEHAVIOR | 0 SQL_MAX_CATALOG_NAME_LEN | 0 SQL_MAX_SCHEMA_NAME_LEN | 128 SQL_MAX_CONCURRENT_ACTIVITIES | 0 SQL_MAX_CURSOR_NAME_LEN | 0 SQL_IDENTIFIER_CASE | 4 SQL_CURSOR_ROLLBACK_BEHAVIOR | 2 SQL_DBMS_VER | 07.00.0000 SQL_DROP_VIEW | 1 SQL_CHAR | 0 SQL_SEARCH_PATTERN_ESCAPE | \ SQL_REAL | 07.00.0000 SQL_GROUP_BY | 2 SQL_DATABASE_NAME | SQL_SQL92_GRANT | 3440 SQL_DROP_TABLE | 7 SQL_ACCESSIBLE_PROCEDURES | False SQL_BATCH_SUPPORT | 3 SQL_MAX_TABLES_IN_SELECT | 0 SQL_ODBC_VER | 03.52 SQL_COLLATION_SEQ | SQL_MAX_STATEMENT_LEN | 0 SQL_DRIVER_VER | 07.00.0000 SQL_DBMS_NAME | Vertica Database SQL_PARAM_INPUT_OUTPUT | vertica_kdl_dsn SQL_INTERVAL_HOUR_TO_SECOND | 0 SQL_PARAM_TYPE_UNKNOWN | 0 SQL_PC_UNKNOWN | 0 SQL_DM_VER | 03.52.0002.0002 SQL_CONVERT_VARCHAR | 12317183 SQL_DRIVER_ODBC_VER | 03.52 SQL_PC_NOT_PSEUDO | 0 SQL_INTERVAL_YEAR_TO_MONTH | 128 SQL_PC_PSEUDO | vertica_kdl_dsn SQL_INTERVAL_DAY | False SQL_FORWARD_ONLY_CURSOR_ATTRIBUTES2 | 4097 SQL_FORWARD_ONLY_CURSOR_ATTRIBUTES1 | 1 SQL_KEYSET_CURSOR_ATTRIBUTES2 | 4097 SQL_SQL92_STRING_FUNCTIONS | 255 SQL_CATALOG_TERM | SQL_INDEX_KEYWORDS | 0 SQL_CREATE_COLLATION | 0 SQL_AGGREGATE_FUNCTIONS | 127 SQL_FLOAT | verticaodbcw.so SQL_IDENTIFIER_QUOTE_CHAR | " SQL_MAX_TABLE_NAME_LEN | 128 SQL_MAX_USER_NAME_LEN | 128 SQL_SYSTEM_FUNCTIONS | 7 SQL_MAX_DRIVER_CONNECTIONS | 0 SQL_SQL92_PREDICATES | 16007 SQL_ODBC_INTERFACE_CONFORMANCE | 1 SQL_SQL92_FOREIGN_KEY_UPDATE_RULE | 0 SQL_MAX_ASYNC_CONCURRENT_STATEMENTS | 0 SQL_NUMERIC_FUNCTIONS | 16121855 SQL_INSERT_STATEMENT | 7 SQL_KEYSET_CURSOR_ATTRIBUTES1 | 4687 SQL_CREATE_SCHEMA | 3 SQL_SQL92_DATETIME_FUNCTIONS | 7 SQL_DROP_SCHEMA | 7 SQL_FILE_USAGE | 0 SQL_MAX_COLUMN_NAME_LEN | 128 SQL_KEYWORDS | SQL_PROCEDURES | False SQL_CREATE_CHARACTER_SET | 0 SQL_DROP_DOMAIN | 0 SQL_NULL_COLLATION | 1 SQL_STANDARD_CLI_CONFORMANCE | 2 SQL_TXN_CAPABLE | 3 SQL_SQL92_VALUE_EXPRESSIONS | 15 SQL_MAX_COLUMNS_IN_SELECT | 0 pooling | 0 SQL_MAX_CHAR_LITERAL_LEN | 0 SQL_ALTER_DOMAIN | 0 SQL_DROP_COLLATION | 0 SQL_PARAM_ARRAY_SELECTS | 3 SQL_CURSOR_COMMIT_BEHAVIOR | 2 SQL_SQL92_RELATIONAL_JOIN_OPERATORS | 474 SQL_MAX_ROW_SIZE_INCLUDES_LONG | False SQL_MAX_COLUMNS_IN_ORDER_BY | 0 SQL_INTERVAL_HOUR | 32000000 SQL_MAX_PROCEDURE_NAME_LEN | 128 SQL_MAX_COLUMNS_IN_INDEX | 0 SQL_CONVERT_FUNCTIONS | 3 SQL_CATALOG_NAME_SEPARATOR | . SQL_ACCESSIBLE_TABLES | False SQL_INTERVAL_DAY_TO_HOUR | 0 threadsafety | 0 SQL_SPECIAL_CHARACTERS | SQL_CREATE_VIEW | 1 SQL_INTERVAL_DAY_TO_SECOND | 511 SQL_DROP_CHARACTER_SET | 0 SQL_USER_NAME | test SQL_TIMEDATE_ADD_INTERVALS | 511 SQL_NO_NULLS | 0 SQL_DATETIME_LITERALS | 65535 SQL_DATA_SOURCE_NAME | vertica_kdl_dsn SQL_DROP_ASSERTION | 0 SQL_MULT_RESULT_SETS | True SQL_ASYNC_MODE | 0 >>>

matthewcornell · January 2014

Would someone from Vertica would please give me some advice about how to proceed? I've done a lot of sleuthing to get to this point, and I've tried to include helpful information. I realize there are three major parties involved here (your odbc driver, pyodbc, and, somehow, rpy2), but we need this stack to work so we can use your great software. Thank you.

[Deleted User] · January 2014

Just to quickly clarify -- is the segfault in Vertica or is it in Python? (Which process is segfaulting?)

I've been assuming that it was in Python, but realized that I should confirm.

matthewcornell · January 2014

Hi Adam. IIUC, it's on the python/client side. The server stays up and I'm able to access it from vsql just fine. I've tried the pyodbc module with the postgres odbc driver, and it's fine. I've also tried the vertica driver from isql, which also does not segfault. And the pyodbc module works fine with vertica if I do not import the rpy2.objects module. Figures.

[Deleted User] · January 2014

Hm... I'm afraid I've got nothin'. This is one of the stranger bugs I've seen recently, for sure.

I think the right way to approach this problem would be to install Python, pyodbc, and rpy2 all with debug symbols; then break out gdb and get a proper C/C++-level stack trace of exactly what's going on at the point where things crash. Both the raw stack itself and, if possible, the values of any local variables in order to cause a crash at that location.

rpy2 is, if I had to guess, most likely manipulating the global environment in some unusual way that we don't expect. But I've no idea what's actually going on, so unfortunately can't comment in too much detail...

One quick thing to check first -- does your system have any unusual ulimits set? "ulimit -a" will list them. For example, if you have a very low limit on open file handles, or process memory, etc., it's possible that pyodbc uses some of whatever resource, our driver uses some, rpy2 uses some, and then the process runs out and crashes. Very unusual -- but then again, this bug seems very unusual, so I figure I might as well ask.

matthewcornell · February 2014

Thank you for the suggested next steps, Adam. I'll start the up gdb learning curve.

ulimit -a on the client:

[cornell@wright ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 515020
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[cornell@wright ~]$

The values are the same on the compute node that I'm connecting to, except:

pending signals                 (-i) 515020
max user processes              (-u) 16071

Do any of the above look off to you?

[Deleted User] · February 2014

Neither of those ulimit values look problematic to me.

If you're not familiar with gdb, you might want to first consider cross-posting this question on the pyodbc or rpy2 lists. Someone on one of those two lists might have some ideas of how to narrow the search.

It may also be that someone else here has some ideas that I don't. If so, please feel free to post!

matthewcornell · February 2014

Thanks, Adam.

python3 + rpy + pyodb + vertica 7 = Linux segfault

Comments

Leave a Comment