python3 + rpy + pyodb + vertica 7 = Linux segfault

matthewcornellmatthewcornell
in General Discussion
Hi Folks,

I've discovered a python-vertica stack configuration that causes a segmentation fault on our Linux cluster. I'd really appreciate your help in deciding how to proceed with a solution. I've attached as much information as I can to help figure this out.

The summary: When running in a virtualenv, if I import rpy2.objects then pyodb and then try to execute certain queries, I get the segfault. Interestingly, if I swap the import order, it seems to work OK (!) Testing on previous versions of pyodb have the segfault happening at connect() instead of execute().

Thanks very much,

matt


** commands to install the stack via virtualenv

/share/apps/python3/bin/python3.3 ~/bin/virtualenv.py --no-site-packages ~/virtualenvs/vertica-test
source ~/virtualenvs/vertica-test/bin/activate

pip install http://pyodbc.googlecode.com/files/pyodbc-3.0.7.zip
-> Successfully installed pyodbc

pip install rpy2
-> Successfully installed rpy2


** commands in interpreter to reproduce the segfault

# NB: bad username or password -> segfault at connect()
# NB: if the import order is switched -> NO segfault

[cornell@wright ~]$ source ~/virtualenvs/vertica-test/bin/activate

(vertica-test)[cornell@wright ~]$ python3.3
Python 3.3.2 (default, May 16 2013, 11:31:48)
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import rpy2.robjects
>>> import pyodbc
>>> conn = pyodbc.connect('DSN=vertica_kdl_dsn;UID=xx;PWD=xx')
>>> curs = conn.cursor()
>>> curs.execute('DROP TABLE IF EXISTS foo')
Segmentation fault


(vertica-test)[cornell@wright ~]$ python3.3
Python 3.3.2 (default, May 16 2013, 11:31:48)
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyodbc
>>> import rpy2.robjects
>>> conn = pyodbc.connect('DSN=vertica_kdl_dsn;UID=xx;PWD=xx')
>>> curs = conn.cursor()
>>> curs.execute('DROP TABLE IF EXISTS foo')
<pyodbc.Cursor object at 0x7f8508b5fe10>
>>>


** versions

Python 3.3.2 (default, May 16 2013, 11:31:48)
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux


>>> rpy2.__version__
'2.3.8'


>>> pyodbc.version
'3.0.7'
>>> pyodbc.SQL_DBMS_VER
18
>>> pyodbc.SQL_DM_VER
171
>>> pyodbc.SQL_DRIVER_ODBC_VER
77
>>> pyodbc.SQL_DRIVER_VER
7
>>> pyodbc.SQL_ODBC_VER
10


$ isql --version
unixODBC 2.2.14


/opt/vertica/lib64:
  lrwxrwxrwx  1 root root       23 Dec 18 09:43 libverticaodbc.so.7.0 -> libverticaodbc.so.7.0.0

[dbadmin@compute-0-0 ~]$ /opt/vertica/bin/adminTools
Vertica Analytic Database 7.0.0-0 Administration Tools


[cornell@wright ~]$ uname -a
Linux wright.cs.umass.edu 2.6.32-279.14.1.el6.x86_64 #1 SMP Tue Nov 6 23:43:09 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
[cornell@wright ~]$ cat /proc/version
Linux version 2.6.32-279.14.1.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Tue Nov 6 23:43:09 UTC 2012
[cornell@wright ~]$


** odbc configuration files

*** /etc/odbcinst.ini

[PostgreSQL]
...

[MySQL]
...

[vertica_driver]
Description     = Vertica 7 ODBC driver
Driver          = /opt/vertica/lib/libverticaodbc.so
Driver64        = /opt/vertica/lib64/libverticaodbc.so
FileUsage       = 1

[ODBC]
Threading       = 1
Trace           = 1
TraceFile       = /tmp/odbctrace.log
Debug           = 1
DebugFile       = /tmp/odbcdebug.log


*** /etc/odbc.ini

[ODBC Data Sources]
vertica_kdl_dsn = kdl database on HP Vertica 7


[vertica_kdl_dsn]
Driver = vertica_driver
Description = ODBC Driver DSN for kdl master database
Servername = compute-0-0
Port = 5433
Database = kdl
Locale = en_US


*** /etc/vertica.ini

[Driver]
DriverManagerEncoding = UTF-16
ODBCInstLib = /usr/lib64/libodbcinst.so
ErrorMessagesPath = /opt/vertica/lib64
LogLevel = 4
LogPath = /tmp


*** odbc trace files: see attached (odbc-trace-2014-01-17)

/tmp/odbctrace.log
/tmp/vertica_odbc_conn_1.log
/tmp/vertica_driver.log



Comments

  • matthewcornellmatthewcornell
    Here are the three trace files I was able to find:

    * /tmp/odbctrace.log

    [ODBC][18732][1389975908.696629][__handles.c][450]
            Exit:[SQL_SUCCESS]
                Environment = 0x26ba190
    [ODBC][18732][1389975908.696678][SQLSetEnvAttr.c][182]
            Entry:           
                Environment = 0x26ba190           
                Attribute = SQL_ATTR_ODBC_VERSION           
                Value = 0x3           
                StrLen = 4
    [ODBC][18732][1389975908.696704][SQLSetEnvAttr.c][349]
            Exit:[SQL_SUCCESS]
    [ODBC][18732][1389975908.696730][SQLAllocHandle.c][364]
            Entry:
                Handle Type = 2
                Input Handle = 0x26ba190
    [ODBC][18732][1389975908.696753][SQLAllocHandle.c][482]
            Exit:[SQL_SUCCESS]
                Output Handle = 0x2671220
    [ODBC][18732][1389975908.696789][SQLDriverConnectW.c][286]
            Entry:           
                Connection = 0x2671220           
                Window Hdl = (nil)           
                Str In = [DSN=vertica_kdl_dsn;xx=test;Password=xx][length = 51]           
                Str Out = (nil)           
                Str Out Max = 0           
                Str Out Ptr = (nil)           
                Completion = 0
            UNICODE Using encoding ASCII 'ISO8859-1' and UNICODE 'UCS-2LE'

    [ODBC][18732][1389975908.727246][SQLDriverConnectW.c][842]
            Exit:[SQL_SUCCESS_WITH_INFO]                   
                Connection Out [[NULL]]
    [ODBC][18732][1389975908.727304][SQLSetConnectAttr.c][321]
            Entry:           
                Connection = 0x2671220           
                Attribute = SQL_ATTR_AUTOCOMMIT           
                Value = (nil)           
                StrLen = -5
    [ODBC][18732][1389975908.727626][SQLSetConnectAttr.c][675]
            Exit:[SQL_SUCCESS]
    [ODBC][18732][1389975908.727700][SQLGetInfo.c][546]
            Entry:           
                Connection = 0x2671220           
                Info Type = SQL_DRIVER_ODBC_VER (77)           
                Info Value = 0x7fffb87dc190           
                Buffer Length = 20           
                StrLen = 0x7fffb87dc1de
    [ODBC][18732][1389975908.727745][SQLGetInfo.c][608]
            Exit:[SQL_SUCCESS]
    [ODBC][18732][1389975908.727770][SQLGetInfo.c][546]
            Entry:           
                Connection = 0x2671220           
                Info Type = SQL_DESCRIBE_PARAMETER (10002)           
                Info Value = 0x7fffb87dc1c0           
                Buffer Length = 2           
                StrLen = 0x7fffb87dc1de
    [ODBC][18732][1389975908.727809][SQLGetInfo.c][608]
            Exit:[SQL_SUCCESS]
    [ODBC][18732][1389975908.727832][SQLGetInfo.c][546]
            Entry:           
                Connection = 0x2671220           
                Info Type = SQL_NEED_LONG_DATA_LEN (111)           
                Info Value = 0x7fffb87dc1c0           
                Buffer Length = 2           
                StrLen = 0x7fffb87dc1de
    [ODBC][18732][1389975908.727870][SQLGetInfo.c][608]
            Exit:[SQL_SUCCESS]
    [ODBC][18732][1389975908.727892][SQLAllocHandle.c][529]
            Entry:
                Handle Type = 3
                Input Handle = 0x2671220
    [ODBC][18732][1389975908.728118][SQLAllocHandle.c][1064]
            Exit:[SQL_SUCCESS]
                Output Handle = 0x28779d0
    [ODBC][18732][1389975908.728148][SQLGetTypeInfo.c][164]
            Entry:           
                Statement = 0x28779d0           
                Data Type = SQL_TYPE_TIMESTAMP
    [ODBC][18732][1389975908.747035][SQLGetTypeInfo.c][314]
            Exit:[SQL_SUCCESS]
    [ODBC][18732][1389975908.747079][SQLFetch.c][158]
            Entry:           
                Statement = 0x28779d0
    [ODBC][18732][1389975908.747124][SQLFetch.c][340]
            Exit:[SQL_SUCCESS]
    [ODBC][18732][1389975908.747151][SQLGetData.c][233]
            Entry:           
                Statement = 0x28779d0           
                Column Number = 3           
                Target Type = 4 SQL_INTEGER           
                Buffer Length = 4           
                Target Value = 0x7fffb87dc1d8           
                StrLen Or Ind = (nil)
    [ODBC][18732][1389975908.747205][SQLGetData.c][497]
            Exit:[SQL_SUCCESS]               
                Buffer = [26]               
                Strlen Or Ind = NULLPTR
    [ODBC][18732][1389975908.747227][SQLGetTypeInfo.c][164]
            Entry:           
                Statement = 0x28779d0           
                Data Type = SQL_VARCHAR
    [ODBC][18732][1389975908.747246][SQLGetTypeInfo.c][186]Error: 24000
    [ODBC][18732][1389975908.747280][SQLGetTypeInfo.c][164]
            Entry:           
                Statement = 0x28779d0           
                Data Type = Unknown(-9)
    [ODBC][18732][1389975908.747300][SQLGetTypeInfo.c][186]Error: 24000
    [ODBC][18732][1389975908.747327][SQLGetTypeInfo.c][164]
            Entry:           
                Statement = 0x28779d0           
                Data Type = SQL_BINARY
    [ODBC][18732][1389975908.747346][SQLGetTypeInfo.c][186]Error: 24000
    [ODBC][18732][1389975908.747384][SQLFreeStmt.c][140]
            Entry:           
                Statement = 0x28779d0           
                Option = 0
    [ODBC][18732][1389975908.747447][SQLFreeStmt.c][246]
            Exit:[SQL_SUCCESS]
    [ODBC][18732][1389975908.747884][SQLAllocHandle.c][529]
            Entry:
                Handle Type = 3
                Input Handle = 0x2671220
    [ODBC][18732][1389975908.748075][SQLAllocHandle.c][1064]
            Exit:[SQL_SUCCESS]
                Output Handle = 0x2879850
    [ODBC][18732][1389975913.528709][SQLFreeStmt.c][140]
            Entry:           
                Statement = 0x2879850           
                Option = 0
    [ODBC][18732][1389975913.528780][SQLFreeStmt.c][246]
            Exit:[SQL_SUCCESS]
    [ODBC][18732][1389975913.528816][SQLExecDirectW.c][173]
            Entry:           
                Statement = 0x2879850           
                SQL = [DROP TABLE IF EXISTS foo][length = 24 (SQL_NTS)]
    [ODBC][18732][1389975913.537378][SQLExecDirectW.c][432]
            Exit:[SQL_SUCCESS_WITH_INFO]
    [ODBC][18732][1389975913.537422][SQLRowCount.c][169]
            Entry:           
                Statement = 0x2879850           
                Row Count = 0x7fffb87dc318
    [ODBC][18732][1389975913.537453][SQLRowCount.c][240]
            Exit:[SQL_SUCCESS]               
                Row Count = 0x7fffb87dc318 -> -1
    [ODBC][18732][1389975913.537479][SQLNumResultCols.c][152]
            Entry:           
                Statement = 0x2879850           
                Column Count = 0x7fffb87dc32e


    * /tmp/vertica_odbc_conn_1.log

    Jan 17 11:25:08 INFO  3457042176 Connection::SQLSetConnectAttr: Attribute: 115
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: in: Database:kdl, type 0
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: in: Description:ODBC Driver DSN for kdl master database, type 0
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: in: Driver:vertica_driver, type 0
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: in: DSN:vertica_kdl_dsn, type 0
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: in: Locale:en_US, type 0
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: in: PASSWORD:********, type 0
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: in: Port:5433, type 0
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: in: Servername:compute-0-0, type 0
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: in: USERNAME:test, type 0
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: out ColumnsAsChar
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: type 0, value: ?
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: out ConnSettings
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: type 0, value: ?
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: out DirectBatchInsert
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: type 0, value: ?
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: out DriverStringConversions
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: type 0, value: ?
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: out Label
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: type 0, value: ?
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: out ResultBufferSize
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: type 0, value: ?
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: out SSLCertFile
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: type 0, value: ?
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: out SSLKeyFile
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: type 0, value: ?
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: out SSLMode
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: type 0, value: ?
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: out TransactionIsolation
    Jan 17 11:25:08 INFO  3457042176 VConnection::UpdateConnectionSettings: type 0, value: ?
    Jan 17 11:25:08 INFO  3457042176 VConnection::Connect: Connect to host compute-0-0:5433, db kdl
    Jan 17 11:25:08 INFO  3457042176 Connection::SQLGetInfoW: InfoType: 77
    Jan 17 11:25:08 INFO  3457042176 Connection::SQLSetConnectAttr: Attribute: 102
    Jan 17 11:25:08 INFO  3457042176 Connection::SQLGetInfoW: InfoType: 77
    Jan 17 11:25:08 INFO  3457042176 Connection::SQLGetInfoW: InfoType: 10002
    Jan 17 11:25:08 INFO  3457042176 Connection::SQLGetInfoW: InfoType: 111
    Jan 17 11:25:08 INFO  3457042176 Statement::SQLGetStmtAttrW: Attribute: 10010
    Jan 17 11:25:08 INFO  3457042176 Statement::SQLGetStmtAttrW: Attribute: 10011
    Jan 17 11:25:08 INFO  3457042176 Statement::SQLGetStmtAttrW: Attribute: 10012
    Jan 17 11:25:08 INFO  3457042176 Statement::SQLGetStmtAttrW: Attribute: 10013
    Jan 17 11:25:08 INFO  3457042176 Connection::ExecuteCatalogFunction: SQLGetTypeInfo
    Jan 17 11:25:08 INFO  3457042176 VMetadataSource::Execute: out_metadataQuery: select * from (select CASE WHEN (UPPER(type_name) = 'CHAR') THEN 'CHAR' WHEN (UPPER(type_name) = 'VARCHAR') THEN 'VARCHAR' ELSE type_name END as data_type_name, CASE WHEN (odbc_type = 1) THEN -8 WHEN (odbc_type = 12) THEN -9 WHEN ( odbc_subtype != 0 ) THEN odbc_subtype ELSE odbc_type END as data_type, column_size, case when type_id not in (5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 114, 115, 116, 117) then null when odbc_type in (-5, 2, 8) then null else E'\'' end as literal_prefix,  case when type_id not in (5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 114, 115, 116, 117) then null when odbc_type in (-5, 2, 8) then null else E'\'' end as literal_suffix, creation_parameters as create_param, 1 as nullable, 1 as case_sensitive, 3 as searchable, 0 as unsigned_attribute, 0 as fixed_prec_scale, 0 as auto_unique, CASE WHEN (UPPER(type_name) = 'CHAR') THEN 'CHAR' WHEN (UPPER(type_name) = 'VARCHAR') THEN 'VARCHAR' ELSE type_name END as local_type_name, min_scale as minimum_scale, max_scale as maximum_scale, CASE WHEN (odbc_type = 1) THEN -8 WHEN (odbc_type = 12) THEN -9 ELSE odbc_type END as sql_data_type, CASE WHEN (odbc_type = 10) THEN (odbc_subtype-100) WHEN (odbc_type = 9) THEN (odbc_subtype-90) ELSE 0 END as sql_datetime_sub, 10 as num_prec_radix, NULL as interval_precision from v_catalog.types order by odbc_type, type_name ) as vmd where data_type = E'93'
    Jan 17 11:25:08 INFO  3457042176 VMetadataSource::Execute: Command status: SELECT, 2 tuples 19 fields
    Jan 17 11:25:08 INFO  3457042176 Statement::SQLGetStmtAttrW: Attribute: 10010
    Jan 17 11:25:08 INFO  3457042176 Statement::SQLGetStmtAttrW: Attribute: 10011
    Jan 17 11:25:08 INFO  3457042176 Statement::SQLGetStmtAttrW: Attribute: 10012
    Jan 17 11:25:08 INFO  3457042176 Statement::SQLGetStmtAttrW: Attribute: 10013
    Jan 17 11:25:13 INFO  3457042176 VDataEngine::Prepare: Original Query: DROP TABLE IF EXISTS foo
    Jan 17 11:25:13 INFO  3457042176 VDataEngine::Prepare: Query is issued as ExecDirect
    Jan 17 11:25:13 INFO  3457042176 VDataEngine::Prepare: Query: DROP TABLE IF EXISTS foo
    Jan 17 11:25:13 INFO  3457042176 VDataEngine::Prepare: Initializing prepared statement: _PLAN0xebe360_0


    * /tmp/vertica_driver.log

    Jan 17 11:25:08 INFO  3457042176 Environment::SQLSetEnvAttr: Attribute: 200
    Jan 17 11:25:08 INFO  3457042176 Environment::SQLGetEnvAttr: Attribute: 200
    Jan 17 11:25:08 INFO  3457042176 VEnvironment::CreateConnection: ODBC Version is: 3


  • matthewcornellmatthewcornell
    Here's some connection information, if that helps: >>> import rpy2.robjects >>> import pyodbc >>> conn = pyodbc.connect('DSN=vertica_kdl_dsn;UserName=xx;Password=xx') >>> for attr in vars(pyodbc): ... try: ... value = conn.getinfo(getattr(pyodbc, attr)) ... print('{:<40s} | {}'.format(attr, value)) ... except: ... pass ... SQL_SQL_CONFORMANCE | 1 SQL_TXN_ISOLATION_OPTION | 10 SQL_PROCEDURE_TERM | function SQL_DROP_TRANSLATION | 0 SQL_CATALOG_USAGE | 0 SQL_SCHEMA_USAGE | 21 SQL_NULLABLE | 0 SQL_STATIC_CURSOR_ATTRIBUTES2 | 0 SQL_STATIC_CURSOR_ATTRIBUTES1 | 0 SQL_MAX_COLUMNS_IN_TABLE | 1600 SQL_GETDATA_EXTENSIONS | 15 SQL_INTERVAL_MONTH | 0 SQL_LIKE_ESCAPE_CLAUSE | True SQL_TIMEDATE_DIFF_INTERVALS | 511 SQL_ROW_UPDATES | False SQL_MAX_ROW_SIZE | 32000000 SQL_INTERVAL_YEAR | 1600 SQL_XOPEN_CLI_YEAR | 1995 SQL_INTERVAL_SECOND | 0 SQL_INTERVAL_MINUTE | 0 SQL_ACTIVE_ENVIRONMENTS | 0 SQL_MAX_INDEX_SIZE | 0 SQL_TIMEDATE_FUNCTIONS | 1998847 SQL_ALTER_TABLE | 62312 SQL_CREATE_TABLE | 1045 SQL_INTERVAL_DAY_TO_MINUTE | 511 SQL_SCOPE_SESSION | vertica_kdl_dsn SQL_SQL92_REVOKE | 13744 SQL_CREATE_TRANSLATION | 0 SQL_MAX_BINARY_LITERAL_LEN | 0 SQL_PARAM_INPUT | 0 SQL_PARAM_ARRAY_ROW_COUNTS | 1 SQL_COLUMN_ALIAS | True SQL_SQL92_ROW_VALUE_CONSTRUCTOR | 3 SQL_SCROLL_OPTIONS | 1 SQL_SCOPE_TRANSACTION | 0 lowercase | 0 SQL_BATCH_ROW_COUNT | 2 SQL_DYNAMIC_CURSOR_ATTRIBUTES2 | 0 SQL_DYNAMIC_CURSOR_ATTRIBUTES1 | 0 SQL_CATALOG_NAME | False SQL_SUBQUERIES | 31 SQL_SCHEMA_TERM | schema SQL_UNION | 3 SQL_NEED_LONG_DATA_LEN | False SQL_DATA_SOURCE_READ_ONLY | False SQL_DDL_INDEX | 0 SQL_NON_NULLABLE_COLUMNS | 1 SQL_MAX_IDENTIFIER_LEN | 128 SQL_QUOTED_IDENTIFIER_CASE | 4 SQL_SERVER_NAME | compute-0-0 SQL_UNKNOWN_TYPE | 0 SQL_NUMERIC | vertica_kdl_dsn SQLWCHAR_SIZE | vertica_kdl_dsn SQL_CORRELATION_NAME | 2 SQL_DESCRIBE_PARAMETER | False SQL_INTERVAL_HOUR_TO_MINUTE | False SQL_EXPRESSIONS_IN_ORDERBY | True SQL_OJ_CAPABILITIES | 127 SQL_TABLE_TERM | table SQL_SQL92_FOREIGN_KEY_DELETE_RULE | 0 SQL_CATALOG_LOCATION | 1 SQL_BOOKMARK_PERSISTENCE | 0 SQL_MAX_COLUMNS_IN_GROUP_BY | 0 SQL_CREATE_DOMAIN | 0 SQL_INTEGRITY | False SQL_DRIVER_NAME | verticaodbcw.so SQL_TYPE_TIME | 0 SQL_DEFAULT_TXN_ISOLATION | 2 SQL_INFO_SCHEMA_VIEWS | 0 SQL_TYPE_TIMESTAMP | 4 SQL_SCOPE_CURROW | 0 SQL_SQL92_NUMERIC_VALUE_FUNCTIONS | 46 SQL_ORDER_BY_COLUMNS_IN_SELECT | True SQL_TYPE_DATE | 21 SQL_CREATE_ASSERTION | 0 SQL_INTERVAL_MINUTE_TO_SECOND | True SQL_NULLABLE_UNKNOWN | vertica_kdl_dsn SQL_MULTIPLE_ACTIVE_TXN | True SQL_STRING_FUNCTIONS | 9732063 SQL_CONCAT_NULL_BEHAVIOR | 0 SQL_MAX_CATALOG_NAME_LEN | 0 SQL_MAX_SCHEMA_NAME_LEN | 128 SQL_MAX_CONCURRENT_ACTIVITIES | 0 SQL_MAX_CURSOR_NAME_LEN | 0 SQL_IDENTIFIER_CASE | 4 SQL_CURSOR_ROLLBACK_BEHAVIOR | 2 SQL_DBMS_VER | 07.00.0000 SQL_DROP_VIEW | 1 SQL_CHAR | 0 SQL_SEARCH_PATTERN_ESCAPE | \ SQL_REAL | 07.00.0000 SQL_GROUP_BY | 2 SQL_DATABASE_NAME | SQL_SQL92_GRANT | 3440 SQL_DROP_TABLE | 7 SQL_ACCESSIBLE_PROCEDURES | False SQL_BATCH_SUPPORT | 3 SQL_MAX_TABLES_IN_SELECT | 0 SQL_ODBC_VER | 03.52 SQL_COLLATION_SEQ | SQL_MAX_STATEMENT_LEN | 0 SQL_DRIVER_VER | 07.00.0000 SQL_DBMS_NAME | Vertica Database SQL_PARAM_INPUT_OUTPUT | vertica_kdl_dsn SQL_INTERVAL_HOUR_TO_SECOND | 0 SQL_PARAM_TYPE_UNKNOWN | 0 SQL_PC_UNKNOWN | 0 SQL_DM_VER | 03.52.0002.0002 SQL_CONVERT_VARCHAR | 12317183 SQL_DRIVER_ODBC_VER | 03.52 SQL_PC_NOT_PSEUDO | 0 SQL_INTERVAL_YEAR_TO_MONTH | 128 SQL_PC_PSEUDO | vertica_kdl_dsn SQL_INTERVAL_DAY | False SQL_FORWARD_ONLY_CURSOR_ATTRIBUTES2 | 4097 SQL_FORWARD_ONLY_CURSOR_ATTRIBUTES1 | 1 SQL_KEYSET_CURSOR_ATTRIBUTES2 | 4097 SQL_SQL92_STRING_FUNCTIONS | 255 SQL_CATALOG_TERM | SQL_INDEX_KEYWORDS | 0 SQL_CREATE_COLLATION | 0 SQL_AGGREGATE_FUNCTIONS | 127 SQL_FLOAT | verticaodbcw.so SQL_IDENTIFIER_QUOTE_CHAR | " SQL_MAX_TABLE_NAME_LEN | 128 SQL_MAX_USER_NAME_LEN | 128 SQL_SYSTEM_FUNCTIONS | 7 SQL_MAX_DRIVER_CONNECTIONS | 0 SQL_SQL92_PREDICATES | 16007 SQL_ODBC_INTERFACE_CONFORMANCE | 1 SQL_SQL92_FOREIGN_KEY_UPDATE_RULE | 0 SQL_MAX_ASYNC_CONCURRENT_STATEMENTS | 0 SQL_NUMERIC_FUNCTIONS | 16121855 SQL_INSERT_STATEMENT | 7 SQL_KEYSET_CURSOR_ATTRIBUTES1 | 4687 SQL_CREATE_SCHEMA | 3 SQL_SQL92_DATETIME_FUNCTIONS | 7 SQL_DROP_SCHEMA | 7 SQL_FILE_USAGE | 0 SQL_MAX_COLUMN_NAME_LEN | 128 SQL_KEYWORDS | SQL_PROCEDURES | False SQL_CREATE_CHARACTER_SET | 0 SQL_DROP_DOMAIN | 0 SQL_NULL_COLLATION | 1 SQL_STANDARD_CLI_CONFORMANCE | 2 SQL_TXN_CAPABLE | 3 SQL_SQL92_VALUE_EXPRESSIONS | 15 SQL_MAX_COLUMNS_IN_SELECT | 0 pooling | 0 SQL_MAX_CHAR_LITERAL_LEN | 0 SQL_ALTER_DOMAIN | 0 SQL_DROP_COLLATION | 0 SQL_PARAM_ARRAY_SELECTS | 3 SQL_CURSOR_COMMIT_BEHAVIOR | 2 SQL_SQL92_RELATIONAL_JOIN_OPERATORS | 474 SQL_MAX_ROW_SIZE_INCLUDES_LONG | False SQL_MAX_COLUMNS_IN_ORDER_BY | 0 SQL_INTERVAL_HOUR | 32000000 SQL_MAX_PROCEDURE_NAME_LEN | 128 SQL_MAX_COLUMNS_IN_INDEX | 0 SQL_CONVERT_FUNCTIONS | 3 SQL_CATALOG_NAME_SEPARATOR | . SQL_ACCESSIBLE_TABLES | False SQL_INTERVAL_DAY_TO_HOUR | 0 threadsafety | 0 SQL_SPECIAL_CHARACTERS | SQL_CREATE_VIEW | 1 SQL_INTERVAL_DAY_TO_SECOND | 511 SQL_DROP_CHARACTER_SET | 0 SQL_USER_NAME | test SQL_TIMEDATE_ADD_INTERVALS | 511 SQL_NO_NULLS | 0 SQL_DATETIME_LITERALS | 65535 SQL_DATA_SOURCE_NAME | vertica_kdl_dsn SQL_DROP_ASSERTION | 0 SQL_MULT_RESULT_SETS | True SQL_ASYNC_MODE | 0 >>>
  • matthewcornellmatthewcornell
    Would someone from Vertica would please give me some advice about how to proceed? I've done a lot of sleuthing to get to this point, and I've tried to include helpful information. I realize there are three major parties involved here (your odbc driver, pyodbc, and, somehow, rpy2), but we need this stack to work so we can use your great software. Thank you.
  • [Deleted User][Deleted User]
    Just to quickly clarify -- is the segfault in Vertica or is it in Python?  (Which process is segfaulting?)

    I've been assuming that it was in Python, but realized that I should confirm.
  • matthewcornellmatthewcornell
    Hi Adam. IIUC, it's on the python/client side. The server stays up and I'm able to access it from vsql just fine. I've tried the pyodbc module with the postgres odbc driver, and it's fine. I've also tried the vertica driver from isql, which also does not segfault. And the pyodbc module works fine with vertica if I do not import the rpy2.objects module. Figures.
  • [Deleted User][Deleted User]
    Hm...  I'm afraid I've got nothin'.  This is one of the stranger bugs I've seen recently, for sure.

    I think the right way to approach this problem would be to install Python, pyodbc, and rpy2 all with debug symbols; then break out gdb and get a proper C/C++-level stack trace of exactly what's going on at the point where things crash.  Both the raw stack itself and, if possible, the values of any local variables in order to cause a crash at that location.

    rpy2 is, if I had to guess, most likely manipulating the global environment in some unusual way that we don't expect.  But I've no idea what's actually going on, so unfortunately can't comment in too much detail...

    One quick thing to check first -- does your system have any unusual ulimits set?  "ulimit -a" will list them.  For example, if you have a very low limit on open file handles, or process memory, etc., it's possible that pyodbc uses some of whatever resource, our driver uses some, rpy2 uses some, and then the process runs out and crashes.  Very unusual -- but then again, this bug seems very unusual, so I figure I might as well ask.


  • matthewcornellmatthewcornell
    Thank you for the suggested next steps, Adam. I'll start the up gdb learning curve.

    ulimit -a on the client: 
    [cornell@wright ~]$ ulimit -a
    core file size          (blocks, -c) 0
    data seg size           (kbytes, -d) unlimited
    scheduling priority             (-e) 0
    file size               (blocks, -f) unlimited
    pending signals                 (-i) 515020
    max locked memory       (kbytes, -l) unlimited
    max memory size         (kbytes, -m) unlimited
    open files                      (-n) 65536
    pipe size            (512 bytes, -p) 8
    POSIX message queues     (bytes, -q) 819200
    real-time priority              (-r) 0
    stack size              (kbytes, -s) 10240
    cpu time               (seconds, -t) unlimited
    max user processes              (-u) 1024
    virtual memory          (kbytes, -v) unlimited
    file locks                      (-x) unlimited
    [cornell@wright ~]$
    The values are the same on the compute node that I'm connecting to, except: 
    pending signals                 (-i) 515020
    max user processes              (-u) 16071

    Do any of the above look off to you?
  • [Deleted User][Deleted User]
    Neither of those ulimit values look problematic to me.

    If you're not familiar with gdb, you might want to first consider cross-posting this question on the pyodbc or rpy2 lists.  Someone on one of those two lists might have some ideas of how to narrow the search.

    It may also be that someone else here has some ideas that I don't.  If so, please feel free to post!
  • matthewcornellmatthewcornell
    Thanks, Adam.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Format
SpoilerCodeQuote
Heading 2Heading 1
Emoji
:):D:(;):/:o:s:p:'(:|B):#:*<3o:)>:)
Url
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file
Home • General DiscussionComment As ...