Vertica node keeps crashing after using flex tables
Hi,
Our company is looking into adopting Vertica as our data warehouse platform, but we're starting to notice recently in our 3-node cluster setup, at least one of the nodes would crash everytime we run our ETL scripts using flex table. We did not see this type of issues before we started using flex to do our ETL, so we thought maybe that's related to that.
Below is the ErrorReport.txt. Any insight into this would be greatly appreciated, thanks!
Austin
BEGIN BACKTRACE
Vertica Backtrace at Tue Apr 7 01:12:21 2015
-------------------------
Vertica Analytic Database v7.1.1-0 $BrandId$
vertica(v7.1.1-0) built by release@build2.verticacorp.com from releases/VER_7_1_RELEASE_BUILD_1_0_20141016@148158 on 'Thu Oct 16 15:55:09 America/New_York 2014' $BuildId$
00400000-04e37000 r-xp 00000000 ca:01 153890 /opt/vertica/bin/vertica
05036000-05d92000 rw-p 04a36000 ca:01 153890 /opt/vertica/bin/vertica
05d92000-05eab000 rw-p 00000000 00:00 0
0642f000-068ef000 rw-p 00000000 00:00 0 [heap]
3163e00000-3163e20000 r-xp 00000000 ca:01 15931 /lib64/ld-2.12.so
316401f000-3164020000 r--p 0001f000 ca:01 15931 /lib64/ld-2.12.so
3164020000-3164021000 rw-p 00020000 ca:01 15931 /lib64/ld-2.12.so
3164021000-3164022000 rw-p 00000000 00:00 0
3164200000-316438a000 r-xp 00000000 ca:01 15932 /lib64/libc-2.12.so
316438a000-316458a000 ---p 0018a000 ca:01 15932 /lib64/libc-2.12.so
316458a000-316458e000 r--p 0018a000 ca:01 15932 /lib64/libc-2.12.so
316458e000-316458f000 rw-p 0018e000 ca:01 15932 /lib64/libc-2.12.so
316458f000-3164594000 rw-p 00000000 00:00 0
3164600000-3164602000 r-xp 00000000 ca:01 15938 /lib64/libdl-2.12.so
3164602000-3164802000 ---p 00002000 ca:01 15938 /lib64/libdl-2.12.so
3164802000-3164803000 r--p 00002000 ca:01 15938 /lib64/libdl-2.12.so
3164803000-3164804000 rw-p 00003000 ca:01 15938 /lib64/libdl-2.12.so
3164a00000-3164a17000 r-xp 00000000 ca:01 15933 /lib64/libpthread-2.12.so
3164a17000-3164c17000 ---p 00017000 ca:01 15933 /lib64/libpthread-2.12.so
3164c17000-3164c18000 r--p 00017000 ca:01 15933 /lib64/libpthread-2.12.so
3164c18000-3164c19000 rw-p 00018000 ca:01 15933 /lib64/libpthread-2.12.so
3164c19000-3164c1d000 rw-p 00000000 00:00 0
3165600000-3165607000 r-xp 00000000 ca:01 15937 /lib64/librt-2.12.so
3165607000-3165806000 ---p 00007000 ca:01 15937 /lib64/librt-2.12.so
3165806000-3165807000 r--p 00006000 ca:01 15937 /lib64/librt-2.12.so
3165807000-3165808000 rw-p 00007000 ca:01 15937 /lib64/librt-2.12.so
3165e00000-3165e22000 r-xp 00000000 ca:01 824 /lib64/libncurses.so.5.7
3165e22000-3166021000 ---p 00022000 ca:01 824 /lib64/libncurses.so.5.7
3166021000-3166022000 rw-p 00021000 ca:01 824 /lib64/libncurses.so.5.7
3166a00000-3166a1d000 r-xp 00000000 ca:01 3202 /lib64/libtinfo.so.5.7
3166a1d000-3166c1d000 ---p 0001d000 ca:01 3202 /lib64/libtinfo.so.5.7
3166c1d000-3166c21000 rw-p 0001d000 ca:01 3202 /lib64/libtinfo.so.5.7
3167200000-3167216000 r-xp 00000000 ca:01 15936 /lib64/libgcc_s-4.4.7-20120601.so.1
3167216000-3167415000 ---p 00016000 ca:01 15936 /lib64/libgcc_s-4.4.7-20120601.so.1
3167415000-3167416000 rw-p 00015000 ca:01 15936 /lib64/libgcc_s-4.4.7-20120601.so.1
7f240e704000-7f2414595000 r--p 00000000 ca:01 275814 /usr/lib/locale/locale-archive
7f2414595000-7f24154da000 r--s 00000000 ca:01 277030 /opt/vertica/share/icu/icudt42l.dat
7f24154da000-7f242563a000 rw-p 00000000 00:00 0
7f242563a000-7f2425a3c000 rw-p 00000000 00:00 0
7f2425a3c000-7f2425a48000 r-xp 00000000 ca:01 14261 /lib64/libnss_files-2.12.so
7f2425a48000-7f2425c48000 ---p 0000c000 ca:01 14261 /lib64/libnss_files-2.12.so
7f2425c48000-7f2425c49000 r--p 0000c000 ca:01 14261 /lib64/libnss_files-2.12.so
7f2425c49000-7f2425c4a000 rw-p 0000d000 ca:01 14261 /lib64/libnss_files-2.12.so
7f2425c4a000-7f2425c4f000 rw-p 00000000 00:00 0
7f2425c5c000-7f2425c5d000 rw-p 00000000 00:00 0
7fff7f594000-7fff7f5a9000 rw-p 00000000 00:00 0 [stack]
7fff7f5ff000-7fff7f600000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Backtrace Generated by Error
Signal: [0x000000000000000b] PID: [0x000000000001d8a2] PC: [0x0000000001987b45] FP: [0x00007f1bc5bd4460] SIGSEGV: SEGV_ACCERR SI_ADDR : [0x00007f21c61fd000]
Comments
Hi Austin
Can you kindly provide us the complete ErrorReport.txt along with the vertica.log files of the crashed node just to have a closer look into this issue?
Regards
Rahul Choudhary
attached
Looks like below queries with analytical function have crashed your cluster:
insert into r4e_mongo_staging.mongo_repbiz_tenant_configurations_managedbyids_staging select cast(id as varchar), r4e_mongo.group_concat(values) over (partition by id) as values from (select maplookup(__raw__,'_id') as id, mapitems(maplookup(__raw__,'managedByUserIDs')) over (partition by _id) from r4e_mongo_staging.flx_tenant_configurations) as a ;
insert into r4e_mongo_staging.mongo_repbiz_tenant_configurations_photos_staging select cast (id as varchar), count(cast (values as varchar)) as count from ( select maplookup(__raw__,'_id') as id, mapitems(maplookup(maplookup(__raw__,'publish'),'photos')) over (partition by _id) from r4e_mongo_staging.flx_tenant_configurations_array) a group by 1;
insert into r4e_mongo_staging.mongo_repbiz_tenant_configurations_links_staging select cast (id as varchar), count(cast (values as varchar)) as count from ( select maplookup(__raw__,'_id') as id, mapitems(maplookup(maplookup(__raw__,'publish'),'links')) over (partition by _id) from r4e_mongo_staging.flx_tenant_configurations_array) a group by 1;
Have you starting using these insert queries recently?If not & ran it earlier as well ever noticed these causing the similiar crash issue?Try running the same query from vsql & see if it causes node/cluster crash again?
Regards
Rahul
Out of curiousity, are you using the flatten_arrays=true parameter when loading in the JSON?