Vertica crash explanation
Hi guys!
Today I have already 4 times when Vertica node goes down.
I'm trying to research what the problem is, but can't found.
My ErrorReport.txt says:
FAULT => 3: NewEENode
Can someone provide link to manual where I can find some explanations about ErrorReport.txt?
Thanks for help.
Today I have already 4 times when Vertica node goes down.
I'm trying to research what the problem is, but can't found.
My ErrorReport.txt says:
Request: INSERT INTO ad_revenue_partners_aggregated SELECT date(ad_revenue_log_p."created_at") AS "created_at", ad_revenue_log_p.platform AS "platform", users.partner AS "partner_reg_id", partners.name AS "partner_name", partners.country_code AS "partner_country_code", partners.platform AS "partner_platform", partners.campaign AS "partner_campaign", partners.direction AS "partner_direction", date(to_timestamp(users.ts)) AS "user_reg_date", user_geo.country_code AS "user_country_code", SUM( CASE WHEN "ad_revenue"."revenue_type" IS NOT NULL THEN CASE WHEN "ad_revenue"."revenue_type" = 0 THEN ad_revenue.revenue/1000 ELSE ad_revenue.revenue END ELSE CASE WHEN "revenue_type"."type" = 0 THEN ad_revenue.revenue/1000 ELSE ad_revenue.revenue END END ) AS "total_profit", SUM(CASE WHEN "revenue_type"."type" = 0 THEN 1 ELSE 0 END) AS "impressions", SUM(CASE WHEN "revenue_type"."type" = 1 THEN 1 ELSE 0 END) AS "clicks" FROM "public"."ad_revenue_log_p" "ad_revenue_log_p" left JOIN "public"."user_geo" "user_geo" ON ("ad_revenue_log_p"."user_id" = "user_geo"."user_id") left JOIN "public"."revenue_type" "revenue_type" ON ("ad_revenue_log_p"."nad_id" = "revenue_type"."nad_id") left JOIN "public"."ad_revenue" "ad_revenue" ON (("ad_revenue_log_p"."nad_id" = "ad_revenue"."nad_id") AND (date("ad_revenue_log_p"."created_at")= date("ad_revenue"."created_at"))) left JOIN "public"."users" "users" ON ("ad_revenue_log_p"."user_id" = "users"."id") left JOIN "public"."partners" "partners" ON ("users"."partner" = "partners"."id") WHERE date("ad_revenue_log_p"."created_at") = '2014-01-09' GROUP BY date(ad_revenue_log_p.created_at), ad_revenue_log_p.platform, users.partner, partners.name, partners.country_code, partners.platform, partners.campaign, partners.direction, date(to_timestamp(users.ts)), user_geo.country_code 2: Root 0: GroupByNothing 1: DataTarget FAULT => 3: NewEENode ParallelUnion ExprEval GroupByHash ParallelUnion ExprEval Join StorageUnion Scan Join StorageUnion Scan Join StorageMerge (PPFAULT) => Scan Sort StorageUnion Join Join Scan StorageUnion Scan StorageUnion Scan Transaction: [0x00a00000003342a0] END THREAD CONTEXT
All the time database falls on this stepFAULT => 3: NewEENode
Can someone provide link to manual where I can find some explanations about ErrorReport.txt?
Thanks for help.
0
Comments
First, apologies for the crashes!
ErrorReport.txt is not for end-user consumption; it is intended for debugging by Vertica Support and Engineering. (It is captured by Support's automated data-collection tools.)
If you receive an ErrorReport.txt, what that should tell you is that something has gone very wrong with the database... Usually something that can't be fixed in the field; at best worked around, sometimes fixed by a hotfix, etc. Not surprising in this context -- you're seeing the database crash of its own accord; that's, of course, something that we work hard to avoid.
If you have an Enterprise account, please go straight to http://my.vertica.com/ and open a support case. They have tools to help understand what's going on here.
If not -- ErrorReport.txt files are often accompanied by a stack trace or core dump. Could you find and post that stack or core?
Thanks,
Adam
I still have Comunity Edition.
Here backtrace that you asked before (if I correctly understood your response)
BEGIN BACKTRACE Vertica Backtrace at Fri Jan 10 17:00:58 2014 ------------------------- Vertica Analytic Database v6.1.2-0 $BrandId$ vertica(v6.1.2-0) built by release@build2.verticacorp.com from releases/VER_6_1_RELEASE_BUILD_2_0_20130527@114751 on 'Mon May 27 22:05:20 2013' $BuildId$ 00400000-04204000 r-xp 00000000 08:01 8966 /opt/vertica/bin/vertica 04403000-045a9000 rw-p 03e03000 08:01 8966 /opt/vertica/bin/vertica 045a9000-047c3000 rw-p 00000000 00:00 0 0530b000-05721000 rw-p 00000000 00:00 0 [heap] 7fde710f2000-7fde72037000 r--s 00000000 08:01 9505 /opt/vertica/share/icu/icudt42l.dat 7fde72037000-7fde72041000 r-xp 00000000 08:01 546 /lib/libnss_nis-2.11.3.so 7fde72041000-7fde72240000 ---p 0000a000 08:01 546 /lib/libnss_nis-2.11.3.so 7fde72240000-7fde72241000 r--p 00009000 08:01 546 /lib/libnss_nis-2.11.3.so 7fde72241000-7fde72242000 rw-p 0000a000 08:01 546 /lib/libnss_nis-2.11.3.so 7fde72242000-7fde72257000 r-xp 00000000 08:01 109 /lib/libnsl-2.11.3.so 7fde72257000-7fde72456000 ---p 00015000 08:01 109 /lib/libnsl-2.11.3.so 7fde72456000-7fde72457000 r--p 00014000 08:01 109 /lib/libnsl-2.11.3.so 7fde72457000-7fde72458000 rw-p 00015000 08:01 109 /lib/libnsl-2.11.3.so 7fde72458000-7fde7245a000 rw-p 00000000 00:00 0 7fde7245a000-7fde72461000 r-xp 00000000 08:01 103 /lib/libnss_compat-2.11.3.so 7fde72461000-7fde72660000 ---p 00007000 08:01 103 /lib/libnss_compat-2.11.3.so 7fde72660000-7fde72661000 r--p 00006000 08:01 103 /lib/libnss_compat-2.11.3.so 7fde72661000-7fde72662000 rw-p 00007000 08:01 103 /lib/libnss_compat-2.11.3.so 7fde72662000-7fde7266e000 r-xp 00000000 08:01 96 /lib/libnss_files-2.11.3.so 7fde7266e000-7fde7286d000 ---p 0000c000 08:01 96 /lib/libnss_files-2.11.3.so 7fde7286d000-7fde7286e000 r--p 0000b000 08:01 96 /lib/libnss_files-2.11.3.so 7fde7286e000-7fde7286f000 rw-p 0000c000 08:01 96 /lib/libnss_files-2.11.3.so 7fde7286f000-7fde72a6f000 rw-p 00000000 00:00 0 7fde72a6f000-7fde82bcf000 rw-p 00000000 00:00 0 7fde82bcf000-7fde82dcf000 rw-p 00000000 00:00 0 7fde82dcf000-7fde82de5000 r-xp 00000000 08:01 512 /lib/libgcc_s.so.1 7fde82de5000-7fde82fe4000 ---p 00016000 08:01 512 /lib/libgcc_s.so.1 7fde82fe4000-7fde82fe5000 rw-p 00015000 08:01 512 /lib/libgcc_s.so.1 7fde82fe5000-7fde82ffc000 r-xp 00000000 08:01 99 /lib/libpthread-2.11.3.so 7fde82ffc000-7fde831fb000 ---p 00017000 08:01 99 /lib/libpthread-2.11.3.so 7fde831fb000-7fde831fc000 r--p 00016000 08:01 99 /lib/libpthread-2.11.3.so 7fde831fc000-7fde831fd000 rw-p 00017000 08:01 99 /lib/libpthread-2.11.3.so 7fde831fd000-7fde83201000 rw-p 00000000 00:00 0 7fde83201000-7fde83203000 r-xp 00000000 08:01 542 /lib/libdl-2.11.3.so 7fde83203000-7fde83403000 ---p 00002000 08:01 542 /lib/libdl-2.11.3.so 7fde83403000-7fde83404000 r--p 00002000 08:01 542 /lib/libdl-2.11.3.so 7fde83404000-7fde83405000 rw-p 00003000 08:01 542 /lib/libdl-2.11.3.so 7fde83405000-7fde8355e000 r-xp 00000000 08:01 105 /lib/libc-2.11.3.so 7fde8355e000-7fde8375d000 ---p 00159000 08:01 105 /lib/libc-2.11.3.so 7fde8375d000-7fde83761000 r--p 00158000 08:01 105 /lib/libc-2.11.3.so 7fde83761000-7fde83762000 rw-p 0015c000 08:01 105 /lib/libc-2.11.3.so 7fde83762000-7fde83767000 rw-p 00000000 00:00 0 7fde83767000-7fde8376e000 r-xp 00000000 08:01 101 /lib/librt-2.11.3.so 7fde8376e000-7fde8396d000 ---p 00007000 08:01 101 /lib/librt-2.11.3.so 7fde8396d000-7fde8396e000 r--p 00006000 08:01 101 /lib/librt-2.11.3.so 7fde8396e000-7fde8396f000 rw-p 00007000 08:01 101 /lib/librt-2.11.3.so 7fde8396f000-7fde839b1000 r-xp 00000000 08:01 172 /lib/libncurses.so.5.7 7fde839b1000-7fde83bb0000 ---p 00042000 08:01 172 /lib/libncurses.so.5.7 7fde83bb0000-7fde83bb5000 rw-p 00041000 08:01 172 /lib/libncurses.so.5.7 7fde83bb5000-7fde83bd3000 r-xp 00000000 08:01 100 /lib/ld-2.11.3.so 7fde83c36000-7fde83dab000 r--p 00000000 08:05 392223 /usr/lib/locale/locale-archive 7fde83dab000-7fde83dcc000 rw-p 00000000 00:00 0 7fde83dcc000-7fde83dd2000 rw-p 00000000 00:00 0 7fde83dd2000-7fde83dd3000 r--p 0001d000 08:01 100 /lib/ld-2.11.3.so 7fde83dd3000-7fde83dd4000 rw-p 0001e000 08:01 100 /lib/ld-2.11.3.so 7fde83dd4000-7fde83dd5000 rw-p 00000000 00:00 0 7fff3b2af000-7fff3b2d0000 rw-p 00000000 00:00 0 [stack] 7fff3b3ac000-7fff3b3ad000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Backtrace Generated by Error Signal: [0x000000000000000b] PID: [0x0000000000006e54] PC: [0x0000000001ad9ec3] FP: [0x00007fddaf4f6cc0] SIGSEGV: SEGV_MAPERR SI_ADDR : [0x00007fdc4b541060] /opt/vertica/bin/vertica(_ZN6Basics9Backtrace11DoBacktraceEiiPvS1_+0x8cc)[0x2ace2fc] /opt/vertica/bin/vertica(_ZN6Basics20GlobalSignalHandlers14logFatalSignalEiPvS1_+0xb6)[0x2b48674] /opt/vertica/bin/vertica[0x2b49d70] /lib/libc.so.6(+0x32230)[0x7fde83437230] /opt/vertica/bin/vertica(_ZN2EE5VEval10PerformSIPILb0EEEvPciiPi+0xcb3)[0x1ad9ec3] /opt/vertica/bin/vertica(_ZN2EE10DataSource11processRungEiyyPNS_10DataHolderE+0x205)[0x1b3ca1b] /opt/vertica/bin/vertica(_ZN2EE10DataSource11processRungEiyyPNS_10DataHolderE+0xc4f)[0x1b3d465] /opt/vertica/bin/vertica(_ZN2EE10DataSource10getResultsEPNS_10DataHolderE+0x185)[0x1b3dc7f] /opt/vertica/bin/vertica(_ZN2EE11PullProcess20getResultsTranslatedEPNS_10DataHolderE+0x1e8)[0x1afabe6] /opt/vertica/bin/vertica(_ZN2EE17PPGetResultsQueue16processQueueItemEPNS_20InternalCommandQueue9QueueItemE+0x32e)[0x1b54616] /opt/vertica/bin/vertica(_ZN2EE20InternalCommandQueue3runEPS0_RKSt4pairIyxE+0x2a4)[0x1b31f14] /opt/vertica/bin/vertica(_ZNK5boost9function0IvEclEv+0x1bb)[0x15979cb] /opt/vertica/bin/vertica(_ZN7Session13ThreadManager12launchThreadERKN5boost9function0IvEE+0x57)[0x1593ffb] /opt/vertica/bin/vertica(thread_proxy+0x80)[0x3475620] /lib/libpthread.so.0(+0x68ca)[0x7fde82feb8ca] /lib/libc.so.6(clone+0x6d)[0x7fde834d4b6d] END BACKTRACE THREAD CONTEXT Thread type: EE Internal Command Queue Thread Request: INSERT INTO ad_revenue_partners_aggregated SELECT date(ad_revenue_log_p."created_at") AS "created_at", ad_revenue_log_p.platform AS "platform", users.partner AS "partner_reg_id", partners.name AS "partner_name", partners.country_code AS "partner_country_code", partners.platform AS "partner_platform", partners.campaign AS "partner_campaign", partners.direction AS "partner_direction", date(to_timestamp(users.ts)) AS "user_reg_date", user_geo.country_code AS "user_country_code", SUM( CASE WHEN "ad_revenue"."revenue_type" IS NOT NULL THEN CASE WHEN "ad_revenue"."revenue_type" = 0 THEN ad_revenue.revenue/1000 ELSE ad_revenue.revenue END ELSE CASE WHEN "revenue_type"."type" = 0 THEN ad_revenue.revenue/1000 ELSE ad_revenue.revenue END END ) AS "total_profit", SUM(CASE WHEN "revenue_type"."type" = 0 THEN 1 ELSE 0 END) AS "impressions", SUM(CASE WHEN "revenue_type"."type" = 1 THEN 1 ELSE 0 END) AS "clicks" FROM "public"."ad_revenue_log_p" "ad_revenue_log_p" left JOIN "public"."user_geo" "user_geo" ON ("ad_revenue_log_p"."user_id" = "user_geo"."user_id") left JOIN "public"."revenue_type" "revenue_type" ON ("ad_revenue_log_p"."nad_id" = "revenue_type"."nad_id") left JOIN "public"."ad_revenue" "ad_revenue" ON (("ad_revenue_log_p"."nad_id" = "ad_revenue"."nad_id") AND (date("ad_revenue_log_p"."created_at")= date("ad_revenue"."created_at"))) left JOIN "public"."users" "users" ON ("ad_revenue_log_p"."user_id" = "users"."id") left JOIN "public"."partners" "partners" ON ("users"."partner" = "partners"."id") WHERE date("ad_revenue_log_p"."created_at") = '2014-01-09' GROUP BY date(ad_revenue_log_p.created_at), ad_revenue_log_p.platform, users.partner, partners.name, partners.country_code, partners.platform, partners.campaign, partners.direction, date(to_timestamp(users.ts)), user_geo.country_code 2: Root 0: GroupByNothing 1: DataTarget FAULT => 3: NewEENode ParallelUnion ExprEval GroupByHash ParallelUnion ExprEval Join StorageUnion Scan Join StorageUnion Scan Join StorageMerge (PPFAULT) => Scan Sort StorageUnion Join Join Scan StorageUnion Scan StorageUnion Scan Transaction: [0x00a00000003342a0] END THREAD CONTEXTIs there any way to determine what happened for me? Maybe I need to tune my Vertica configuration or create some projections?Thanks. And another "Thanks" for your great product, I really enjoy it
Thanks for your helpful reply with the stack trace.
It would be helpful to see some additional diagnostic information about your issue. Could you run the diagnostic collection tool for us and point us to the output? The instructions for the diagnostic tools can be found here:
https://my.vertica.com/docs/7.0.x/HTML/index.htm#Authoring/_IntrabookContent/Links/CollectingDiagnos...
Thanks!
Jason
I agree, it is best to follow up off-thread. I will do that right away.
Jason