Vertica crash explanation

Comments

• [Deleted User]

  January 2014

  Hi Dmitriy,
  
  First, apologies for the crashes!
  
  ErrorReport.txt is not for end-user consumption; it is intended for debugging by Vertica Support and Engineering.  (It is captured by Support's automated data-collection tools.)
  
  If you receive an ErrorReport.txt, what that should tell you is that something has gone very wrong with the database...  Usually something that can't be fixed in the field; at best worked around, sometimes fixed by a hotfix, etc.  Not surprising in this context -- you're seeing the database crash of its own accord; that's, of course, something that we work hard to avoid.
  
  If you have an Enterprise account, please go straight to http://my.vertica.com/ and open a support case.  They have tools to help understand what's going on here.
  
  If not -- ErrorReport.txt files are often accompanied by a stack trace or core dump.  Could you find and post that stack or core?
  
  Thanks,
  Adam

  0
• Dmitriy_Kononov

  January 2014

  Thanks for respond.
  
  I still have Comunity Edition.
  
  Here backtrace that you asked before (if I correctly understood your response)
  

  BEGIN BACKTRACE Vertica Backtrace at Fri Jan 10 17:00:58 2014 ------------------------- Vertica Analytic Database v6.1.2-0 $BrandId$ vertica(v6.1.2-0) built by release@build2.verticacorp.com from releases/VER_6_1_RELEASE_BUILD_2_0_20130527@114751 on 'Mon May 27 22:05:20 2013' $BuildId$ 00400000-04204000 r-xp 00000000 08:01 8966                               /opt/vertica/bin/vertica 04403000-045a9000 rw-p 03e03000 08:01 8966                               /opt/vertica/bin/vertica 045a9000-047c3000 rw-p 00000000 00:00 0  0530b000-05721000 rw-p 00000000 00:00 0                                  [heap] 7fde710f2000-7fde72037000 r--s 00000000 08:01 9505                       /opt/vertica/share/icu/icudt42l.dat 7fde72037000-7fde72041000 r-xp 00000000 08:01 546                        /lib/libnss_nis-2.11.3.so 7fde72041000-7fde72240000 ---p 0000a000 08:01 546                        /lib/libnss_nis-2.11.3.so 7fde72240000-7fde72241000 r--p 00009000 08:01 546                        /lib/libnss_nis-2.11.3.so 7fde72241000-7fde72242000 rw-p 0000a000 08:01 546                        /lib/libnss_nis-2.11.3.so 7fde72242000-7fde72257000 r-xp 00000000 08:01 109                        /lib/libnsl-2.11.3.so 7fde72257000-7fde72456000 ---p 00015000 08:01 109                        /lib/libnsl-2.11.3.so 7fde72456000-7fde72457000 r--p 00014000 08:01 109                        /lib/libnsl-2.11.3.so 7fde72457000-7fde72458000 rw-p 00015000 08:01 109                        /lib/libnsl-2.11.3.so 7fde72458000-7fde7245a000 rw-p 00000000 00:00 0  7fde7245a000-7fde72461000 r-xp 00000000 08:01 103                        /lib/libnss_compat-2.11.3.so 7fde72461000-7fde72660000 ---p 00007000 08:01 103                        /lib/libnss_compat-2.11.3.so 7fde72660000-7fde72661000 r--p 00006000 08:01 103                        /lib/libnss_compat-2.11.3.so 7fde72661000-7fde72662000 rw-p 00007000 08:01 103                        /lib/libnss_compat-2.11.3.so 7fde72662000-7fde7266e000 r-xp 00000000 08:01 96                         /lib/libnss_files-2.11.3.so 7fde7266e000-7fde7286d000 ---p 0000c000 08:01 96                         /lib/libnss_files-2.11.3.so 7fde7286d000-7fde7286e000 r--p 0000b000 08:01 96                         /lib/libnss_files-2.11.3.so 7fde7286e000-7fde7286f000 rw-p 0000c000 08:01 96                         /lib/libnss_files-2.11.3.so 7fde7286f000-7fde72a6f000 rw-p 00000000 00:00 0  7fde72a6f000-7fde82bcf000 rw-p 00000000 00:00 0  7fde82bcf000-7fde82dcf000 rw-p 00000000 00:00 0  7fde82dcf000-7fde82de5000 r-xp 00000000 08:01 512                        /lib/libgcc_s.so.1 7fde82de5000-7fde82fe4000 ---p 00016000 08:01 512                        /lib/libgcc_s.so.1 7fde82fe4000-7fde82fe5000 rw-p 00015000 08:01 512                        /lib/libgcc_s.so.1 7fde82fe5000-7fde82ffc000 r-xp 00000000 08:01 99                         /lib/libpthread-2.11.3.so 7fde82ffc000-7fde831fb000 ---p 00017000 08:01 99                         /lib/libpthread-2.11.3.so 7fde831fb000-7fde831fc000 r--p 00016000 08:01 99                         /lib/libpthread-2.11.3.so 7fde831fc000-7fde831fd000 rw-p 00017000 08:01 99                         /lib/libpthread-2.11.3.so 7fde831fd000-7fde83201000 rw-p 00000000 00:00 0  7fde83201000-7fde83203000 r-xp 00000000 08:01 542                        /lib/libdl-2.11.3.so 7fde83203000-7fde83403000 ---p 00002000 08:01 542                        /lib/libdl-2.11.3.so 7fde83403000-7fde83404000 r--p 00002000 08:01 542                        /lib/libdl-2.11.3.so 7fde83404000-7fde83405000 rw-p 00003000 08:01 542                        /lib/libdl-2.11.3.so 7fde83405000-7fde8355e000 r-xp 00000000 08:01 105                        /lib/libc-2.11.3.so 7fde8355e000-7fde8375d000 ---p 00159000 08:01 105                        /lib/libc-2.11.3.so 7fde8375d000-7fde83761000 r--p 00158000 08:01 105                        /lib/libc-2.11.3.so 7fde83761000-7fde83762000 rw-p 0015c000 08:01 105                        /lib/libc-2.11.3.so 7fde83762000-7fde83767000 rw-p 00000000 00:00 0  7fde83767000-7fde8376e000 r-xp 00000000 08:01 101                        /lib/librt-2.11.3.so 7fde8376e000-7fde8396d000 ---p 00007000 08:01 101                        /lib/librt-2.11.3.so 7fde8396d000-7fde8396e000 r--p 00006000 08:01 101                        /lib/librt-2.11.3.so 7fde8396e000-7fde8396f000 rw-p 00007000 08:01 101                        /lib/librt-2.11.3.so 7fde8396f000-7fde839b1000 r-xp 00000000 08:01 172                        /lib/libncurses.so.5.7 7fde839b1000-7fde83bb0000 ---p 00042000 08:01 172                        /lib/libncurses.so.5.7 7fde83bb0000-7fde83bb5000 rw-p 00041000 08:01 172                        /lib/libncurses.so.5.7 7fde83bb5000-7fde83bd3000 r-xp 00000000 08:01 100                        /lib/ld-2.11.3.so 7fde83c36000-7fde83dab000 r--p 00000000 08:05 392223                     /usr/lib/locale/locale-archive 7fde83dab000-7fde83dcc000 rw-p 00000000 00:00 0  7fde83dcc000-7fde83dd2000 rw-p 00000000 00:00 0  7fde83dd2000-7fde83dd3000 r--p 0001d000 08:01 100                        /lib/ld-2.11.3.so 7fde83dd3000-7fde83dd4000 rw-p 0001e000 08:01 100                        /lib/ld-2.11.3.so 7fde83dd4000-7fde83dd5000 rw-p 00000000 00:00 0  7fff3b2af000-7fff3b2d0000 rw-p 00000000 00:00 0                          [stack] 7fff3b3ac000-7fff3b3ad000 r-xp 00000000 00:00 0                          [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]  Backtrace Generated by Error Signal: [0x000000000000000b] PID: [0x0000000000006e54] PC: [0x0000000001ad9ec3] FP: [0x00007fddaf4f6cc0] SIGSEGV: SEGV_MAPERR SI_ADDR : [0x00007fdc4b541060] /opt/vertica/bin/vertica(_ZN6Basics9Backtrace11DoBacktraceEiiPvS1_+0x8cc)[0x2ace2fc] /opt/vertica/bin/vertica(_ZN6Basics20GlobalSignalHandlers14logFatalSignalEiPvS1_+0xb6)[0x2b48674] /opt/vertica/bin/vertica[0x2b49d70] /lib/libc.so.6(+0x32230)[0x7fde83437230] /opt/vertica/bin/vertica(_ZN2EE5VEval10PerformSIPILb0EEEvPciiPi+0xcb3)[0x1ad9ec3] /opt/vertica/bin/vertica(_ZN2EE10DataSource11processRungEiyyPNS_10DataHolderE+0x205)[0x1b3ca1b] /opt/vertica/bin/vertica(_ZN2EE10DataSource11processRungEiyyPNS_10DataHolderE+0xc4f)[0x1b3d465] /opt/vertica/bin/vertica(_ZN2EE10DataSource10getResultsEPNS_10DataHolderE+0x185)[0x1b3dc7f] /opt/vertica/bin/vertica(_ZN2EE11PullProcess20getResultsTranslatedEPNS_10DataHolderE+0x1e8)[0x1afabe6] /opt/vertica/bin/vertica(_ZN2EE17PPGetResultsQueue16processQueueItemEPNS_20InternalCommandQueue9QueueItemE+0x32e)[0x1b54616] /opt/vertica/bin/vertica(_ZN2EE20InternalCommandQueue3runEPS0_RKSt4pairIyxE+0x2a4)[0x1b31f14] /opt/vertica/bin/vertica(_ZNK5boost9function0IvEclEv+0x1bb)[0x15979cb] /opt/vertica/bin/vertica(_ZN7Session13ThreadManager12launchThreadERKN5boost9function0IvEE+0x57)[0x1593ffb] /opt/vertica/bin/vertica(thread_proxy+0x80)[0x3475620] /lib/libpthread.so.0(+0x68ca)[0x7fde82feb8ca] /lib/libc.so.6(clone+0x6d)[0x7fde834d4b6d] END BACKTRACE THREAD CONTEXT Thread type: EE Internal Command Queue Thread Request: INSERT INTO ad_revenue_partners_aggregated SELECT date(ad_revenue_log_p."created_at") AS "created_at", ad_revenue_log_p.platform AS "platform", users.partner AS "partner_reg_id", partners.name AS "partner_name", partners.country_code AS "partner_country_code", partners.platform AS "partner_platform", partners.campaign AS "partner_campaign", partners.direction AS "partner_direction", date(to_timestamp(users.ts)) AS "user_reg_date", user_geo.country_code AS "user_country_code", SUM( CASE WHEN "ad_revenue"."revenue_type" IS NOT NULL THEN CASE WHEN "ad_revenue"."revenue_type" = 0 THEN ad_revenue.revenue/1000 ELSE ad_revenue.revenue END ELSE CASE WHEN "revenue_type"."type" = 0 THEN ad_revenue.revenue/1000 ELSE ad_revenue.revenue END END ) AS "total_profit", SUM(CASE WHEN "revenue_type"."type" = 0 THEN 1 ELSE 0 END) AS "impressions", SUM(CASE WHEN "revenue_type"."type" = 1 THEN 1 ELSE 0 END) AS "clicks" FROM "public"."ad_revenue_log_p" "ad_revenue_log_p" left JOIN "public"."user_geo" "user_geo" ON ("ad_revenue_log_p"."user_id" = "user_geo"."user_id") left JOIN "public"."revenue_type" "revenue_type" ON ("ad_revenue_log_p"."nad_id" = "revenue_type"."nad_id") left JOIN "public"."ad_revenue" "ad_revenue" ON (("ad_revenue_log_p"."nad_id" = "ad_revenue"."nad_id") AND (date("ad_revenue_log_p"."created_at")= date("ad_revenue"."created_at"))) left JOIN "public"."users" "users" ON ("ad_revenue_log_p"."user_id" = "users"."id") left JOIN "public"."partners" "partners" ON ("users"."partner" = "partners"."id") WHERE date("ad_revenue_log_p"."created_at") = '2014-01-09' GROUP BY date(ad_revenue_log_p.created_at), ad_revenue_log_p.platform, users.partner, partners.name, partners.country_code, partners.platform, partners.campaign, partners.direction, date(to_timestamp(users.ts)), user_geo.country_code            2: Root            0: GroupByNothing            1: DataTarget FAULT =>   3: NewEENode               ParallelUnion               ExprEval               GroupByHash               ParallelUnion               ExprEval               Join               StorageUnion               Scan               Join               StorageUnion               Scan               Join               StorageMerge  (PPFAULT) => Scan               Sort               StorageUnion               Join               Join               Scan               StorageUnion               Scan               StorageUnion               Scan Transaction: [0x00a00000003342a0] END THREAD CONTEXT 

  Is there any way to determine what happened for me? Maybe I need to tune my Vertica configuration or create some projections?
   Thanks. And another "Thanks" for your great product, I really enjoy it

  0
• [Deleted User]

  January 2014

  Hello Dmitriy,
  
  Thanks for your helpful reply with the stack trace.
  
  It would be helpful to see some additional diagnostic information about your issue. Could you run the diagnostic collection tool for us and point us to the output? The instructions for the diagnostic tools can be found here:
  
  https://my.vertica.com/docs/7.0.x/HTML/index.htm#Authoring/_IntrabookContent/Links/CollectingDiagnos...
  
  Thanks!
  
  Jason
  

  1
• Dmitriy_Kononov

  January 2014

  Yes sure, but this diagnostic results contain some private information about server, can you, please give your email, I will send it to you?

  0
• [Deleted User]

  January 2014

  Hi Dmitriy,
  
  I agree, it is best to follow up off-thread. I will do that right away.
  
  Jason
  

  0