Sudden node shutdown
Hello,
One of the nodes on our 3 node AWS cluster started to shut down with the following message:
One of the nodes on our 3 node AWS cluster started to shut down with the following message:
2014-08-03 22:26:54.328 EEcmdq:0x7f32f0cec950 [Main] <PANIC> Received fatal signal SIGSEGV.
2014-08-03 22:26:54.328 EEcmdq:0x7f32f0cec950 [Main] <PANIC> Info: si_code: 2, si_pid: -2087288392, si_uid: 32564, si_addr: 0x7f34839681b8
This has been happening the last 2 days out of the sudden, happened multiple times and only to the same node.
Can someone help me to troubleshoot this?
Thank you,
Tibor
0
Comments
You may run the below and see if some unusual things
$ dmesg
If you are a customer of vertica, you may open a case and support engineer will help you
Thanks for your response. Nothing runs on that node except Vertica.
The output of dmesg is below. Does this have anything unusual?
Thanks for any hints,
Tibor
Initializing cgroup subsys cpusetInitializing cgroup subsys cpu
Linux version 2.6.32-358.14.1.el6.x86_64 (mockbuild@x86-022.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC) ) #1 SMP Mon Jun 17 15:54:20 EDT 2013
Command line: console=ttyS0 ro root=LABEL=_/
KERNEL supported cpus:
Intel GenuineIntel
AMD AuthenticAMD
Centaur CentaurHauls
ACPI in unprivileged domain disabled
released 0 pages of unused memory
BIOS-provided physical RAM map:
Xen: 0000000000000000 - 00000000000a0000 (usable)
Xen: 00000000000a0000 - 0000000000100000 (reserved)
Xen: 0000000000100000 - 0000000780000000 (usable)
DMI not present or invalid.
e820 update range: 0000000000000000 - 0000000000001000 (usable) ==> (reserved)
e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
last_pfn = 0x780000 max_arch_pfn = 0x400000000
x2apic enabled by BIOS, switching to x2apic ops
last_pfn = 0x100000 max_arch_pfn = 0x400000000
initial memory mapped : 0 - 20000000
init_memory_mapping: 0000000000000000-0000000100000000
0000000000 - 0100000000 page 4k
kernel direct mapping tables up to 100000000 @ 100000-905000
init_memory_mapping: 0000000100000000-0000000780000000
0100000000 - 0780000000 page 4k
kernel direct mapping tables up to 780000000 @ 88c4000-bcdf000
RAMDISK: 0203c000 - 04c78000
No NUMA configuration found
Faking a node at 0000000000000000-0000000780000000
Bootmem setup node 0 0000000000000000-0000000780000000
NODE_DATA [0000000000008000 - 000000000003bfff]
bootmap [00000000008b9000 - 00000000009a8fff] pages f0
(8 early reservations) ==> bootmem [0000000000 - 0780000000]
#0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000]
#1 [000887b000 - 00088c4000] XEN PAGETABLES ==> [000887b000 - 00088c4000]
#2 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000]
#3 [0001000000 - 000201b0a4] TEXT DATA BSS ==> [0001000000 - 000201b0a4]
#4 [000203c000 - 0004c78000] RAMDISK ==> [000203c000 - 0004c78000]
#5 [0004c78000 - 000887b000] XEN START INFO ==> [0004c78000 - 000887b000]
#6 [0000100000 - 00008b9000] PGTABLE ==> [0000100000 - 00008b9000]
#7 [00088c4000 - 000bcde000] PGTABLE ==> [00088c4000 - 000bcde000]
Zone PFN ranges:
DMA 0x00000001 -> 0x00001000
DMA32 0x00001000 -> 0x00100000
Normal 0x00100000 -> 0x00780000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
0: 0x00000001 -> 0x000000a0
0: 0x00000100 -> 0x00780000
On node 0 totalpages: 7864223
DMA zone: 56 pages used for memmap
DMA zone: 1980 pages reserved
DMA zone: 1963 pages, LIFO batch:0
DMA32 zone: 14280 pages used for memmap
DMA32 zone: 1030200 pages, LIFO batch:31
Normal zone: 93184 pages used for memmap
Normal zone: 6722560 pages, LIFO batch:31
SFI: Simple Firmware Interface v0.7 http://simplefirmware.org
SMP: Allowing 16 CPUs, 0 hotplug CPUs
nr_irqs_gsi: 16
PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
PCI: Warning: Cannot find a gap in the 32bit address range
PCI: Unassigned devices with 32bit resource registers may break!
Allocating PCI resources starting at 780100000 (gap: 780100000:400000)
Booting paravirtualized kernel on Xen
Xen version: 4.2.amazon (preserve-AD)
NR_CPUS:4096 nr_cpumask_bits:16 nr_cpu_ids:16 nr_node_ids:1
PERCPU: Embedded 31 pages/cpu @ffff880028050000 s94552 r8192 d24232 u126976
pcpu-alloc: s94552 r8192 d24232 u126976 alloc=31*4096
pcpu-alloc: [0] 00 [0] 01 [0] 02 [0] 03 [0] 04 [0] 05 [0] 06 [0] 07
pcpu-alloc: [0] 08 [0] 09 [0] 10 [0] 11 [0] 12 [0] 13 [0] 14 [0] 15
trying to map vcpu_info 0 at ffff88002805b020, mfn 809c0a, offset 32
cpu 0 using vcpu_info at ffff88002805b020
trying to map vcpu_info 1 at ffff88002807a020, mfn 809beb, offset 32
cpu 1 using vcpu_info at ffff88002807a020
trying to map vcpu_info 2 at ffff880028099020, mfn 809bcc, offset 32
cpu 2 using vcpu_info at ffff880028099020
trying to map vcpu_info 3 at ffff8800280b8020, mfn 809bad, offset 32
cpu 3 using vcpu_info at ffff8800280b8020
trying to map vcpu_info 4 at ffff8800280d7020, mfn 809b8e, offset 32
cpu 4 using vcpu_info at ffff8800280d7020
trying to map vcpu_info 5 at ffff8800280f6020, mfn 809b6f, offset 32
cpu 5 using vcpu_info at ffff8800280f6020
trying to map vcpu_info 6 at ffff880028115020, mfn 809b50, offset 32
cpu 6 using vcpu_info at ffff880028115020
trying to map vcpu_info 7 at ffff880028134020, mfn 809b31, offset 32
cpu 7 using vcpu_info at ffff880028134020
trying to map vcpu_info 8 at ffff880028153020, mfn 809b12, offset 32
cpu 8 using vcpu_info at ffff880028153020
trying to map vcpu_info 9 at ffff880028172020, mfn 809af3, offset 32
cpu 9 using vcpu_info at ffff880028172020
trying to map vcpu_info 10 at ffff880028191020, mfn 809ad4, offset 32
cpu 10 using vcpu_info at ffff880028191020
trying to map vcpu_info 11 at ffff8800281b0020, mfn 809ab5, offset 32
cpu 11 using vcpu_info at ffff8800281b0020
trying to map vcpu_info 12 at ffff8800281cf020, mfn 809a96, offset 32
cpu 12 using vcpu_info at ffff8800281cf020
trying to map vcpu_info 13 at ffff8800281ee020, mfn 809a77, offset 32
cpu 13 using vcpu_info at ffff8800281ee020
trying to map vcpu_info 14 at ffff88002820d020, mfn 809a58, offset 32
cpu 14 using vcpu_info at ffff88002820d020
trying to map vcpu_info 15 at ffff88002822c020, mfn 809a39, offset 32
cpu 15 using vcpu_info at ffff88002822c020
Xen: using vcpu_info placement
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 7754723
Policy zone: Normal
Kernel command line: console=ttyS0 ro root=LABEL=_/
PID hash table entries: 4096 (order: 3, 32768 bytes)
Checking aperture...
No AGP bridge found
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
software IO TLB at phys 0x20000000 - 0x24000000
Memory: 30772628k/31457280k available (5222k kernel code, 388k absent, 684264k reserved, 7120k data, 1264k init)
Hierarchical RCU implementation.
NR_IRQS:33024 nr_irqs:400
Console: colour dummy device 80x25
console [tty0] enabled
console [hvc0] enabled
console [ttyS0] enabled
allocated 125829120 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
Xen: using vcpuop timer interface
installing Xen timer for CPU 0
alloc irq_desc for 399 on node 0
alloc kstat_irqs on node 0
Detected 2800.042 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 5600.08 BogoMIPS (lpj=2800042)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux: Initializing.
SELinux: Starting in permissive mode
Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
Initializing cgroup subsys perf_event
Initializing cgroup subsys net_prio
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
alternatives: switching to unfair spinlock
SMP alternatives: switching to UP code
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 21438 entries in 85 pages
alloc irq_desc for 398 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 397 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 396 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 395 on node 0
alloc kstat_irqs on node 0
Performance Events: unsupported p6 CPU model 62 no PMU driver, software events only.
NMI watchdog disabled (cpu0): hardware events not enabled
installing Xen timer for CPU 1
alloc irq_desc for 394 on node 0
alloc kstat_irqs on node 0
SMP alternatives: switching to SMP code
alloc irq_desc for 393 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 392 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 391 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 390 on node 0
alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
installing Xen timer for CPU 2
alloc irq_desc for 389 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 388 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 387 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 386 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 385 on node 0
alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
installing Xen timer for CPU 3
alloc irq_desc for 384 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 383 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 382 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 381 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 380 on node 0
alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
installing Xen timer for CPU 4
alloc irq_desc for 379 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 378 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 377 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 376 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 375 on node 0
alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
installing Xen timer for CPU 5
alloc irq_desc for 374 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 373 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 372 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 371 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 370 on node 0
alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
installing Xen timer for CPU 6
alloc irq_desc for 369 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 368 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 367 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 366 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 365 on node 0
alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
installing Xen timer for CPU 7
alloc irq_desc for 364 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 363 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 362 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 361 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 360 on node 0
alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
installing Xen timer for CPU 8
alloc irq_desc for 359 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 358 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 357 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 356 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 355 on node 0
alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
installing Xen timer for CPU 9
alloc irq_desc for 354 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 353 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 352 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 351 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 350 on node 0
alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
installing Xen timer for CPU 10
alloc irq_desc for 349 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 348 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 347 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 346 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 345 on node 0
alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
installing Xen timer for CPU 11
alloc irq_desc for 344 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 343 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 342 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 341 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 340 on node 0
alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
installing Xen timer for CPU 12
alloc irq_desc for 339 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 338 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 337 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 336 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 335 on node 0
alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
installing Xen timer for CPU 13
alloc irq_desc for 334 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 333 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 332 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 331 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 330 on node 0
alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
installing Xen timer for CPU 14
alloc irq_desc for 329 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 328 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 327 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 326 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 325 on node 0
alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
installing Xen timer for CPU 15
alloc irq_desc for 324 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 323 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 322 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 321 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 320 on node 0
alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
Brought up 16 CPUs
sizeof(vma)=200 bytes
sizeof(page)=56 bytes
sizeof(inode)=592 bytes
sizeof(dentry)=192 bytes
sizeof(ext3inode)=800 bytes
sizeof(buffer_head)=104 bytes
sizeof(skbuff)=232 bytes
sizeof(task_struct)=2648 bytes
devtmpfs: initialized
Grant table initialized
regulator: core version 0.5
NET: Registered protocol family 16
alloc irq_desc for 319 on node 0
alloc kstat_irqs on node 0
PCI: Fatal: No config space access function found
bio: create slab <bio-0> at 0
ACPI: Interpreter disabled.
xen_balloon: Initialising balloon driver.
last_pfn = 0x780000 max_arch_pfn = 0x400000000
vgaarb: loaded
SCSI subsystem initialized
libata version 3.00 loaded.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: System does not support PCI
PCI: System does not support PCI
NetLabel: Initializing
NetLabel: domain hash size = 128
NetLabel: protocols = UNLABELED CIPSOv4
NetLabel: unlabeled traffic allowed by default
Switching to clocksource xen
pnp: PnP ACPI: disabled
PCI: max bus depth: 0 pci_try_num: 1
NET: Registered protocol family 2
IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
NET: Registered protocol family 1
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 45296k freed
platform rtc_cmos: registered platform RTC device (no PNP device found)
audit: initializing netlink socket (disabled)
type=2000 audit(1406791297.714:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 32768
SELinux: Registering netfilter hooks
alg: No test for stdrng (krng)
ksign: Installing public key data
Loading keyring
- Added public key 9399FA7596B20C85
- User ID: Red Hat, Inc. (Kernel Module GPG key)
- Added public key D4A26C9CCD09BEDA
- User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
ipmi message handler version 39.2
IPMI System Interface driver.
ipmi_si: Adding default-specified kcs state machine
ipmi_si: Trying default-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0
Could not set up I/O space
Trying to free nonexistent resource <0000000000000ca2-0000000000000ca2>
Trying to free nonexistent resource <0000000000000ca3-0000000000000ca3>
ipmi_si: Adding default-specified smic state machine
ipmi_si: Trying default-specified smic state machine at i/o address 0xca9, slave address 0x0, irq 0
Could not set up I/O space
Trying to free nonexistent resource <0000000000000ca9-0000000000000ca9>
Trying to free nonexistent resource <0000000000000caa-0000000000000caa>
Trying to free nonexistent resource <0000000000000cab-0000000000000cab>
ipmi_si: Adding default-specified bt state machine
ipmi_si: Trying default-specified bt state machine at i/o address 0xe4, slave address 0x0, irq 0
Could not set up I/O space
Trying to free nonexistent resource <00000000000000e4-00000000000000e4>
Trying to free nonexistent resource <00000000000000e5-00000000000000e5>
Trying to free nonexistent resource <00000000000000e6-00000000000000e6>
ipmi_si: Unable to find any System Interface(s)
alloc irq_desc for 318 on node 0
alloc kstat_irqs on node 0
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
crash memory driver: version 1.1
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input0
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
PNP: No PS/2 controller found. Probing ports directly.
mice: PS/2 mouse device common for all mice
rtc_cmos: probe of rtc_cmos failed with error -16
cpuidle: using governor ladder
cpuidle: using governor menu
EFI Variables Facility v0.08 2004-May-17
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
XENBUS: Device with no driver: device/vbd/2049
XENBUS: Device with no driver: device/vbd/2064
XENBUS: Device with no driver: device/vbd/2080
XENBUS: Device with no driver: device/vif/0
XENBUS: Device with no driver: device/console/0
drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
Initalizing network drop monitor service
Freeing unused kernel memory: 1264k freed
Write protecting the kernel read-only data: 10240k
Freeing unused kernel memory: 904k freed
Freeing unused kernel memory: 1672k freed
dracut: dracut-004-303.el6
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com
udev: starting version 147
dracut: Starting plymouth daemon
xlblk_init: register_blkdev major: 202
alloc irq_desc for 317 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 316 on node 0
alloc kstat_irqs on node 0
blkfront: xvde1: barriers disabled
alloc irq_desc for 315 on node 0
alloc kstat_irqs on node 0
blkfront: xvdf: barriers disabled
xvdf: unknown partition table
blkfront: xvdg: barriers disabled
xvdg: unknown partition table
md: bind<xvdf>
md: bind<xvdg>
md: raid0 personality registered for level 0
bio: create slab <bio-1> at 1
md/raid0:md127: md_size is 639940608 sectors.
md: RAID0 configuration for md127 - 1 zone
md: zone0=[xvdf/xvdg]
zone-offset= 0KB, device-offset= 0KB, size= 319970304KB
md127: detected capacity change from 0 to 327649591296
md127: unknown partition table
EXT4-fs (xvde1): mounted filesystem with ordered data mode. Opts:
dracut: Mounted root filesystem /dev/xvde1
SELinux: Disabled at runtime.
SELinux: Unregistering netfilter hooks
type=1404 audit(1406791298.748:2): selinux=0 auid=4294967295 ses=4294967295
dracut:
dracut: Switching root
readahead-collector: starting
udev: starting version 147
Initialising Xen virtual ethernet driver.
alloc irq_desc for 314 on node 0
alloc kstat_irqs on node 0
microcode: CPU0 sig=0x306e4, pf=0x1, revision=0x415
platform microcode: firmware: requesting intel-ucode/06-3e-04
microcode: CPU1 sig=0x306e4, pf=0x1, revision=0x415
platform microcode: firmware: requesting intel-ucode/06-3e-04
microcode: CPU2 sig=0x306e4, pf=0x1, revision=0x415
platform microcode: firmware: requesting intel-ucode/06-3e-04
microcode: CPU3 sig=0x306e4, pf=0x1, revision=0x415
platform microcode: firmware: requesting intel-ucode/06-3e-04
microcode: CPU4 sig=0x306e4, pf=0x1, revision=0x415
platform microcode: firmware: requesting intel-ucode/06-3e-04
microcode: CPU5 sig=0x306e4, pf=0x1, revision=0x415
platform microcode: firmware: requesting intel-ucode/06-3e-04
microcode: CPU6 sig=0x306e4, pf=0x1, revision=0x415
platform microcode: firmware: requesting intel-ucode/06-3e-04
microcode: CPU7 sig=0x306e4, pf=0x1, revision=0x415
platform microcode: firmware: requesting intel-ucode/06-3e-04
microcode: CPU8 sig=0x306e4, pf=0x1, revision=0x415
platform microcode: firmware: requesting intel-ucode/06-3e-04
microcode: CPU9 sig=0x306e4, pf=0x1, revision=0x415
platform microcode: firmware: requesting intel-ucode/06-3e-04
microcode: CPU10 sig=0x306e4, pf=0x1, revision=0x415
platform microcode: firmware: requesting intel-ucode/06-3e-04
microcode: CPU11 sig=0x306e4, pf=0x1, revision=0x415
platform microcode: firmware: requesting intel-ucode/06-3e-04
microcode: CPU12 sig=0x306e4, pf=0x1, revision=0x415
platform microcode: firmware: requesting intel-ucode/06-3e-04
microcode: CPU13 sig=0x306e4, pf=0x1, revision=0x415
platform microcode: firmware: requesting intel-ucode/06-3e-04
microcode: CPU14 sig=0x306e4, pf=0x1, revision=0x415
platform microcode: firmware: requesting intel-ucode/06-3e-04
microcode: CPU15 sig=0x306e4, pf=0x1, revision=0x415
platform microcode: firmware: requesting intel-ucode/06-3e-04
Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
EXT4-fs (md127): mounted filesystem with ordered data mode. Opts:
readahead-disable-service: delaying service auditd
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
ip6_tables: (C) 2000-2006 Netfilter Core Team
nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
eth0: no IPv6 routers present
readahead-collector: starting delayed service auditd
readahead-collector: sorting
readahead-collector: finished
If you have a PANIC vertica should had printed a back trace of the issue in the ErrorReport.txt file that you find under the catalog directory.
Please check there what caused the PANIC, if it is an statement, please try to run it again to see if it is reproducible.
Please also print the back trace here and tell us what Version of Vertica you are using.
Thanks,
Eugenia
Thanks for helping with this. Attached are 3 traces that caused the PANIC subsequently.
It looks like all the 3 crashed were generated by 2 very similar queries (cross join on a table).
Tried to run both queries multiple times just now but couldnt reproduce the behavior.
Both queries ran usually in 3 minutes. Please let me know how we can prevent this happening or if I can provide anything else.
Thank you for your help,
Tibor
BEGIN BACKTRACE
Vertica Backtrace at Sun Aug 3 22:26:53 2014
-------------------------
Vertica Analytic Database v7.0.1-0 $BrandId$
vertica(v7.0.1-0) built by release@build2.verticacorp.com from releases/VER_7_0_RELEASE_BUILD_1_0_20140212@130255 on 'Wed Feb 12 19:00:56 America/New_York 2014' $BuildId$
00400000-0498a000 r-xp 00000000 ca:41 392074 /opt/vertica/bin/vertica
04b89000-04e15000 rw-p 04589000 ca:41 392074 /opt/vertica/bin/vertica
04e15000-04f25000 rw-p 00000000 00:00 0
050c3000-054f7000 rw-p 00000000 00:00 0 [heap]
3e79a00000-3e79a20000 r-xp 00000000 ca:41 4585 /lib64/ld-2.12.so
3e79c1f000-3e79c20000 r--p 0001f000 ca:41 4585 /lib64/ld-2.12.so
3e79c20000-3e79c21000 rw-p 00020000 ca:41 4585 /lib64/ld-2.12.so
3e79c21000-3e79c22000 rw-p 00000000 00:00 0
3e79e00000-3e79f8b000 r-xp 00000000 ca:41 4702 /lib64/libc-2.12.so
3e79f8b000-3e7a18a000 ---p 0018b000 ca:41 4702 /lib64/libc-2.12.so
3e7a18a000-3e7a18e000 r--p 0018a000 ca:41 4702 /lib64/libc-2.12.so
3e7a18e000-3e7a18f000 rw-p 0018e000 ca:41 4702 /lib64/libc-2.12.so
3e7a18f000-3e7a194000 rw-p 00000000 00:00 0
3e7a200000-3e7a202000 r-xp 00000000 ca:41 6333 /lib64/libdl-2.12.so
3e7a202000-3e7a402000 ---p 00002000 ca:41 6333 /lib64/libdl-2.12.so
3e7a402000-3e7a403000 r--p 00002000 ca:41 6333 /lib64/libdl-2.12.so
3e7a403000-3e7a404000 rw-p 00003000 ca:41 6333 /lib64/libdl-2.12.so
3e7a600000-3e7a617000 r-xp 00000000 ca:41 33708 /lib64/libpthread-2.12.so
3e7a617000-3e7a817000 ---p 00017000 ca:41 33708 /lib64/libpthread-2.12.so
3e7a817000-3e7a818000 r--p 00017000 ca:41 33708 /lib64/libpthread-2.12.so
3e7a818000-3e7a819000 rw-p 00018000 ca:41 33708 /lib64/libpthread-2.12.so
3e7a819000-3e7a81d000 rw-p 00000000 00:00 0
3e7b200000-3e7b207000 r-xp 00000000 ca:41 33714 /lib64/librt-2.12.so
3e7b207000-3e7b406000 ---p 00007000 ca:41 33714 /lib64/librt-2.12.so
3e7b406000-3e7b407000 r--p 00006000 ca:41 33714 /lib64/librt-2.12.so
3e7b407000-3e7b408000 rw-p 00007000 ca:41 33714 /lib64/librt-2.12.so
3e7b600000-3e7b622000 r-xp 00000000 ca:41 12401 /lib64/libncurses.so.5.7
3e7b622000-3e7b821000 ---p 00022000 ca:41 12401 /lib64/libncurses.so.5.7
3e7b821000-3e7b822000 rw-p 00021000 ca:41 12401 /lib64/libncurses.so.5.7
3e7c200000-3e7c21d000 r-xp 00000000 ca:41 3740 /lib64/libtinfo.so.5.7
3e7c21d000-3e7c41d000 ---p 0001d000 ca:41 3740 /lib64/libtinfo.so.5.7
3e7c41d000-3e7c421000 rw-p 0001d000 ca:41 3740 /lib64/libtinfo.so.5.7
3e7d600000-3e7d616000 r-xp 00000000 ca:41 33711 /lib64/libgcc_s-4.4.7-20120601.so.1
3e7d616000-3e7d815000 ---p 00016000 ca:41 33711 /lib64/libgcc_s-4.4.7-20120601.so.1
3e7d815000-3e7d816000 rw-p 00015000 ca:41 33711 /lib64/libgcc_s-4.4.7-20120601.so.1
7f35f5735000-7f35fb5c6000 r--p 00000000 ca:41 8176 /usr/lib/locale/locale-archive
7f35fb5c6000-7f35fc50b000 r--s 00000000 ca:41 400203 /opt/vertica/share/icu/icudt42l.dat
7f35fc50b000-7f360c66b000 rw-p 00000000 00:00 0
7f360c66b000-7f360ca7f000 rw-p 00000000 00:00 0
7f360ca7f000-7f360ca8b000 r-xp 00000000 ca:41 3449 /lib64/libnss_files-2.12.so
7f360ca8b000-7f360cc8b000 ---p 0000c000 ca:41 3449 /lib64/libnss_files-2.12.so
7f360cc8b000-7f360cc8c000 r--p 0000c000 ca:41 3449 /lib64/libnss_files-2.12.so
7f360cc8c000-7f360cc8d000 rw-p 0000d000 ca:41 3449 /lib64/libnss_files-2.12.so
7f360cc8d000-7f360cc92000 rw-p 00000000 00:00 0
7f360cc95000-7f360cca2000 rw-p 00000000 00:00 0
7fffdcc87000-7fffdcc9f000 rw-p 00000000 00:00 0 [stack]
7fffdcdff000-7fffdce00000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Backtrace Generated by Error
Signal: [0x000000000000000b] PID: [0x0000000000000803] PC: [0x0000000001699184] FP: [0x00007f3469e5a460] SIGSEGV: SEGV_ACCERR SI_ADDR : [0x00007f34839681b8]
/opt/vertica/bin/vertica(_ZN6Basics9Backtrace11DoBacktraceEiiPvS1_+0x8cc)[0x33a011e]
/opt/vertica/bin/vertica(_ZN6Basics20GlobalSignalHandlers14logFatalSignalEiPvS1_+0xc7)[0x341f305]
/opt/vertica/bin/vertica[0x341f8f3]
/lib64/libc.so.6[0x3e79e329a0]
/opt/vertica/bin/vertica(_ZN2EE11GroupByHash3runEv+0xf9e)[0x1699184]
/opt/vertica/bin/vertica[0x15ef298]
/lib64/libc.so.6[0x3e79e43bf0]
END BACKTRACE
THREAD CONTEXT
Thread type: EE Internal Command Queue Thread
Request: INSERT INTO dev.brand_household_summary SELECT a.brand_id AS anchor_brand_id, b.brand_id, count(DISTINCT b.user_id) AS user_count FROM ((dev.f_household_brand a JOIN dev.f_household_brand b ON ((a.user_id = b.user_id))) JOIN dev.top_top_brands ttb ON ((b.brand_id = ttb.brand_id))) GROUP BY a.brand_id, b.brand_id
68: Send
0: FifoBuffer
4: Router
3: ValExpr
5: Copy
FAULT => 9: NewEENode
10) ExprEval depth=(0) parent=0, peer#0; outTup nCol=4,nkey=0,inlSz=160,fixSz=32
11) ParallelMerge (1) 10, #0; nInputs=3; out 3,0,152,24
12) GroupByHash (2) 11, #0; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
13) ParallelUnion (3) 12, #0; out 3,0,152,24
14) NetworkRecv (4) 13, #0; CRPstate=2; out 3,0,152,24
15) NetworkSend (5) 14, #0; out 3,0,152,24
16) ParallelUnion (6) 15, #0; nInputs=3; out 3,0,152,24
17) GroupByHash (7) 16, #0; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
18) ParallelUnion (8) 17, #0; nInputs=3; out 3,0,152,24
19) GroupByPipe (9) 18, #0; out 3,0,152,24
20) GroupByHash (10) 19, #0; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
21) ParallelUnion (11) 20, #0; out 3,0,152,24
22) NetworkRecv (12) 21, #0; CRPstate=2; out 3,0,152,24
23) NetworkSend (13) 22, #0; out 3,0,152,24
24) ParallelUnion (14) 23, #0; nInputs=3; out 3,0,152,24
25) GroupByHash (15) 24, #0; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
26) StorageUnion (16) 25, #0; out 3,0,152,24
27) GroupByPipe (17) 26, #0; out 3,0,152,24
28) Join (18) 27, #0; nInputs=2; CRPstate=2; out 3,0,152,24
29) Scan (19) 28, #0; out 1,0,8,8
30) NetworkRecv (19) 28, #0; CRPstate=4; out 2,0,80,16
31) NetworkSend (20) 30, #0; out 2,0,80,16
32) GroupByHash (21) 31, #0; CRPstate=4;hash sca=16; kvlen=(16,0) out 2,0,80,16
33) Join (22) 32, #0; nInputs=2; CRPstate=4; out 2,0,80,16
34) NetworkRecv (23) 33, #0; CRPstate=4; out 2,0,80,16
35) NetworkSend (24) 34, #0; out 2,0,80,16
36) GroupByPipe (25) 35, #0; out 2,0,80,16
37) StorageUnion (26) 36, #0; out 2,0,80,16
38) GroupByPipe (27) 37, #0; out 2,0,80,16
39) Scan (28) 38, #0; out 2,0,80,16
40) GroupByPipe (23) 33, #0; out 1,0,72,8
41) StorageUnion (24) 40, #0; out 1,0,72,8
42) GroupByPipe (25) 41, #0; out 1,0,72,8
43) Scan (26) 42, #0; out 1,0,72,8
44) GroupByHash (15) 24, #1; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
45) StorageUnion (16) 44, #1; out 3,0,152,24
(PPFAULT) => GroupByHash (id=46) (15) 24, #2; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
47) StorageUnion (16) 46, #2; out 3,0,152,24
48) GroupByPipe (9) 18, #1; out 3,0,152,24
49) GroupByHash (10) 48, #1; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
50) ParallelUnion (11) 49, #1; out 3,0,152,24
51) GroupByPipe (9) 18, #2; out 3,0,152,24
52) GroupByHash (10) 51, #2; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
53) ParallelUnion (11) 52, #2; out 3,0,152,24
54) GroupByHash (7) 16, #1; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
55) ParallelUnion (8) 54, #1; out 3,0,152,24
56) GroupByHash (7) 16, #2; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
57) ParallelUnion (8) 56, #2; out 3,0,152,24
58) GroupByHash (2) 11, #1; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
59) ParallelUnion (3) 58, #1; out 3,0,152,24
60) GroupByHash (2) 11, #2; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
61) ParallelUnion (3) 60, #2; out 3,0,152,24
Transaction: [0x00a000000026b856]
END THREAD CONTEXT
BEGIN BACKTRACE
Vertica Backtrace at Sun Aug 3 23:55:27 2014
-------------------------
Vertica Analytic Database v7.0.1-0 $BrandId$
vertica(v7.0.1-0) built by release@build2.verticacorp.com from releases/VER_7_0_RELEASE_BUILD_1_0_20140212@130255 on 'Wed Feb 12 19:00:56 America/New_York 2014' $BuildId$
00400000-0498a000 r-xp 00000000 ca:41 392074 /opt/vertica/bin/vertica
04b89000-04e15000 rw-p 04589000 ca:41 392074 /opt/vertica/bin/vertica
04e15000-04f25000 rw-p 00000000 00:00 0
05850000-05c84000 rw-p 00000000 00:00 0 [heap]
3e79a00000-3e79a20000 r-xp 00000000 ca:41 4585 /lib64/ld-2.12.so
3e79c1f000-3e79c20000 r--p 0001f000 ca:41 4585 /lib64/ld-2.12.so
3e79c20000-3e79c21000 rw-p 00020000 ca:41 4585 /lib64/ld-2.12.so
3e79c21000-3e79c22000 rw-p 00000000 00:00 0
3e79e00000-3e79f8b000 r-xp 00000000 ca:41 4702 /lib64/libc-2.12.so
3e79f8b000-3e7a18a000 ---p 0018b000 ca:41 4702 /lib64/libc-2.12.so
3e7a18a000-3e7a18e000 r--p 0018a000 ca:41 4702 /lib64/libc-2.12.so
3e7a18e000-3e7a18f000 rw-p 0018e000 ca:41 4702 /lib64/libc-2.12.so
3e7a18f000-3e7a194000 rw-p 00000000 00:00 0
3e7a200000-3e7a202000 r-xp 00000000 ca:41 6333 /lib64/libdl-2.12.so
3e7a202000-3e7a402000 ---p 00002000 ca:41 6333 /lib64/libdl-2.12.so
3e7a402000-3e7a403000 r--p 00002000 ca:41 6333 /lib64/libdl-2.12.so
3e7a403000-3e7a404000 rw-p 00003000 ca:41 6333 /lib64/libdl-2.12.so
3e7a600000-3e7a617000 r-xp 00000000 ca:41 33708 /lib64/libpthread-2.12.so
3e7a617000-3e7a817000 ---p 00017000 ca:41 33708 /lib64/libpthread-2.12.so
3e7a817000-3e7a818000 r--p 00017000 ca:41 33708 /lib64/libpthread-2.12.so
3e7a818000-3e7a819000 rw-p 00018000 ca:41 33708 /lib64/libpthread-2.12.so
3e7a819000-3e7a81d000 rw-p 00000000 00:00 0
3e7b200000-3e7b207000 r-xp 00000000 ca:41 33714 /lib64/librt-2.12.so
3e7b207000-3e7b406000 ---p 00007000 ca:41 33714 /lib64/librt-2.12.so
3e7b406000-3e7b407000 r--p 00006000 ca:41 33714 /lib64/librt-2.12.so
3e7b407000-3e7b408000 rw-p 00007000 ca:41 33714 /lib64/librt-2.12.so
3e7b600000-3e7b622000 r-xp 00000000 ca:41 12401 /lib64/libncurses.so.5.7
3e7b622000-3e7b821000 ---p 00022000 ca:41 12401 /lib64/libncurses.so.5.7
3e7b821000-3e7b822000 rw-p 00021000 ca:41 12401 /lib64/libncurses.so.5.7
3e7c200000-3e7c21d000 r-xp 00000000 ca:41 3740 /lib64/libtinfo.so.5.7
3e7c21d000-3e7c41d000 ---p 0001d000 ca:41 3740 /lib64/libtinfo.so.5.7
3e7c41d000-3e7c421000 rw-p 0001d000 ca:41 3740 /lib64/libtinfo.so.5.7
3e7d600000-3e7d616000 r-xp 00000000 ca:41 33711 /lib64/libgcc_s-4.4.7-20120601.so.1
3e7d616000-3e7d815000 ---p 00016000 ca:41 33711 /lib64/libgcc_s-4.4.7-20120601.so.1
3e7d815000-3e7d816000 rw-p 00015000 ca:41 33711 /lib64/libgcc_s-4.4.7-20120601.so.1
7f1284114000-7f1289fa5000 r--p 00000000 ca:41 8176 /usr/lib/locale/locale-archive
7f1289fa5000-7f128aeea000 r--s 00000000 ca:41 400203 /opt/vertica/share/icu/icudt42l.dat
7f128aeea000-7f129b04a000 rw-p 00000000 00:00 0
7f129b04a000-7f129b45e000 rw-p 00000000 00:00 0
7f129b45e000-7f129b46a000 r-xp 00000000 ca:41 3449 /lib64/libnss_files-2.12.so
7f129b46a000-7f129b66a000 ---p 0000c000 ca:41 3449 /lib64/libnss_files-2.12.so
7f129b66a000-7f129b66b000 r--p 0000c000 ca:41 3449 /lib64/libnss_files-2.12.so
7f129b66b000-7f129b66c000 rw-p 0000d000 ca:41 3449 /lib64/libnss_files-2.12.so
7f129b66c000-7f129b671000 rw-p 00000000 00:00 0
7f129b674000-7f129b681000 rw-p 00000000 00:00 0
7fffc746f000-7fffc7488000 rw-p 00000000 00:00 0 [stack]
7fffc74e9000-7fffc74ea000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Backtrace Generated by Error
Signal: [0x000000000000000b] PID: [0x0000000000003986] PC: [0x0000000001699184] FP: [0x00007f10cd659460] SIGSEGV: SEGV_ACCERR SI_ADDR : [0x00007f109e3721b8]
/opt/vertica/bin/vertica(_ZN6Basics9Backtrace11DoBacktraceEiiPvS1_+0x8cc)[0x33a011e]
/opt/vertica/bin/vertica(_ZN6Basics20GlobalSignalHandlers14logFatalSignalEiPvS1_+0xc7)[0x341f305]
/opt/vertica/bin/vertica[0x341f8f3]
/lib64/libc.so.6[0x3e79e329a0]
/opt/vertica/bin/vertica(_ZN2EE11GroupByHash3runEv+0xf9e)[0x1699184]
/opt/vertica/bin/vertica[0x15ef298]
/lib64/libc.so.6[0x3e79e43bf0]
END BACKTRACE
THREAD CONTEXT
Thread type: EE Internal Command Queue Thread
Request: INSERT INTO dev.brand_household_summary SELECT a.brand_id AS anchor_brand_id, b.brand_id, count(DISTINCT b.user_id) AS user_count FROM ((dev.f_household_brand a JOIN dev.f_household_brand b ON ((a.user_id = b.user_id))) JOIN dev.top_top_brands ttb ON ((b.brand_id = ttb.brand_id))) GROUP BY a.brand_id, b.brand_id
67: Send
0: FifoBuffer
4: Router
3: ValExpr
5: Copy
FAULT => 9: NewEENode
10) ExprEval depth=(0) parent=0, peer#0; outTup nCol=4,nkey=0,inlSz=160,fixSz=32
11) ParallelMerge (1) 10, #0; nInputs=3; out 3,0,152,24
12) GroupByHash (2) 11, #0; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
13) ParallelUnion (3) 12, #0; out 3,0,152,24
14) NetworkRecv (4) 13, #0; CRPstate=2; out 3,0,152,24
15) NetworkSend (5) 14, #0; out 3,0,152,24
16) ParallelUnion (6) 15, #0; nInputs=3; out 3,0,152,24
17) GroupByHash (7) 16, #0; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
18) ParallelUnion (8) 17, #0; nInputs=3; out 3,0,152,24
19) GroupByPipe (9) 18, #0; out 3,0,152,24
20) GroupByHash (10) 19, #0; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
21) ParallelUnion (11) 20, #0; out 3,0,152,24
22) NetworkRecv (12) 21, #0; CRPstate=2; out 3,0,152,24
23) NetworkSend (13) 22, #0; out 3,0,152,24
24) ParallelUnion (14) 23, #0; nInputs=3; out 3,0,152,24
25) GroupByHash (15) 24, #0; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
26) StorageUnion (16) 25, #0; out 3,0,152,24
27) GroupByPipe (17) 26, #0; out 3,0,152,24
28) Join (18) 27, #0; nInputs=2; CRPstate=2; out 3,0,152,24
29) Scan (19) 28, #0; out 1,0,8,8
30) NetworkRecv (19) 28, #0; CRPstate=4; out 2,0,80,16
31) NetworkSend (20) 30, #0; out 2,0,80,16
32) GroupByHash (21) 31, #0; CRPstate=4;hash sca=16; kvlen=(16,0) out 2,0,80,16
33) Join (22) 32, #0; nInputs=2; CRPstate=4; out 2,0,80,16
34) NetworkRecv (23) 33, #0; CRPstate=4; out 2,0,80,16
35) NetworkSend (24) 34, #0; out 2,0,80,16
36) GroupByPipe (25) 35, #0; out 2,0,80,16
37) StorageUnion (26) 36, #0; out 2,0,80,16
38) GroupByPipe (27) 37, #0; out 2,0,80,16
39) Scan (28) 38, #0; out 2,0,80,16
40) GroupByPipe (23) 33, #0; out 1,0,72,8
41) StorageUnion (24) 40, #0; out 1,0,72,8
42) GroupByPipe (25) 41, #0; out 1,0,72,8
43) Scan (26) 42, #0; out 1,0,72,8
44) GroupByHash (15) 24, #1; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
45) StorageUnion (16) 44, #1; out 3,0,152,24
(PPFAULT) => GroupByHash (id=46) (15) 24, #2; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
47) StorageUnion (16) 46, #2; out 3,0,152,24
48) GroupByPipe (9) 18, #1; out 3,0,152,24
49) GroupByHash (10) 48, #1; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
50) ParallelUnion (11) 49, #1; out 3,0,152,24
51) GroupByPipe (9) 18, #2; out 3,0,152,24
52) GroupByHash (10) 51, #2; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
53) ParallelUnion (11) 52, #2; out 3,0,152,24
54) GroupByHash (7) 16, #1; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
55) ParallelUnion (8) 54, #1; out 3,0,152,24
56) GroupByHash (7) 16, #2; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
57) ParallelUnion (8) 56, #2; out 3,0,152,24
58) GroupByHash (2) 11, #1; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
59) ParallelUnion (3) 58, #1; out 3,0,152,24
60) GroupByHash (2) 11, #2; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
61) ParallelUnion (3) 60, #2; out 3,0,152,24
Transaction: [0x00a000000026bcee]
END THREAD CONTEXT
BEGIN BACKTRACE
Vertica Backtrace at Mon Aug 4 01:51:17 2014
-------------------------
Vertica Analytic Database v7.0.1-0 $BrandId$
vertica(v7.0.1-0) built by release@build2.verticacorp.com from releases/VER_7_0_RELEASE_BUILD_1_0_20140212@130255 on 'Wed Feb 12 19:00:56 America/New_York 2014' $BuildId$
00400000-0498a000 r-xp 00000000 ca:41 392074 /opt/vertica/bin/vertica
04b89000-04e15000 rw-p 04589000 ca:41 392074 /opt/vertica/bin/vertica
04e15000-04f25000 rw-p 00000000 00:00 0
057ab000-05bdf000 rw-p 00000000 00:00 0 [heap]
3e79a00000-3e79a20000 r-xp 00000000 ca:41 4585 /lib64/ld-2.12.so
3e79c1f000-3e79c20000 r--p 0001f000 ca:41 4585 /lib64/ld-2.12.so
3e79c20000-3e79c21000 rw-p 00020000 ca:41 4585 /lib64/ld-2.12.so
3e79c21000-3e79c22000 rw-p 00000000 00:00 0
3e79e00000-3e79f8b000 r-xp 00000000 ca:41 4702 /lib64/libc-2.12.so
3e79f8b000-3e7a18a000 ---p 0018b000 ca:41 4702 /lib64/libc-2.12.so
3e7a18a000-3e7a18e000 r--p 0018a000 ca:41 4702 /lib64/libc-2.12.so
3e7a18e000-3e7a18f000 rw-p 0018e000 ca:41 4702 /lib64/libc-2.12.so
3e7a18f000-3e7a194000 rw-p 00000000 00:00 0
3e7a200000-3e7a202000 r-xp 00000000 ca:41 6333 /lib64/libdl-2.12.so
3e7a202000-3e7a402000 ---p 00002000 ca:41 6333 /lib64/libdl-2.12.so
3e7a402000-3e7a403000 r--p 00002000 ca:41 6333 /lib64/libdl-2.12.so
3e7a403000-3e7a404000 rw-p 00003000 ca:41 6333 /lib64/libdl-2.12.so
3e7a600000-3e7a617000 r-xp 00000000 ca:41 33708 /lib64/libpthread-2.12.so
3e7a617000-3e7a817000 ---p 00017000 ca:41 33708 /lib64/libpthread-2.12.so
3e7a817000-3e7a818000 r--p 00017000 ca:41 33708