Vertica Node crashed
Hi,
We have 10-node Vertica cluster.
One of the node crashed and I'm not able to figure out the root cause. Anyone can help please.
Attached the vertica.log from the problematic node below.
I found in the error_messages table row with "The UDx zygote process is down, restarting it..." massage, I don't know if it related.
Thanks
...
sole/mcdb/derby -Xmx2048m -Xms1024m -XX:MaxPermSize=256m -jar /opt/vconsole/lib/webui.war
root 10801 1 10859 0 82 Jul13 ? 00:00:00 /opt/vconsole/vendor/oracle/java/jre/1.6/bin/java -Dvertica.home=/opt/vertica -Dvconsole.home=/opt/vconsole -Djava.library.path=/opt/vconsole/lib -Dderby.system.home=/opt/vconsole/mcdb/derby -Xmx2048m -Xms1024m -XX:MaxPermSize=256m -jar /opt/vconsole/lib/webui.war
root 10801 1 10860 0 82 Jul13 ? 00:04:43 /opt/vconsole/vendor/oracle/java/jre/1.6/bin/java -Dvertica.home=/opt/vertica -Dvconsole.home=/opt/vconsole -Djava.library.path=/opt/vconsole/lib -Dderby.system.home=/opt/vconsole/mcdb/derby -Xmx2048m -Xms1024m -XX:MaxPermSize=256m -jar /opt/vconsole/lib/webui.war
root 10801 1 10861 0 82 Jul13 ? 00:04:41 /opt/vconsole/vendor/oracle/java/jre/1.6/bin/java -Dvertica.home=/opt/vertica -Dvconsole.home=/opt/vconsole -Djava.library.path=/opt/vconsole/lib -Dderby.system.home=/opt/vconsole/mcdb/derby -Xmx2048m -Xms1024m -XX:MaxPermSize=256m -jar /opt/vconsole/lib/webui.war
root 10801 1 10862 0 82 Jul13 ? 00:04:42 /opt/vconsole/vendor/oracle/java/jre/1.6/bin/java -Dvertica.home=/opt/vertica -Dvconsole.home=/opt/vconsole -Djava.library.path=/opt/vconsole/lib -Dderby.system.home=/opt/vconsole/mcdb/derby -Xmx2048m -Xms1024m -XX:MaxPermSize=256m -jar /opt/vconsole/lib/webui.war
root 10801 1 10863 0 82 Jul13 ? 00:04:46 /opt/vconsole/vendor/oracle/java/jre/1.6/bin/java -Dvertica.home=/opt/vertica -Dvconsole.home=/opt/vconsole -Djava.library.path=/opt/vconsole/lib -Dderby.system.home=/opt/vconsole/mcdb/derby -Xmx2048m -Xms1024m -XX:MaxPermSize=256m -jar /opt/vconsole/lib/webui.war
root 10801 1 10864 0 82 Jul13 ? 00:04:42 /opt/vconsole/vendor/oracle/java/jre/1.6/bin/java -Dvertica.home=/opt/vertica -Dvconsole.home=/opt/vconsole -Djava.library.path=/opt/vconsole/lib -Dderby.system.home=/opt/vconsole/mcdb/derby -Xmx2048m -Xms1024m -XX:MaxPermSize=256m -jar /opt/vconsole/lib/webui.war
root 10801 1 10865 0 82 Jul13 ? 00:04:42 /opt/vconsole/vendor/oracle/java/jre/1.6/bin/java -Dvertica.home=/opt/vertica -Dvconsole.home=/opt/vconsole -Djava.library.path=/opt/vconsole/lib -Dderby.system.home=/opt/vconsole/mcdb/derby -Xmx2048m -Xms1024m -XX:MaxPermSize=256m -jar /opt/vconsole/lib/webui.war
root 10801 1 10866 0 82 Jul13 ? 00:04:40 /opt/vconsole/vendor/oracle/java/jre/1.6/bin/java -Dvertica.home=/opt/vertica -Dvconsole.home=/opt/vconsole -Djava.library.path=/opt/vconsole/lib -Dderby.system.home=/opt/vconsole/mcdb/derby -Xmx2048m -Xms1024m -XX:MaxPermSize=256m -jar /opt/vconsole/lib/webui.war
root 10
2015-09-02 07:56:55.912 Main:0x6908b10 [Main] <INFO> Handling signal: 15
2015-09-02 08:02:23.483 INFO New log
2015-09-02 08:02:23.483 unknown:0x7fddccf0a700 [Init] <INFO> Log /db/catalog/vertica/v_vertica_node0010_catalog/vertica.log opened; #1
2015-09-02 08:02:23.483 unknown:0x7fddccf0a700 [Init] <INFO> Processing command line: /opt/vertica/bin/vertica -D /db/catalog/vertica/v_vertica_node0010_catalog -C vertica -n v_vertica_node0010 -h 111.11.111.10 -p 5433 -P 4803 -Y ipv4
2015-09-02 08:02:23.483 unknown:0x7fddccf0a700 [Init] <INFO> Starting up Vertica Analytic Database v7.1.1-10
2015-09-02 08:02:23.483 unknown:0x7fddccf0a700 [Init] <INFO> Project Codename: Dragline
2015-09-02 08:02:23.483 unknown:0x7fddccf0a700 [Init] <INFO> vertica(v7.1.1-10) built by release@build2.verticacorp.com from releases/VER_7_1_RELEASE_BUILD_1_10_20150320@158323 on 'Fri Mar 20 10:26:33 America/New_York 2015' $BuildId$
2015-09-02 08:02:23.483 unknown:0x7fddccf0a700 [Init] <INFO> 64-bit Optimized Build
2015-09-02 08:02:23.483 unknown:0x7fddccf0a700 [Init] <INFO> Compiler Version: 4.1.2 20080704 (Red Hat 4.1.2-55)
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/5081: Total swap memory used: 0
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4435: Process size resident set: 28495872
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/5075: Total Memory free + cache: 150111891456
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 0 current scaling frequency: 2200000 kHz
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 1 current scaling frequency: 2200000 kHz
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 2 current scaling frequency: 2200000 kHz
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 3 current scaling frequency: 2200000 kHz
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 4 current scaling frequency: 2200000 kHz
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 5 current scaling frequency: 2200000 kHz
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 6 current scaling frequency: 2200000 kHz
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 7 current scaling frequency: 2200000 kHz
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 8 current scaling frequency: 2200000 kHz
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 9 current scaling frequency: 2200000 kHz
2015-09-02 08:02:23.485 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 10 current scaling frequency: 2200000 kHz
root 10801 1 10859 0 82 Jul13 ? 00:00:00 /opt/vconsole/vendor/oracle/java/jre/1.6/bin/java -Dvertica.home=/opt/vertica -Dvconsole.home=/opt/vconsole -Djava.library.path=/opt/vconsole/lib -Dderby.system.home=/opt/vconsole/mcdb/derby -Xmx2048m -Xms1024m -XX:MaxPermSize=256m -jar /opt/vconsole/lib/webui.war
root 10801 1 10860 0 82 Jul13 ? 00:04:43 /opt/vconsole/vendor/oracle/java/jre/1.6/bin/java -Dvertica.home=/opt/vertica -Dvconsole.home=/opt/vconsole -Djava.library.path=/opt/vconsole/lib -Dderby.system.home=/opt/vconsole/mcdb/derby -Xmx2048m -Xms1024m -XX:MaxPermSize=256m -jar /opt/vconsole/lib/webui.war
root 10801 1 10861 0 82 Jul13 ? 00:04:41 /opt/vconsole/vendor/oracle/java/jre/1.6/bin/java -Dvertica.home=/opt/vertica -Dvconsole.home=/opt/vconsole -Djava.library.path=/opt/vconsole/lib -Dderby.system.home=/opt/vconsole/mcdb/derby -Xmx2048m -Xms1024m -XX:MaxPermSize=256m -jar /opt/vconsole/lib/webui.war
root 10801 1 10862 0 82 Jul13 ? 00:04:42 /opt/vconsole/vendor/oracle/java/jre/1.6/bin/java -Dvertica.home=/opt/vertica -Dvconsole.home=/opt/vconsole -Djava.library.path=/opt/vconsole/lib -Dderby.system.home=/opt/vconsole/mcdb/derby -Xmx2048m -Xms1024m -XX:MaxPermSize=256m -jar /opt/vconsole/lib/webui.war
root 10801 1 10863 0 82 Jul13 ? 00:04:46 /opt/vconsole/vendor/oracle/java/jre/1.6/bin/java -Dvertica.home=/opt/vertica -Dvconsole.home=/opt/vconsole -Djava.library.path=/opt/vconsole/lib -Dderby.system.home=/opt/vconsole/mcdb/derby -Xmx2048m -Xms1024m -XX:MaxPermSize=256m -jar /opt/vconsole/lib/webui.war
root 10801 1 10864 0 82 Jul13 ? 00:04:42 /opt/vconsole/vendor/oracle/java/jre/1.6/bin/java -Dvertica.home=/opt/vertica -Dvconsole.home=/opt/vconsole -Djava.library.path=/opt/vconsole/lib -Dderby.system.home=/opt/vconsole/mcdb/derby -Xmx2048m -Xms1024m -XX:MaxPermSize=256m -jar /opt/vconsole/lib/webui.war
root 10801 1 10865 0 82 Jul13 ? 00:04:42 /opt/vconsole/vendor/oracle/java/jre/1.6/bin/java -Dvertica.home=/opt/vertica -Dvconsole.home=/opt/vconsole -Djava.library.path=/opt/vconsole/lib -Dderby.system.home=/opt/vconsole/mcdb/derby -Xmx2048m -Xms1024m -XX:MaxPermSize=256m -jar /opt/vconsole/lib/webui.war
root 10801 1 10866 0 82 Jul13 ? 00:04:40 /opt/vconsole/vendor/oracle/java/jre/1.6/bin/java -Dvertica.home=/opt/vertica -Dvconsole.home=/opt/vconsole -Djava.library.path=/opt/vconsole/lib -Dderby.system.home=/opt/vconsole/mcdb/derby -Xmx2048m -Xms1024m -XX:MaxPermSize=256m -jar /opt/vconsole/lib/webui.war
root 10
2015-09-02 07:56:55.912 Main:0x6908b10 [Main] <INFO> Handling signal: 15
2015-09-02 08:02:23.483 INFO New log
2015-09-02 08:02:23.483 unknown:0x7fddccf0a700 [Init] <INFO> Log /db/catalog/vertica/v_vertica_node0010_catalog/vertica.log opened; #1
2015-09-02 08:02:23.483 unknown:0x7fddccf0a700 [Init] <INFO> Processing command line: /opt/vertica/bin/vertica -D /db/catalog/vertica/v_vertica_node0010_catalog -C vertica -n v_vertica_node0010 -h 111.11.111.10 -p 5433 -P 4803 -Y ipv4
2015-09-02 08:02:23.483 unknown:0x7fddccf0a700 [Init] <INFO> Starting up Vertica Analytic Database v7.1.1-10
2015-09-02 08:02:23.483 unknown:0x7fddccf0a700 [Init] <INFO> Project Codename: Dragline
2015-09-02 08:02:23.483 unknown:0x7fddccf0a700 [Init] <INFO> vertica(v7.1.1-10) built by release@build2.verticacorp.com from releases/VER_7_1_RELEASE_BUILD_1_10_20150320@158323 on 'Fri Mar 20 10:26:33 America/New_York 2015' $BuildId$
2015-09-02 08:02:23.483 unknown:0x7fddccf0a700 [Init] <INFO> 64-bit Optimized Build
2015-09-02 08:02:23.483 unknown:0x7fddccf0a700 [Init] <INFO> Compiler Version: 4.1.2 20080704 (Red Hat 4.1.2-55)
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/5081: Total swap memory used: 0
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4435: Process size resident set: 28495872
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/5075: Total Memory free + cache: 150111891456
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 0 current scaling frequency: 2200000 kHz
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 1 current scaling frequency: 2200000 kHz
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 2 current scaling frequency: 2200000 kHz
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 3 current scaling frequency: 2200000 kHz
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 4 current scaling frequency: 2200000 kHz
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 5 current scaling frequency: 2200000 kHz
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 6 current scaling frequency: 2200000 kHz
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 7 current scaling frequency: 2200000 kHz
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 8 current scaling frequency: 2200000 kHz
2015-09-02 08:02:23.484 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 9 current scaling frequency: 2200000 kHz
2015-09-02 08:02:23.485 unknown:0x7fddccf0a700 <LOG> @[initializing]: 00000/4441: Processor 10 current scaling frequency: 2200000 kHz
...
0