Vertica system crash
Vertica Databse is keep crashing with below error messages in /var/log/syslog file.
Could somebody help to resolve this issue? Let me know if you need more details.
We are using Vertica Community Edition and the version is 6.0.1-0
kernel: [2611787.489533] traps: vertica-export.[24371] general protection ip:7fb3e6918fcf sp:7fff097a1050 error:0 in libverticaodbc.so.6.0.1[7fb3e67b9000+13cb000]
compress-core.sh: Filesystem too full, not generating core core.python2.7.ATT-LA-MDV-1-DAF.1474216267.24371 (81% / 70%)
kernel: [2611787.550948] dump_emit: __kernel_write returned -32!
kernel: [2611787.609822] vertica-export.[24372]: segfault at 400001000 ip 00007f8cd6dbdfcf sp 00007fffa1e02550 error 4 in libverticaodbc.so.6.0.1[7f8cd6c5e000+13cb000]
compress-core.sh: Filesystem too full, not generating core core.python2.7.ATT-LA-MDV-1-DAF.1474216267.24372 (81% / 70%)
kernel: [2611787.619932] dump_emit: __kernel_write returned -32!
compress-core.sh: Filesystem too full, not generating core core.python2.7.ATT-LA-MDV-1-DAF.1474216268.24373 (81% / 70%)
kernel: [2611787.713901] dump_emit: __kernel_write returned -32!
kernel: [2611788.100006] vertica-export.[24374]: segfault at 31 ip 00007f09c99e7fcf sp 00007ffdf0753a50 error 4 in libverticaodbc.so.6.0.1[7f09c9888000+13cb000]
DAF compress-core.sh: Filesystem too full, not generating core core.python2.7.ATT-LA-MDV-1-DAF.1474216268.24374 (81% / 70%)
kernel: [2611788.109627] dump_emit: __kernel_write returned -32!
compress-core.sh: Filesystem too full, not generating core core.python2.7.ATT-LA-MDV-1-DAF.1474216268.24370 (81% / 70%)
kernel: [2611788.589361] dump_emit: __kernel_write returned -32!
Comments
From the error log msg it seems that you don`t have space left, or the temporary area is getting full.
Thanks Adrian.
To load the files into Vertica, we are using RAM as temp area for faster loading. We do have enough memory in our system.
However, I have pointed the temp area to an internal HD which has around 2Tb space. I know, there will be some performance degradataion, but I really want to stop this Vertica crash!!!
Hope this change will resolve the issue. I will update this thread in few days.
Thanks,
Sri.
You're on 6.0.1? That's ancient. You should upgrade to at least 7.2.x.
--Sharon
We already upgraded vertica on one of our lab server to 7.2.x version, last week and we were still seeing the crash happening.
We are upgrading to 8.x now and verify that it is still happening.
Upgrading won`t fix it. Find the root couse.
- do you get the error logs scrapped and you can even try to attched them here so some of us can take a look.
Otherwise you will continue with this and get frustrated with Vertica just becouse of some simple conf change.
here is the error report with backtrace.
Ok ,
Before you go ahead and upgrade again or do some Vertica changes you have to make sure you hosts are well setup.
The OOM killer is a fail safe task of your system so it makes sure that the house won't and it will sacrifice the hungries on the table, in this case Vertica is doing all the "damage" resource usage.
1 - Make sure Vertica is the only process running on that system, i mean as a main one , don`t run mysql, vertica console some samba server or any other resource intensive processes.
2 - You don`t alot of memory ! ok fine , make sure you give it swap space.
3 - ok i gave it swap but is stil swaping
!!! hmm ...have you changed the default values of the swappines ? you don`t know wat is this ? ok read this here and you will understand .
Also see the log of you system and match the node shutdown times with the error.txt file and the messesges log file "
After you change this and you node still goes down "vertica getting killed" , look at the resource pool the user who is running the queryes is using , limit that pool , put some rules for it, monitor is limits and adjust them as needed, starv the session by limiting the memory -- there are many things you can do but i recomend you look as you host first and than make changes in Vertica
After changing the Vertica temp location from memory cache to internal HD, we are not seeing any crashes.
It looks like the Vertica is crashing when memory gets full and no space to use for it's temp folder operation and it is happening in all versions of Vertica.