glibc memory bloat
skeswani
Employee
It appears that the memory of my vertica process increases over time and the process eventually crashes with an OOM (out-of-memory) kill. Can you help me
0
It appears that the memory of my vertica process increases over time and the process eventually crashes with an OOM (out-of-memory) kill. Can you help me
Answers
glibc has a performance enhancement that result in glibc holding on to memory that application is not using and/or does not need.
Over time this additional memory may result in the kernel killing the application (i.e. vertica) with a OOM kill.
This is more pronounced with multi-threaded application and can result in frequent OOM (out-of-memory) crashes
see
https://stackoverflow.com/questions/48651432/glibc-application-holding-onto-unused-memory-until-just-before-exit
https://sourceware.org/bugzilla/show_bug.cgi?id=11261
This is not considered a glibc bug, since there are tunables that allow the user to control this behavior.
There are a few things a vertica admin can do to prevent OOMs
Use the tunable that glibc provides to control memory growth
in ~/.bashrc on ALL nodes add the following line
log out and log back in then restart vertica database
OR if you are unable to edit ~/.bashrc, always start the database using admintools with the VERTICA_ADMINTOOLS_PASSTHROUGH variable set as follows
you can check the environment variables are set correctly using the following commands
To check if the environment variable is set correctly
To ensure that the environment has been applied to the process correctly (check all nodes)
Run a UDx (User Defined Extension) that trims the memory periodically
With g++ version 4.8 or less, compile the following User Defined Extenstion
( attached : udx_malloc_trim.txt, since cpp files are sometimes blocked. rename this file to udx_malloc_trim.cpp before you compile it)
After you have compiled the extension, install it as shown below
Connect vsql to THE NODE where Vertica RSS memory is large and run MallocTrim(1). This will trim the vertica memory usage and prevent an OOM
Here is simple shell script to check for memory use and run MallocTrim when the memory usage exceeds 11GB
Although, I use 11 GB in this example, you should use 85% of memory on the machine as a trigger to run the trim.
(i.e. if your hosts has 100GB RAM, then you should call MallocTrim when the Vertica RSS size exceeds 85GB)
Note: This only applies to vertica version 9.1 and less. Starting with vertica version 9.2 , this is automatically done by the database and no admin intervention is needed