RES mem is higher then memory_inuse_kb
Alok_Tiwari
Vertica Customer
Best Answers
-
sahil_kumar
Vertica Employee Employee
glibc has a performance enhancement that result in glibc holding on to memory that application is not using and/or does not need.
Over time this additional memory may result in the kernel killing the application.
https://stackoverflow.com/questions/48651432/glibc-application-holding-onto-unused-memory-until-just-before-exit
https://sourceware.org/bugzilla/show_bug.cgi?id=11261
This is not considered a glibc bug, since there are tunables that allow the user to control this behavior.5 -
sahil_kumar
Vertica Employee Employee
@Alok_Tiwari request you to open a support ticket for better analysis
Regarding your above question
in ~/.bashrc on ALL nodes add the following line
$ export MALLOC_ARENA_MAX=4
log out and log back in then restart vertica database
OR if you are unable to edit ~/.bashrc, always start the database using admintools with the VERTICA_ADMINTOOLS_PASSTHROUGH variable set as follows
$ export VERTICA_ADMINTOOLS_PASSTHROUGH=MALLOC_ARENA_MAX=4 admintools -t start_db -d mydb
you can check the environment variables are set correctly using the following commands
To check if the environment variable is set correctly
$ set | grep ARENA
$ MALLOC_ARENA_MAX=4
To ensure that the environment has been applied to the process correctly (check all nodes)
$ xargs --null --max-args=1 echo < /proc/$(pgrep vertica$)/environ | grep ARENA5
Answers
This is an old discussion however our current ITOM software still says this: "You can tune the memory used by adjusting the MALLOC_ARENA_MAX environment variable. This parameter enables you to control the memory growth in Vertica. If you don't tune this parameter, over a period of time, the Vertica process may crash due to memory issues."
Can someone confirm this defect was never fixed and all current versions of Vertica still fail without this workaround?
yes. it should be fixed in latest versions. Please check the below link to ensure that you have the glibc version which has all the fixes.
https://www.vertica.com/kb/Known-Issues-in-Linux-Kernel-and-Components-Vertica-Impact/Content/SupportNotices/Known-Issues-in-Linux-Kernel-and-Components-Vertica-Impact.htm
Well, the link to the Known-Issues-in-Linux-Kernel-and-Components-Vertica-Impact page shows this bug, 11261, as "Determination: Works as expected" which is not helpful at all! Apparently, it only affected RHEL 6 and 7 not 8 or 9. Both vertica 11 and 23 support RHEL 7.x therefore I suppose every customer on RHEL 7 still needs to unconditionally apply this manual workaround?
with latest vertica versions, it is not mandatory since we perform memory trimming automatically when RES mem usage increases. For more information, please check the below link
https://docs.vertica.com/12.0.x/en/admin/managing-db/memory-usage-reporting/
Thank you Sruthi for the 12.x pointer - and I see Vertica 11 doc does not have this link therefore, I presume, the MALLOC_ARENA_MAX workaround is still required for customers installing Vertica 11.x on RHEL 7.x.
Memory trimming feature is available in 11.x as well. please find the doc link below for the same.
https://www.vertica.com/docs/11.1.x/HTML/Content/Authoring/AdministratorsGuide/ManageMemory/MemoryTrimming.htm
GREAT to hear... so... what I am hearing is that new installs of Vertica 11.x ans 12.x NEVER need this workaround any more and so I will ask our ITOM SA IEs to remove the workaround from our docs.
as far as I know that workaround is not needed because vertica does memory trimming automatically. But, if you face issues, please open a support case.