glibc memory bloat

Answers

• Options

  skeswani - Select Field - Employee

  October 2019 edited October 2019

  glibc has a performance enhancement that result in glibc holding on to memory that application is not using and/or does not need.
  Over time this additional memory may result in the kernel killing the application (i.e. vertica) with a OOM kill.
  This is more pronounced with multi-threaded application and can result in frequent OOM (out-of-memory) crashes
  see
  https://stackoverflow.com/questions/48651432/glibc-application-holding-onto-unused-memory-until-just-before-exit
  https://sourceware.org/bugzilla/show_bug.cgi?id=11261

  This is not considered a glibc bug, since there are tunables that allow the user to control this behavior.
  There are a few things a vertica admin can do to prevent OOMs

  Use the tunable that glibc provides to control memory growth

  in ~/.bashrc on ALL nodes add the following line

  $ export MALLOC_ARENA_MAX=4

  log out and log back in then restart vertica database
  OR if you are unable to edit ~/.bashrc, always start the database using admintools with the VERTICA_ADMINTOOLS_PASSTHROUGH variable set as follows

  $ export VERTICA_ADMINTOOLS_PASSTHROUGH=MALLOC_ARENA_MAX=4 admintools -t start_db -d mydb

  you can check the environment variables are set correctly using the following commands
  To check if the environment variable is set correctly

  $ set | grep ARENA
  $ MALLOC_ARENA_MAX=4

  To ensure that the environment has been applied to the process correctly (check all nodes)

  $ xargs --null --max-args=1 echo < /proc/$(pgrep vertica$)/environ | grep ARENA
  $ MALLOC_ARENA_MAX=4 <== Verified

  Run a UDx (User Defined Extension) that trims the memory periodically

  With g++ version 4.8 or less, compile the following User Defined Extenstion

  /usr/bin/g++ -D HAVE_LONG_INT_64 -I /opt/vertica/sdk/include -Wall -shared -Wno-unused-value -fPIC -o /tmp/mtUDX.so /tmp/udx_malloc_trim.cpp /opt/vertica/sdk/include/Vertica.cpp

  ( attached : udx_malloc_trim.txt, since cpp files are sometimes blocked. rename this file to udx_malloc_trim.cpp before you compile it)

  After you have compiled the extension, install it as shown below

  $ vsql
  DROP LIBRARY if exists MallocTrim CASCADE;
  NOTICE 4185: Nothing was dropped
  DROP LIBRARY
  create library MallocTrim as '/tmp/mtUDX.so';
  CREATE LIBRARY
  CREATE transform FUNCTION mallocTrim AS LANGUAGE 'C++' NAME 'MallocTrimFactory' LIBRARY MallocTrim NOT FENCED;
  CREATE TRANSFORM FUNCTION

  Connect vsql to THE NODE where Vertica RSS memory is large and run MallocTrim(1). This will trim the vertica memory usage and prevent an OOM

  $vsql -h $vertica_host_ip
  skeswani=> select MallocTrim(1) over ();
  host | what | when | value
  -----------------+------+--------+-----------
  skeswani-laptop | RSS | before | 237502464
  skeswani-laptop | RSS | after | 228950016
  (2 rows)

  Here is simple shell script to check for memory use and run MallocTrim when the memory usage exceeds 11GB
  Although, I use 11 GB in this example, you should use 85% of memory on the machine as a trigger to run the trim.
  (i.e. if your hosts has 100GB RAM, then you should call MallocTrim when the Vertica RSS size exceeds 85GB)

  $ pages=$(cut -f2 -d ' ' /proc/$(pgrep vertica$)/statm)
  $ let mem="pages * 4096"
  $ if [ "$mem" -gt "11811160064" ]; then vsql -h $vertica_host_ip -a -c "select MallocTrim(1) over ();"; fi

  Note: This only applies to vertica version 9.1 and less. Starting with vertica version 9.2 , this is automatically done by the database and no admin intervention is needed

  udx_malloc_trim.txt 4.7K

  0