Options

Timer thread backtrace

Hello! 

I'm installing the Vertica Community Edition in an LXC container and I'll handily confess to suppressing a few errors in the install process to get to this point (notably /opt/vertica/share/binlib/read/get-disks has problems, and there's no 2GB swap partition). 

In spite of that, however, I seem to be able to run admintools -t createdb without too much difficulty, and it even stays up for a few minutes before failing. I dug into the error logs and this seems to be the root of my problems:
Backtrace Generated by Error  Signal: [0x0000000000000008] PID: [0x00000000000031bf] PC: [0x0000000002a23b98] FP: [0x00007f485a67fc20]SI_ADDR : [0x0000000002a23b98]  /opt/vertica/bin/vertica(_ZN6Basics9Backtrace11DoBacktraceEiiPvS1_+0x8cc)[0x33a011e]  /opt/vertica/bin/vertica(_ZN6Basics20GlobalSignalHandlers14logFatalSignalEiPvS1_+0xc7)[0x341f305]  /opt/vertica/bin/vertica[0x341f8f3]  /lib/x86_64-linux-gnu/libc.so.6(+0x364a0)[0x7f48f5ba34a0]  /opt/vertica/bin/vertica(_ZN3Mon29CheckForLowDiskSpaceTimerTask15runTaskInternalEb+0x478)[0x2a23b98]  /opt/vertica/bin/vertica(_ZN4Util16TimerServiceTask7runTaskEb+0x4b)[0x3581d33]  /opt/vertica/bin/vertica(_ZN4Util20TimerServiceTaskList18TaskSchedulingInfo10threadShimEPv+0x215)[0x3587aeb]  /opt/vertica/bin/vertica(_ZNK5boost9function0IvEclEv+0x1bb)[0x135f28b]  /opt/vertica/bin/vertica(_ZN7Session13ThreadManager12launchThreadERKN5boost9function0IvEE+0x57)[0x327229f]  /opt/vertica/bin/vertica(thread_proxy+0x80)[0x3a179c0]  /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f48f5753e9a]  /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f48f5c613fd]  END BACKTRACE  THREAD CONTEXT  Thread type: Timer Thread  Request: Unknown request  Transaction: [0x00a000000000008c]  END THREAD CONTEXT
I'm afraid I'm not clear on what exactly is going on here; can someone explain to me what's happening (and what I can do to resolve it?)

Comments

  • Options
    Hi Dave,

    Hm...  I'm honestly not sure what's going on here.

    The stack trace in question is in code that periodically checks the free disk space in all storage locations, to make sure they have not overflowed.  The fact that "get-disks" had trouble is a warning sign to me.  That said, I'm not immediately sure how this would result in a crash.

    Vertica is not often run (and not officially tested) inside LXC containers -- we're typically sliced by "fraction of a data center", not "fraction of a machine".  Different mentality.  That said, I know that we have been run successfully in containers elsewhere; there are cases where it's quite useful.  I would encourage you to tweak your container configuration; maybe see if you can get more of the installation checks to pass.

    Adam

  • Options
    As a means of testing -- within the container, can you as a user see available disk space?, available # of inodes?, etc.
  • Options
    So, I ran into this thread yesterday, which seems to have a similar issue. Like them, if I df -i, I get a drive mounted with 0 inodes, etc...:
    ubuntu@box82:~$ df -i Filesystem       Inodes IUsed    IFree IUse% Mounted on /dev/xvdh             0     0        0     - / cgroup         31438790    10 31438780    1% /sys/fs/cgroup none           31438790    68 31438722    1% /run none           31438790     1 31438789    1% /run/lock none           31438790     1 31438789    1% /run/shm
    I've had someone suggest I could mount a filesystem within a file within the container and try to point Vertica at that; I'm aware that it's a pretty hacky solution and I'm not sure how happy Vertica would be about it.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file