vertica cluster shutting down

Ravi_Vijay

Hi All, We have a 6 node vertica cluster, which is shutting down itself every couple of hours. Appending logs just before shutting down. If anybody has any idea, what could be the cause of this sudden shutdown, do let us know. Memory utilization and disk space usage is way below available on each of the nodes. Logs 2013-09-16 07:44:06.798 unknown:0x7fe8f73b0700 [SAL] Large LRU usage: 0 free 0 in use 2013-09-16 07:44:06.798 unknown:0x7fe8f73b0700 [SAL] Typical LRU usage: 0 free 0 in use 2013-09-16 07:44:06.798 unknown:0x7fe8f73b0700 [SAL] Large LRU usage: 0 free 0 in use 2013-09-16 07:44:06.798 unknown:0x7fe8f73b0700 [SAL] Typical LRU usage: 0 free 0 in use 2013-09-16 07:44:06.798 unknown:0x7fe8f73b0700 [SAL] Large LRU usage: 0 free 0 in use 2013-09-16 07:44:06.798 unknown:0x7fe8f73b0700 [SAL] Typical LRU usage: 0 free 0 in use 2013-09-16 07:44:06.798 unknown:0x7fe8f73b0700 [SAL] Large LRU usage: 0 free 0 in use 2013-09-16 07:44:06.798 unknown:0x7fe8f73b0700 [Init] Global pool memory usage: NewPool(0x4712380) 'GlobalPool': totalDtors 0 totalSize 1340080128 (2753944 unused) totalChunks 15 2013-09-16 07:44:06.798 unknown:0x7fe8f73b0700 [Init] SAL global pool memory usage: NewPool(0x47023e0) 'SALGlobalPool': totalDtors 0 totalSize 98566144 (68721984 unused) totalChunks 5 2013-09-16 07:44:06.798 unknown:0x7fe8f73b0700 [Init] SS::stopPoller() 2013-09-16 07:44:06.798 unknown:0x7fe8f73b0700 [Init] DC::shutDown() 2013-09-16 07:44:06.798 unknown:0x7fe8f73b0700 [Init] Shutdown complete. Exiting. ~ Thanks in Advance, RAvi.

[Deleted User]

Hi Ravi, Seems that it Shutdown by itself in good condition. Are all the vertica.log have Shutdown complete? If so, I normally see this when the cluster find itself UNSAFE so it shutdown. It can be a spread issues that can be related to memory swap or network issues that maybe dropping UPD package, there are tools to monitor this but you will be better off opening a support ticket so they can help you to research that as this also could be different situations. Hope this helps, Eugenia

[Deleted User]

Hi Ravi, Hm, that's rather unusual... Unfortunately, there's nothing particularly exciting in those last few log lines. (Other than that Vertica appears to be performing a clean shutdown for some reason, not just crashing.) Vertica logs a *lot* during shutdown; most likely the interesting event is much earlier in the logs. Could you try looking backwards, back through a minute or two before the last successful queries were run? See if there are any WARNING or ERROR lines; also any lines that generally look suspicious. (Also, I have to ask the obvious question -- you're sure someone isn't being mean and logging into your cluster as a superuser and doing a "select shutdown();"? Not likely, but, well, have to ask just in case...) If you have a 6-node cluster, that's the (paid) Vertica EE, so I'd actually suggest that you bring this one up with Vertica Technical Support. They have some tools that can help track down weird issues like this much more quickly. Though you're of course welcome to ask here too :-) Thanks, Adam