High CPU on 1 node of 6 node cluster

bmurrell

I'm running v 12.0.3. Sometimes we have a high CPU on 1 node for a number of hours that I can't explain. This is 100% CPU, load avg of >50 for 3-4 hours.
It's not always the same node.
Sessions/reads/writes/mergeouts all look even across each node.
What system views can I look at to see what that node is doing?
There are no non-vertica processes running.

SruthiA

please check query_consumption view to see which query is consuming high cpu cycles.

bmurrell

Thanks SruthiA.

I've looked at this view, summing all cpu_cycles_us for the last 15 mins across all nodes, and they're pretty even.
However, in the last 15 mins, node 5 of the 6 nodes has been at 95%, while all other nodes are at 50%.
So something else is causing Vertica on that node to have high CPU.
A 'top' is showing vertica as the highest consumer.
How can I tell what it's doing?

Bryan_H

Two items to check: look at per-node resource usage in "select * from dc_execution_summaries;" though you may need to aggregate over time and per node in this table, you might be able to find a transaction and statement that uses more resources on node 5.
You may also run vertica process profile on that node during peak usage by running /opt/vertica/bin/vstack
However, if it comes to running vstack, you likely should open a support case and upload it there for analysis.
Also please consider upgrading to latest 12.0.4 patch as this provides a number of fixes and enhancements for resource management.