Options

High CPU on 1 node of 6 node cluster

bmurrellbmurrell Community Edition User

I'm running v 12.0.3. Sometimes we have a high CPU on 1 node for a number of hours that I can't explain. This is 100% CPU, load avg of >50 for 3-4 hours.
It's not always the same node.
Sessions/reads/writes/mergeouts all look even across each node.
What system views can I look at to see what that node is doing?
There are no non-vertica processes running.

Tagged:

Answers

  • Options
    SruthiASruthiA Vertica Employee Administrator

    please check query_consumption view to see which query is consuming high cpu cycles.

  • Options
    bmurrellbmurrell Community Edition User

    Thanks SruthiA.

    I've looked at this view, summing all cpu_cycles_us for the last 15 mins across all nodes, and they're pretty even.
    However, in the last 15 mins, node 5 of the 6 nodes has been at 95%, while all other nodes are at 50%.
    So something else is causing Vertica on that node to have high CPU.
    A 'top' is showing vertica as the highest consumer.
    How can I tell what it's doing?

  • Options
    Bryan_HBryan_H Vertica Employee Administrator

    Two items to check: look at per-node resource usage in "select * from dc_execution_summaries;" though you may need to aggregate over time and per node in this table, you might be able to find a transaction and statement that uses more resources on node 5.
    You may also run vertica process profile on that node during peak usage by running /opt/vertica/bin/vstack
    However, if it comes to running vstack, you likely should open a support case and upload it there for analysis.
    Also please consider upgrading to latest 12.0.4 patch as this provides a number of fixes and enhancements for resource management.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file