Options

SUM() aggregate returns NULL when a duplicate COUNT(DISTINCT) is merged into a shared grouping set

SUM() aggregate returns NULL when a duplicate COUNT(DISTINCT) is merged into a shared grouping set

Summary

After upgrading from 25.1.0-3 to 26.2.0-2, a query that combines SUM()
with multiple COUNT(DISTINCT) aggregates started returning NULL for the
SUM() column on a subset of rows (approximately 0.4-0.7% of rows in our
workload). The same query returns correct values on 25.1.0-3 for the same
data.

Trigger

The problem appears when the SELECT list contains both:

  • COUNT(DISTINCT order_id)
  • another COUNT(DISTINCT <expression derived from order_id>) (e.g. a
    CASE WHEN ... THEN order_id ELSE NULL END inside a COUNT(DISTINCT))

in the same query, alongside a SUM() aggregate.

Removing COUNT(DISTINCT order_id) makes SUM() return correct values
for all rows. (We did not separately test removing only the CASE-based
duplicate while keeping COUNT(DISTINCT order_id).)

Root cause (as far as we can tell from EXPLAIN VERBOSE)

With multiple COUNT(DISTINCT) aggregates in one query, the optimizer
splits the input into several grouping sets, one per distinct aggregate,
so each can compute its own distinct count. SUM() is not distinct-aware,
so the optimizer attaches it to exactly one of these grouping sets rather
than duplicating it across all of them.

In our failing plan, SUM() was attached to the grouping set keyed on
order_id. The optimizer then detected that COUNT(DISTINCT order_id)
and COUNT(DISTINCT <CASE expression that also resolves to order_id>)
are effectively counting the same column, and merged their two grouping
sets into one. During that merge, the SUM() aggregate that had been
attached to the order_id grouping set was dropped, producing NULL for
the rows computed through that merged set.

Plan comparison

Failing plan — lower GROUPBY HASH node:

[PASTE MASKED EXPLAIN VERBOSE HERE - FAILING QUERY]

Key lines to note:
Aggregates: sum(<SVAR>)
Group By: order_date_local, country_cd, order_id, <SVAR>, <SVAR>, <SVAR>, <SVAR>
Grouping Sets: (order_date_local, country_cd, order_id, <SVAR>),
               (order_date_local, country_cd, <SVAR>),
               (order_date_local, country_cd, <SVAR>),
               (order_date_local, country_cd, <SVAR>),
               (order_date_local, country_cd, <SVAR>)

Working plan — same query with COUNT(DISTINCT order_id) moved into a
separate subquery joined back on the GROUP BY keys:

[PASTE MASKED EXPLAIN VERBOSE HERE - WORKING QUERY]

Key lines to note:
Aggregates: sum(<SVAR>)
Group By: order_date_local, country_cd, <SVAR>, <SVAR>, <SVAR>, <SVAR>, <SVAR>
Grouping Sets: (order_date_local, country_cd, <SVAR>, <SVAR>),
               (order_date_local, country_cd, <SVAR>),
               (order_date_local, country_cd, <SVAR>),
               (order_date_local, country_cd, <SVAR>),
               (order_date_local, country_cd, order_id)

In the failing plan, order_id is absorbed into a 4-column grouping set
together with one of the CASE-derived expressions, and the sum()
attached to that set is dropped. In the working plan, order_id gets its
own single-column grouping set, and sum() survives.

Both plans report identical cost for the top-level GROUPBY node
(96214601 in our environment), so this is not a cost-based plan choice
difference — it is a difference in how grouping sets are merged for
equivalent-looking distinct expressions.

Things ruled out

  • EnableAggressiveParallelJoins = 0 — still reproduces. Confirmed the
    setting was actually applied at session level via
    configuration_parameters.current_level.

  • Rewriting the duplicate as order_id || '' to avoid literal expression
    matching — still reproduces (the two expressions are still recognized
    as functionally dependent on the same column).

  • Adding ELSE 0 to the SUM's CASE expression (SUM(CASE WHEN ... THEN amt ELSE 0 END)) — still returns NULL for the affected rows. This is
    significant: with ELSE 0, a NULL result is not possible under correct
    aggregate semantics regardless of how the CASE condition evaluates.
    This indicates the aggregate slot itself is being discarded during the
    grouping-set merge, not that the CASE expression is evaluating
    differently.

Environment

  • Vertica 26.2.0-2 (regression from 25.1.0-3; same query is correct on
    25.1.0-3 against the same data)

  • Eon Mode, 6 nodes

  • Source table is an external Parquet table on S3 (via LOAD EXTERNAL TABLE)

  • No statistics collected on the fact table (NO STATISTICS shown
    throughout the plan; row count estimated at ~289M from file size)

  • Query includes 2 LEFT OUTER JOINs; the plan shows (PUSHED GROUPING) Partial Aggs: sum(<SVAR>) at the join level, i.e. partial aggregation
    is pushed down into the join before the grouping-set split occurs

Reproduction

Not yet reproducible on a small, isolated, native (non-external) table.
The failing behavior requires the optimizer to choose the grouping-sets
strategy for multiple distinct aggregates. On small native tables
(under roughly 1M rows, no statistics), the optimizer instead chooses a
self-join strategy (a separate GROUPBY per distinct aggregate, joined via
MERGEJOIN), which does not exhibit the issue.

This suggests the trigger is tied to plan selection based on row-count
estimates (our real table is estimated at ~289M rows with NO STATISTICS)
rather than to the aggregate structure alone. We are able to reproduce
reliably against our own external table but cannot yet provide a
self-contained repro script. We can make anonymized EXPLAIN VERBOSE
output, a scrutinize bundle, or (if useful) temporary access to a
non-production cluster available on request.

Question

Is this a known issue in 26.2? Is there a configuration parameter that
controls whether COUNT(DISTINCT) expressions resolving to the same base
column are merged into a shared grouping set, so that we can disable that
specific behavior without restructuring every affected query while a fix
is pending?

Tagged:

Answers

  • Options
    moshegmosheg Vertica Employee Administrator

    I attempted to reproduce this on 26.1.0 and 26.2.0-2
    (Eon, external Parquet table on S3, no statistics, two LEFT OUTER JOINs, SUM(CASE ... ELSE 0 END) combined with COUNT(DISTINCT order_id) and three COUNT(DISTINCT CASE ... THEN order_id END)).

    On both versions the optimizer produced the plan shape you show for the failing query - a lower GROUPBY HASH where order_id shares a grouping set with a CASE-derived expression and sum() is attached to that set, and the results matched a row by row comparison against your subquery workaround.
    So the grouping-set layout alone does not cause the NULL; the trigger is more specific to your query.

    The notable difference from my test is "(PUSHED GROUPING) Partial Aggs: sum()" at the join level in your plan:
    the SUM is partially aggregated below the joins before the grouping-set split.
    My plan does not push the partial aggregate.
    That is where I would focus.

    I do not know about a configuration parameter that controls merging COUNT(DISTINCT) expressions on the same base column into a shared grouping set.
    Please open a support case with the scrutinize bundle, both EXPLAIN VERBOSE outputs,
    and the masked query text so engineering can reproduce it against your exact plan.
    In the meantime, the subquery workaround is the correct interim solution
    APPROXIMATE_COUNT_DISTINCT(order_id) is an alternative if an exact count is not required.

This discussion has been closed.