Skip to content

feat(query-engine): add DDSketch aggregation type - #792

Merged
milindsrivastava1997 merged 6 commits into
mainfrom
feat/ddsketch-aggregation
Oct 6, 2026
Merged

milindsrivastava1997 merged 6 commits into
mainfrom
feat/ddsketch-aggregation

Conversation

@zzylol

@zzylol zzylol commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds a DDSketch quantile aggregation to the ASAPQuery data plane and query engine. This is the DDSketch slice of #787 (and the DDSketch aggregation type from #762); CountSketch and UnivMon are not included.

The accumulator wraps asap_sketchlib::DDSketch, the implementation sketch-bench measures, so the optimizer's DD cost rows describe what ASAPQuery runs. It is needed for the data-plane vs ClickHouse evaluation (#789), which compares DDSketch on both sides (quantilesDD(α) in ClickHouse).

  • Type: AggregationType::DDSketch (aliases DDSketchAccumulator, ddsketch, dd). It is single-population like DatasketchesKLL, and capability matching serves Statistic::Quantile from it.
  • Accumulator: DDSketchAccumulator handles update, quantile query, merge, and msgpack/JSON serialization. A mismatched alpha or a different accumulator type is rejected on merge. Negative values go to a mirrored store and zeros to a zero bucket, as in ClickHouse's quantilesDD; non-finite values are dropped. An empty sketch returns NaN.
  • Config: a DDSketch aggregation requires parameters.alpha in (0, 1) and fails validation otherwise (InvalidAlpha). DDSketch::new panics outside that range, so the check runs before the accumulator is built.
  • Dependency: asap_sketchlib moves from 94d76f6 to 5158bf2 for the zero bucket and negative store (fix(ddsketch): retain negative observations and zeros asap_sketchlib#141), which no release includes yet. No other sketch's tests change.
  • Planner: builds {alpha} with a default of 0.01, overridable via sketch_parameters.DDSketch.alpha, and adds the sketch's properties (mergeable, not subtractable).

Not included

  • The planner still maps quantiles to DatasketchesKLL (map_statistic_to_precompute_operator). Choosing DD is part of --planner milp (Optimizer: move sketch-bench #129 MILP into asap-planner-rs (--planner milp) #753).
  • The MILP optimizer skips DDSketch (OPTIMIZER_SKIPPED_AGG_TYPES) until it has a parameter grid and atomic_costs mapping (Add optimizer aggregation families and capability/SLA metadata #762). Without the skip it would propose DD candidates with no alpha.
  • A deterministic choice between quantile configs. If a KLL and a DDSketch config share a metric and window, capability matching picks whichever HashMap order puts first (already true of KLL vs HydraKLL). Breaking ties by aggregation_id is a follow-up.
  • Subtraction for sliding windows. DD bucket counts could support it, but the accumulator only merges, matching KLL.

Tests

  • Accumulator unit tests: quantiles stay within α on 1..=10,000; zero and negative inputs are counted; an empty sketch returns NaN; kwargs are validated; a merge equals a single sketch over all values (directly and through merge_with); mismatched α or type is rejected; serialization round-trips.
  • Factory: the configured α reaches the accumulator; reset clears state; missing, out-of-range or string α is rejected; config_is_keyed agrees with the updater.
  • Config: YAML loads with a valid alpha and is rejected without one.
  • Capability matching: a quantile query matches a DDSketch config.
  • Dependency: asap_sketchlib moves from 94d76f6 to 5158bf2 for the zero bucket and negative store (fix(ddsketch): retain negative observations and zeros asap_sketchlib#141), which no release includes yet. No other sketch's tests change.
  • Planner: default α, and the override.
  • End to end (e2e_grouped_quantile_over_ddsketch_is_within_alpha): Remote Write → precompute → one DDSketch per group → quantile by (job) (0.9, …). Each job has two instances and twenty zeros; each group's estimate is within α of the true p90, which the zeros move outside the bound if they were dropped.

cargo test for asap_types, promql_utilities, asap_planner and query_engine_rust passes; cargo test --workspace passes (1,187 tests). cargo clippy --workspace --all-targets --all-features --locked -- -D warnings and cargo fmt --check are clean.

Part of #787.

🤖 Generated with Claude Code

Add a DDSketch quantile aggregation backed by asap_sketchlib::DDSketch,
the implementation sketch-bench measures for the planner's cost table.

- AggregationType::DDSketch (aliases: DDSketchAccumulator, ddsketch, dd),
  single-population, served for Statistic::Quantile by capability matching.
- DDSketchAccumulator: update, quantile query, merge, msgpack/JSON output.
  Non-positive values are dropped, as DDSketch indexes positive values only.
- Accumulator factory updater; config validation requires an `alpha`
  parameter in (0, 1).
- Planner: DDSketch parameter building (default alpha 0.01, overridable via
  sketch_parameters.DDSketch.alpha) and sketch properties. The MILP optimizer
  skips DDSketch until it has a parameter grid and cost rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol marked this pull request as ready for review October 6, 2026 19:10
zzylol and others added 2 commits October 6, 2026 19:23
…gatives

asap_sketchlib#141 gives DDSketch a zero bucket and a negative store, as
ClickHouse's quantilesDD has. At the old pin, zero and negative samples were
dropped, so quantiles over data containing them were computed over the
positive part only. No release includes the fix yet, so the git pin moves.

The accumulator's doc comment and drop test now describe the kept values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each job now has two instances feeding one DDSketch, sent in time order
since the watermark is per group, plus ten zeros per instance. The zeros
move the true p90 from 90·scale to 88·scale, outside the alpha bound, so
the test fails if zeros are dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol and others added 3 commits October 6, 2026 19:40
- Move the DDSketch e2e test off port 19421, which the legacy-feature
  topk range test also binds.
- State the accuracy bound as |x̂ - x| <= alpha·|x|; (1+alpha)/(1-alpha)
  is the bucket width, not the guarantee.
- Count the negative store in memory_usage_bytes.
- Drop issue numbers from code comments, per AGENTS.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
merge_accumulators_batch had no DDSketch arm, so merging N buckets went
through merge_with, which clones the running sketch at every step.
DDSketchAccumulator::merge_multiple clones the first sketch once and
merges the rest into it in place, as the KLL and CMS arms do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@milindsrivastava1997
milindsrivastava1997 merged commit a15224c into main Oct 6, 2026
10 checks passed
@milindsrivastava1997
milindsrivastava1997 deleted the feat/ddsketch-aggregation branch October 6, 2026 21:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants