Skip to content

refactor: execute Planner precompute DAGs for live raw ingest - #796

Open
zzylol wants to merge 4 commits into
feat/planspace-compile-oncefrom
refactor/precompute-runs-planner-dag
Open

zzylol wants to merge 4 commits into
feat/planspace-compile-oncefrom
refactor/precompute-runs-planner-dag

Conversation

@zzylol

@zzylol zzylol commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Stacked on #795.

Why

Live raw ingest evaluated summary computation in the backend: RawDagProgram interpreted each SummaryAgg's SummaryInputExpr per sample, and merge_*_panes_for_window "merged" panes (always a single pane) while silently dropping merge errors. Under the agreed layering, Planner's physical compile owns computation; the backend owns ingestion, routing, pane assignment, lateness and storage.

What

  • Repin Planner to integration/planner-for-backend@ca422df, which merges feat(physical): compile summaries over raw-sample precompute boundaries ASAPPlanner#488: precompute::compile accepts a raw time-series scan as a boundary (rows [$population label map, $timestamp, value]).
  • Control plane retains, for every raw stored output, Planner's compiled precompute graph in InstalledPostAsapDag::native_programs. Validation admits raw sample inputs (bound as ingestion inputs); pricing is unchanged (raw outputs are still priced by their stored state).
  • Live worker keeps each open pane's admitted samples; when the backend closes a pane (watermark, flush, shutdown) or forwards a late sample, RawDagProgram::build runs the installed Planner graph over that pane as one typed batch (time-ordered, one sample per series and timestamp). Pane assignment, completeness and lateness are unchanged in the worker.
  • merge_panes_for_window / merge_sketch_panes_for_window are replaced by a plain remove.

Before this PR

sum_over_time(asap_demo_gauge[5s]): each sample went through RawDagProgram::apply → update_single(weight) on a backend-owned updater; the Planner DAG only chose the kernel.

After this PR

The same output installs with its Planner graph (Project → SummaryBuild → ScopeTimestamp) and each closed pane executes it. For the 16-query probe (exact / Epsilon / EpsilonDelta, 75 raw outputs), every raw output now compiles through Planner (11/75 before, via the general compiler). live_panes_execute_planner_dag_with_identical_states feeds one batch per scrape through the worker for every distinct raw output of those plans (Sum, Count, Min, Max, Rate, Increase, DDSketch, grouped variants; ≥50 outputs) and asserts byte-identical stored states versus the previous per-sample kernel path.

Behaviour differences

  • A pane's state is built when the pane closes, not per sample; open panes hold their samples (memory is O(samples in open panes), each build bounded by Planner's default 64 MiB limit).
  • Samples in a pane are delivered in timestamp order with one sample per series and timestamp (first kept). Counter results no longer depend on arrival order, and a resent identical-timestamp sample is not double-counted.
  • Plans installed without raw Planner graphs (compiled before this PR) no longer install.

Remaining

  • CMS/CountSketch heaps still use the backend heap kernel (stored heap readout does not yet decode Planner's weighted-frequency state); the item interpreter in raw_dag.rs remains only for them.
  • Backfill and Remote Write revisions still use the per-sample interpreter; the next stacked PR moves them.
  • PrecomputeMaterialization computation fields (aggregation_type, parameters, spatial_filter, aggregated_labels, sample_update_rule) are still cross-checked against the DAG and are not removed here.

Validation

cargo fmt --all -- --check; cargo clippy --workspace --all-targets --locked -- -D warnings; cargo test --workspace --locked --lib; cargo test -p data_plane --locked --test asapquery_compatibility_process_e2e -- --test-threads=1 (26 passed); cargo test -p control_plane --locked --test native_rate_topk. New tests: live/Planner equivalence, counter ordering (fails without the ordering fix), late forwarded correction, raw output requires its Planner graph (fails on the base). Reviewed by a separate reviewer agent; findings addressed in the last commit.

🤖 Generated with Claude Code

zzylol and others added 4 commits September 30, 2026 07:15
…ompute

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each raw stored output's native program is Planner's precompute graph from
its raw sample boundary to the output. Validation admits raw sample inputs
bound as ingestion inputs; pricing keeps charging raw outputs by their
stored state. Tests that meant "a derived maintenance program exists" now
skip raw-ingest programs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker keeps each open pane's admitted samples and, when the pane
closes (or a late sample is forwarded), executes the output's installed
Planner graph over them as one typed batch. Pane assignment, completeness
and lateness stay in the worker. The single-pane merge helpers become a
plain remove. Heaps keep their kernel until stored heap readout decodes
Planner's weighted frequency state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A pane's samples reach the Planner graph in timestamp order with one sample
per series and timestamp, so out-of-order arrival or a resent sample no
longer fails a counter pane. A pane is removed only after it builds; each
thread decodes an installed graph once; buffered samples share their series
key. Tests cover watermark closure across scrape batches, counter ordering
and late forwarded corrections.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant