Skip to content

refactor: execute Planner precompute DAGs for backfill and revisions - #797

Open
zzylol wants to merge 3 commits into
refactor/precompute-runs-planner-dagfrom
refactor/precompute-dag-backfill-revisions
Open

zzylol wants to merge 3 commits into
refactor/precompute-runs-planner-dagfrom
refactor/precompute-dag-backfill-revisions

Conversation

@zzylol

@zzylol zzylol commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #796.

Why

After #796 the live path executes Planner's precompute graph, but backfill and Remote Write revisions still evaluated SummaryInputExpr per sample in the backend (RawDagProgram::apply / validate_sample), and re-derived that interpreter per request.

What

  • Backfill (build_dag_accumulator) hands each window's samples to the output's installed Planner graph, as live ingest does.
  • Remote Write revisions (accept_revision) buffer each stored window's samples (pane assignment, retention and the correction horizon stay in the backend) and build each window with the Planner graph. Request admission runs the same graph over the request's matching samples and rejects the whole request if it fails. Known-empty counter panes of a captured revision keep the family's empty state.
  • The per-sample interpreter in raw_dag.rs becomes private and is used only by heaps (see Remaining); the interpreter-only validation and the unused value-projection field are removed.

Before this PR

A backfilled sum_over_time(asap_demo_gauge[5s]) window and a revised window were built by program.apply per sample, while the live pane of the same samples was built by the Planner graph.

After this PR

All three paths call RawDagProgram::build: live_panes_execute_planner_dag_with_identical_states now also builds every live pane's samples through build_dag_accumulator and asserts byte-identical states; the process e2e revision tests (continuous_revisions_*, native_*_ensemble_revises_and_recovers) run the revision path end to end.

Behaviour differences

  • Revision windows are bounded by the Planner build limit (max_checkpoint_bytes) and, as before, by built state size; overlapping windows share each buffered sample's series key.
  • Admission rejects (400) a request whose values the Planner graph rejects (e.g. a non-finite value) instead of applying the old kernel's per-value check; a Planner memory limit stays a typed resource error.
  • A non-empty revision window that yields no population is an error rather than a silently missing window.
  • Backfill and revision windows are delivered in timestamp order with one sample per series and timestamp, as live panes are since refactor: execute Planner precompute DAGs for live raw ingest #796.

Remaining

  • CMS/CountSketch heaps still use the backend heap kernel (the item interpreter stays private to them); the next stacked PR moves them.
  • empty_state uses the backend kernel factory to create an empty family state for known-empty counter panes and the revision codec check.
  • PrecomputeMaterialization computation fields remain (cross-checked against the DAG).

Validation

cargo fmt --all -- --check; cargo clippy --workspace --all-targets --locked -- -D warnings; cargo test --workspace --locked --lib; cargo test -p data_plane --locked --test asapquery_compatibility_process_e2e -- --test-threads=1 (26 passed). New tests: backfill equivalence, Planner admission (including a typed memory-limit error that fails when errors are stringified). Reviewed by a separate reviewer agent; findings addressed in the last commit.

🤖 Generated with Claude Code

zzylol and others added 3 commits September 30, 2026 07:39
Backfill hands each window's samples to the output's installed Planner
graph, as live ingest does; a window from which the graph admits no
population is not written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r graph

Remote Write revisions buffer each stored window's samples and build it with
the installed Planner graph; request admission runs the same graph and
rejects the whole request if it fails. Known-empty counter panes of a
captured revision keep the family's empty state. The per-sample interpreter
in raw_dag.rs is now private to heaps, whose stored readout still decodes
only the backend heap kernel. Tests cover backfill equivalence and Planner
admission.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Planner execution errors keep their type, so a memory limit during revision
admission or window building stays a resource error rather than a 400 input
rejection. Revisions use the installed raw programs instead of re-deriving
them per request, count buffered samples once however many windows share
them, and fail a non-empty window that yields no population. Heaps keep
their install-time update check while they use the kernel interpreter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant