Conversation
…ompute Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each raw stored output's native program is Planner's precompute graph from its raw sample boundary to the output. Validation admits raw sample inputs bound as ingestion inputs; pricing keeps charging raw outputs by their stored state. Tests that meant "a derived maintenance program exists" now skip raw-ingest programs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker keeps each open pane's admitted samples and, when the pane closes (or a late sample is forwarded), executes the output's installed Planner graph over them as one typed batch. Pane assignment, completeness and lateness stay in the worker. The single-pane merge helpers become a plain remove. Heaps keep their kernel until stored heap readout decodes Planner's weighted frequency state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A pane's samples reach the Planner graph in timestamp order with one sample per series and timestamp, so out-of-order arrival or a resent sample no longer fails a counter pane. A pane is removed only after it builds; each thread decodes an installed graph once; buffered samples share their series key. Tests cover watermark closure across scrape batches, counter ordering and late forwarded corrections. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #795.
Why
Live raw ingest evaluated summary computation in the backend:
RawDagPrograminterpreted eachSummaryAgg'sSummaryInputExprper sample, andmerge_*_panes_for_window"merged" panes (always a single pane) while silently dropping merge errors. Under the agreed layering, Planner's physical compile owns computation; the backend owns ingestion, routing, pane assignment, lateness and storage.What
integration/planner-for-backend@ca422df, which merges feat(physical): compile summaries over raw-sample precompute boundaries ASAPPlanner#488:precompute::compileaccepts a raw time-series scan as a boundary (rows[$population label map, $timestamp, value]).InstalledPostAsapDag::native_programs. Validation admits raw sample inputs (bound as ingestion inputs); pricing is unchanged (raw outputs are still priced by their stored state).RawDagProgram::buildruns the installed Planner graph over that pane as one typed batch (time-ordered, one sample per series and timestamp). Pane assignment, completeness and lateness are unchanged in the worker.merge_panes_for_window/merge_sketch_panes_for_windoware replaced by a plain remove.Before this PR
sum_over_time(asap_demo_gauge[5s]): each sample went throughRawDagProgram::apply→update_single(weight)on a backend-owned updater; the Planner DAG only chose the kernel.After this PR
The same output installs with its Planner graph (
Project → SummaryBuild → ScopeTimestamp) and each closed pane executes it. For the 16-query probe (exact / Epsilon / EpsilonDelta, 75 raw outputs), every raw output now compiles through Planner (11/75 before, via the general compiler).live_panes_execute_planner_dag_with_identical_statesfeeds one batch per scrape through the worker for every distinct raw output of those plans (Sum, Count, Min, Max, Rate, Increase, DDSketch, grouped variants; ≥50 outputs) and asserts byte-identical stored states versus the previous per-sample kernel path.Behaviour differences
Remaining
raw_dag.rsremains only for them.PrecomputeMaterializationcomputation fields (aggregation_type,parameters,spatial_filter,aggregated_labels,sample_update_rule) are still cross-checked against the DAG and are not removed here.Validation
cargo fmt --all -- --check;cargo clippy --workspace --all-targets --locked -- -D warnings;cargo test --workspace --locked --lib;cargo test -p data_plane --locked --test asapquery_compatibility_process_e2e -- --test-threads=1(26 passed);cargo test -p control_plane --locked --test native_rate_topk. New tests: live/Planner equivalence, counter ordering (fails without the ordering fix), late forwarded correction, raw output requires its Planner graph (fails on the base). Reviewed by a separate reviewer agent; findings addressed in the last commit.🤖 Generated with Claude Code