You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Implement the window-composition rule of logical Pass 2 in #509 (docs/design_docs/proposals/planner-layering.md, Example 3 Pattern B), starting with tumbling windows. Computations with the same summary input data share one tumbling-window summary. Each query's window is answered by merging the tumbling windows it covers. The tumbling length must divide both the query window and the evaluation interval. For example, [5m] evaluated every 1m uses 1-minute KLLs and merges 5 of them.
Blocked on
#511 (unified operators). Today a summary's window lives inside its input (SummaryAgg { KeepPreAsap(TimeRange{w, …}) }), so the window is part of the state's identity. A clean window-composition rule needs KeepPreAsap removed first, which is part of the #511 implementation.
The evaluation interval isn't visible to planning. It exists only in RepeatedDemand (crates/types/src/workload.rs). Pass 1/2 (search_workload_with_targets) receive only (id, expr, accuracy).
The window framework (tumbling/sliding/EH) is always None in production.selected_window_framework comes only from provider evidence, which only tests register (summary_maintenance_cost/model.rs, bind_window_framework_candidate).
The planner never builds a SummaryMerge. Its doc says "inserted by a deployment's stage allocator". The physical planner already compiles SummaryMerge as union + merge.
Queries with different windows never share.feat: share identical summary producers across queries after Pass 1 #515 shares only structurally identical SummaryAggs, and the window is inside the child. different_producers_are_not_shared (crates/planner/tests/summary_sharing.rs) covers this.
Merging is lossless for KLL (fixed k) and for exact Sum/Count/Min/Max. Rate/Increase need counter-reset handling across window boundaries. Offline empirical evidence rejects merges.
Proposed design (to revisit after #511; recommendations, not final)
Merge count: give SummaryMerge the query window, so it merges W/g tumbling windows. No new node type. Drop the "stage allocator only" note.
Evaluation interval: pass RepeatedDemand into Pass 1/2. g = gcd of the windows and evaluation intervals, or of the windows alone when no interval is known.
Phase/origin: the planner records only g. Window origin and phase belong to physical planning (stage 2).
Delete select_shared_panes.
Different accuracy across siblings: reuse the strictest-consumer sizing from Pass 2 rule 2. Window accuracy is Exact for mergeable families only.
Families: start with KLL and exact Sum/Count/Min/Max. Exclude Rate/Increase and non-mergeable intents.
A single repeating query (Pattern B) gets a tumbling candidate through the same path, with a group of one. The independent candidate is kept, and selection compares the two.
Candidate growth: one tumbling candidate per maximal sibling group, not every subset.
Tests to add
quantile_over_time(0.99, lat[5m]) every 1m plus quantile_over_time(0.5, lat[15m]) every 5m. Expect one shared 1-minute state, a per-query SummaryMerge, and independent candidates still present.
Single-query Pattern B.
Update different_producers_are_not_shared, using costs under which sharing loses.
Later
Sliding windows and Exponential Histograms. EH also needs query-time materialization and the summary subtract/delete design, which is a TODO in #509.
Goal
Implement the window-composition rule of logical Pass 2 in #509 (
docs/design_docs/proposals/planner-layering.md, Example 3 Pattern B), starting with tumbling windows. Computations with the same summary input data share one tumbling-window summary. Each query's window is answered by merging the tumbling windows it covers. The tumbling length must divide both the query window and the evaluation interval. For example,[5m]evaluated every 1m uses 1-minute KLLs and merges 5 of them.Blocked on
#511 (unified operators). Today a summary's window lives inside its input (
SummaryAgg { KeepPreAsap(TimeRange{w, …}) }), so the window is part of the state's identity. A clean window-composition rule needsKeepPreAsapremoved first, which is part of the #511 implementation.Current state on main (107ab28)
RepeatedDemand(crates/types/src/workload.rs). Pass 1/2 (search_workload_with_targets) receive only (id, expr, accuracy).Nonein production.selected_window_frameworkcomes only from provider evidence, which only tests register (summary_maintenance_cost/model.rs,bind_window_framework_candidate).SummaryMerge. Its doc says "inserted by a deployment's stage allocator". The physical planner already compilesSummaryMergeas union + merge.SummaryAggs, and the window is inside the child.different_producers_are_not_shared(crates/planner/tests/summary_sharing.rs) covers this.select_shared_panes(pane_sharing.rs) is dead code. It has no callers outside its own tests and duplicates feat: share identical summary producers across queries after Pass 1 #515's class pricing.Proposed design (to revisit after #511; recommendations, not final)
SummaryMergethe query window, so it merges W/g tumbling windows. No new node type. Drop the "stage allocator only" note.selected_window_framework = Tumblingfrom the rule itself. Exact representation depends on docs: define unified operators and SQL/PromQL scalar boundaries #511's post-KeepPreAsapIR.RepeatedDemandinto Pass 1/2. g = gcd of the windows and evaluation intervals, or of the windows alone when no interval is known.select_shared_panes.Exactfor mergeable families only.Tests to add
quantile_over_time(0.99, lat[5m])every 1m plusquantile_over_time(0.5, lat[15m])every 5m. Expect one shared 1-minute state, a per-querySummaryMerge, and independent candidates still present.different_producers_are_not_shared, using costs under which sharing loses.Later
Sliding windows and Exponential Histograms. EH also needs query-time materialization and the summary subtract/delete design, which is a TODO in #509.