Skip to content

Pass 2: window-composition rule (tumbling windows) — blocked on #511 #518

Description

@zzylol

Goal

Implement the window-composition rule of logical Pass 2 in #509 (docs/design_docs/proposals/planner-layering.md, Example 3 Pattern B), starting with tumbling windows. Computations with the same summary input data share one tumbling-window summary. Each query's window is answered by merging the tumbling windows it covers. The tumbling length must divide both the query window and the evaluation interval. For example, [5m] evaluated every 1m uses 1-minute KLLs and merges 5 of them.

Blocked on

#511 (unified operators). Today a summary's window lives inside its input (SummaryAgg { KeepPreAsap(TimeRange{w, …}) }), so the window is part of the state's identity. A clean window-composition rule needs KeepPreAsap removed first, which is part of the #511 implementation.

Current state on main (107ab28)

  • The evaluation interval isn't visible to planning. It exists only in RepeatedDemand (crates/types/src/workload.rs). Pass 1/2 (search_workload_with_targets) receive only (id, expr, accuracy).
  • The window framework (tumbling/sliding/EH) is always None in production. selected_window_framework comes only from provider evidence, which only tests register (summary_maintenance_cost/model.rs, bind_window_framework_candidate).
  • The planner never builds a SummaryMerge. Its doc says "inserted by a deployment's stage allocator". The physical planner already compiles SummaryMerge as union + merge.
  • Queries with different windows never share. feat: share identical summary producers across queries after Pass 1 #515 shares only structurally identical SummaryAggs, and the window is inside the child. different_producers_are_not_shared (crates/planner/tests/summary_sharing.rs) covers this.
  • select_shared_panes (pane_sharing.rs) is dead code. It has no callers outside its own tests and duplicates feat: share identical summary producers across queries after Pass 1 #515's class pricing.
  • Merging is lossless for KLL (fixed k) and for exact Sum/Count/Min/Max. Rate/Increase need counter-reset handling across window boundaries. Offline empirical evidence rejects merges.

Proposed design (to revisit after #511; recommendations, not final)

  1. Merge count: give SummaryMerge the query window, so it merges W/g tumbling windows. No new node type. Drop the "stage allocator only" note.
  2. Tumbling length g: represent it in the state's input window so feat: share identical summary producers across queries after Pass 1 #515 identity applies unchanged, and set selected_window_framework = Tumbling from the rule itself. Exact representation depends on docs: define unified operators and SQL/PromQL scalar boundaries #511's post-KeepPreAsap IR.
  3. Evaluation interval: pass RepeatedDemand into Pass 1/2. g = gcd of the windows and evaluation intervals, or of the windows alone when no interval is known.
  4. Phase/origin: the planner records only g. Window origin and phase belong to physical planning (stage 2).
  5. Delete select_shared_panes.
  6. Different accuracy across siblings: reuse the strictest-consumer sizing from Pass 2 rule 2. Window accuracy is Exact for mergeable families only.
  7. Families: start with KLL and exact Sum/Count/Min/Max. Exclude Rate/Increase and non-mergeable intents.
  8. A single repeating query (Pattern B) gets a tumbling candidate through the same path, with a group of one. The independent candidate is kept, and selection compares the two.
  9. Candidate growth: one tumbling candidate per maximal sibling group, not every subset.

Tests to add

  • quantile_over_time(0.99, lat[5m]) every 1m plus quantile_over_time(0.5, lat[15m]) every 5m. Expect one shared 1-minute state, a per-query SummaryMerge, and independent candidates still present.
  • Single-query Pattern B.
  • Update different_producers_are_not_shared, using costs under which sharing loses.

Later

Sliding windows and Exponential Histograms. EH also needs query-time materialization and the summary subtract/delete design, which is a TODO in #509.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions