Skip to content

feat(planner): Pass 2 identical-expression rule (shared-input variants) - #588

Draft
zzylol wants to merge 1 commit into
stack/572-b5-executorfrom
stack/509-e1-pass2-identical
Draft

zzylol wants to merge 1 commit into
stack/572-b5-executorfrom
stack/509-e1-pass2-identical

Conversation

@zzylol

@zzylol zzylol commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Rebased on main d4869a7 (DF 54).

Why

#509 Stage 1 is Pass 1 (local alternatives) + Pass 2 (ASAP-aware CSE). Pass 2's identical-expression rule was missing. In Example 1 both queries read http_requests_total[1m], so the doc's "Raw with a shared input" plan was never a candidate. The facade also called share_common_sub_dags unconditionally while the devtool never did, so the two disagreed. In practice the facade's merge was a no-op for PromQL: the scan had no unique key, and CSE's legality rule refuses to share without one. Part of #580; stacked on #587 (#572).

What

  • Pass 2 identical-expression rule (asap-logical-optimizer/src/pass2/identical_expressions.rs). Stage 1 returns sharing variants: Pass 1 over the queries as written, and Pass 1 over the queries with identical sub-DAGs merged by the existing share_common_sub_dags. The shared variant is added only when the merge removes nodes. Shared candidates are labelled "… · shared input" and numbered after the independent ones.
  • Selection treats the variant as an outer choice (asap-plan-selection). The per-target DP runs once per variant, and the cheapest wins. In the shared variant, identical summary producers are also merged after composition. They are priced the same way in the DP, in exhaustive enumeration and in the final full check. The coupling guard also checks pairs of targets that read a common input. Fallback and the final full check apply per variant.
  • One library path. plan_selection::plan_stages runs Stage 1 (Pass 1 + Pass 2) → Stage 2 → Stage 3. The facade calls it and no longer pre-merges. The devtool calls it and only serializes the result (with display set so every candidate is priced).
  • Scan key. with_promql_series_identity now declares (series identity, ts) as the identified PromQL scan's unique key. A series has at most one sample per timestamp. CSE's legality rule needs this key to share the range selector.

How it was checked

  • DP = exhaustive on Example 1 (48 combinations, P44) and on a new workload where sharing merges a whole target: sum by (job) (rate(x[1m])) and max by (job) (rate(x[1m])), 16 + 8 combinations, where a shared candidate wins.
  • Un-ignored stage1_keeps_independent_and_shared_variants and stage3_shared_input_is_not_costlier. Hydra was dropped from both loops; it stays covered by the still-ignored stage1_q2_summary_families_are_heap_sketches_and_hydra, because Pass 1 has no Hydra alternative.
  • New stage3_selects_a_shared_input_plan: the selected plan shares the input, and it saves exactly one scan plus one range node.
  • Acceptance spec, its 48-row table and the viewer fixture are regenerated from the planner's output.

Before this PR (Example 1)

1 → 24 → 24 → 1. Selected P20 "Q1 exact (Sum acc, Rate acc) · Q2 exact (Sum acc)", 79.401 cpu ms. The scan (23.2) and range (4.0) are priced twice, once per query.

After this PR (Example 1)

1 → 48 → 48 → 1 (24 independent + 24 shared; 16 Count-Min + heap candidates invalid as before). Selected P44 "Q1 exact (Sum acc, Rate acc) · Q2 exact (Sum acc) · shared input", 52.201 cpu ms. The same choices as P20, with one scan → range:

Node P20 P44
scan 4,000,000 samples 23.2 × 2 23.2
time range 60 s 4.0 × 2 4.0
Rate acc build + finalize (Q1) 5.0 5.0
Sum acc build + finalize (Q1) 1.0 1.0
Sum acc build + finalize (Q2) 5.0 5.0
sort 1M rows / 100 partitions + limit (Q2) 14.001 14.001
Total 79.401 52.201

The best CountSketch + heap plan is P48 at 164.201 (shared).

Gate

cargo fmt --all --check ✓; cargo clippy --workspace --all-targets --all-features -- -D warnings ✓; cargo test --workspace: 1,514 passed / 10 ignored (#587: 1,508 / 12). The difference is 4 new tests (2 Pass 2 unit tests, shared_target_dp_equals_exhaustive, stage3_selects_a_shared_input_plan) and 2 un-ignored. Viewer: python3 -m unittest discover -s tools/dag-viewer -p test_render.py, 29 OK (6 skipped).

Refs #509, #580, #572.

🤖 Generated with Claude Code

…variant

Stage 1 now keeps two forms of the workload (#509 Pass 2): the queries as
written, and, when CSE merges something, the queries with identical
sub-DAGs shared. Stage 3 treats the variant as an outer choice: the
per-target dynamic program runs once per variant (the coupling guard also
checks targets reading a common input in the shared variant) and the
cheapest result wins.

plan_selection::plan_stages runs Stage 1 -> 2 -> 3; the facade and the
stage_pipeline devtool both call it, so the facade no longer pre-merges
and the devtool shows the shared variants.

A PromQL series has one sample per timestamp, so the identified scan now
declares (series identity, ts) as a key, which CSE's legality rule needs
to share the range selector.

Example 1: 1 -> 48 -> 48 -> 1; P44 (P20's choices over a shared input)
is selected at 52.201 vs 79.401.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the stack/509-e1-pass2-identical branch from 41c0c78 to a52409e Compare October 5, 2026 06:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant