feat(rqe-optimizer): family properties, shared fixed-size sketches and DeltaSet key tracker

[milindsrivastava1997]

Part of #143 (S4b), including the cost-model part of #142. S7 was already done in #145.

Cost formulas by family shape

G = card(G), a_D = x/y open windows, n = L/x windows merged per query, C = (L − x)/y + 1 closed windows, T = query interval.

	KLL (one sketch per group)	exact-sum (shared accumulator, export divides by group count)	HydraKLL (one fixed-size shared sketch)
m in the row	one group's sketch	whole accumulator ÷ groups	whole sketch
c_mrg in the row	one group's merge	whole fold ÷ groups	whole-sketch merge
c_ins / c_qry	per sample / per group	per sample / per probe = group	per sample / per probe = group
Ingest CPU	λ·a_D·c_ins	λ·a_D·c_ins	λ·a_D·c_ins
Ingest memory	G·m·a_D	G·m·a_D	m·a_D
Merge CPU	G·(n−1)·c_mrg/T	same	(n−1)·c_mrg/T
Merge memory	G·m	same	m
Query CPU	G·c_qry/T	same	G·c_qry/T
Query memory	G·8B	same	G·8B
Storage memory	G·m·C	same	m·C
Latency	G·(c_qry + (n−1)·c_mrg)	same	(n−1)·c_mrg + G·c_qry

A deployment that needs a key tracker also pays for a DeltaSet (exact-delta-set, m_ds per key). It is priced like an exact accumulator on the same G, x and y, in every phase:

Phase	Tracker adds
Ingest	CPU λ·a_D·c_ins_ds, memory G·m_ds·a_D
Merge	CPU G·(n−1)·c_mrg_ds/T (union of the windows' key sets), memory G·m_ds
Query	CPU G·c_qry_ds/T (enumerate keys to probe)
Storage	memory G·m_ds·C
Latency	G·(c_qry_ds + (n−1)·c_mrg_ds)

Design decisions (please approve)

These were settled in a design session. Reviewers, please approve them along with the code.

#	Question	Decision	Why
1	Where do family properties live?	Hardcoded family_properties(variant) in lib.rs, next to DEPLOYABLE_FAMILIES, not in the cost JSON	They are facts about the algorithm, not measurements
2	Include subtractable?	No; it comes back with S9	Subtract is deferred and nothing would read it
3	Include mergeable?	Yes, as mergeable_across_windows, true for every family; a non-mergeable family is eligible only when L == x	Every measured family merges, but the rule belongs in the model
4	Cost shape of a shared fixed-size sketch (Hydra)	Instances per window = 1 instead of card(G) for ingest memory, merge, storage and latency; query stays card(G) · c_qry (one probe per group)	Hydra rows are whole-sketch; the exact-* rows are already per group
5	Pruning with mixed shapes	Pass facts and compare the cost model's real costs	card(G) is no longer a common multiplier within a group
6	Hydra accuracy vs group count	Candidate only when card(G) ≤ query_accuracy["subpopulations"]; more measured points in S5 (comment on #143)	Conservative, uses data already in the row; a curve needs new measurements
7	How the key tracker is priced	Deployment.key_tracker carries the exact-delta-set row, priced as an exact accumulator in all phases (ingest, merge, query, storage, latency); no tracker row means no Hydra candidate	A query over L needs the union of keys across the merged windows, so merge and query costs are real
8	Is hydra-kll deployable?	No: in families() only, planned only with --allow-undeployable-families	Not ready for ASAPQuery yet
9	AutoSketch baseline	Skips shared fixed-size families for now; ranking them properly is #159	Its per-instance memory ranking can't compare shared and per-group sketches
10	Property names	mergeable_across_windows, one_fixed_size_sketch_for_all_groups, needs_delta_set_key_tracker	"Subpopulation-aware" would read as wrong for exact-* (they hold all groups but are priced per group)
11	Unknown variant	No default: family_properties panics, and a test covers every families() entry	A guessed default would misprice silently
12	Tracker invariant (from review)	priced_parts asserts a key tracker is present exactly when the family needs one	Pricing a Hydra without its tracker would undercount silently
13	Tracker deployability (from review; reverses an earlier fix)	The tracker row is used whenever its family is a candidate; no separate deployability check	Hydra's own deployability already gates the pair, and ASAPQuery runs the tracker as DeltaSetAggregator; the check would silently drop Hydra if it became deployable without the tracker being listed
14	What the group-count gate models (from review)	Hydra keyed by the joined G value alone (one subkey per sample), so load = card(G). The current rows fan out to every label subset; re-measuring without fan-out is #165	That's the intended sketch. A `2^
15	Where validity is decided (from review)	is_eligible(r, d, facts) holds the group-count gate and the tracker-presence check; facts is threaded through eligible_deployments_for and enumerate::*	The MILP, enumeration and pruning agree on what is valid
16	Several tracker rows (from review)	Panic if the cost table has more than one exact-delta-set row	Otherwise the price depends on row order
17	Hydra merge accuracy (from review)	Out of scope here, tracked in #166	Needs measurements or a refuse-until-measured rule

Interface changes later sessions depend on

• FamilyProperties and family_properties(variant) (lib.rs). Fields: mergeable_across_windows, one_fixed_size_sketch_for_all_groups, needs_delta_set_key_tracker. An unknown variant panics instead of getting a default; a test checks that every families() entry and KEY_TRACKER_FAMILY has one. Deployment::properties() is a shorthand.
  • All families are mergeable. A non-mergeable family would be eligible only when L == x.
  • Only hydra-kll has the other two set. The exact-* accumulators are Shared + PerKey in the design doc's terms and are false here, because the export already prices them per group.
  • subtractable is left out until Subtract (S9) needs it.
• Deployment.key_tracker: Option<AtomicCostEntry>: the exact-delta-set row for families that need a tracker, None otherwise. S8 reads this field to flag the tracker in the plan. If the cost table has no tracker row, families that need one are not candidates.
• KEY_TRACKER_FAMILY = "exact-delta-set". It is in no capability's families(), so it is never a candidate on its own.
• hydra-kll joins Capability::Quantile.families() but not DEPLOYABLE_FAMILIES, so it is planned only with --allow-undeployable-families.
• is_eligible(r, d, facts) (was is_eligible(r, d)), and facts added to eligible_deployments_for, enumerate::{unservable, brute_force, for_each_mapping, for_each_mapping_while}. Eligibility now also requires a key tracker exactly when the family needs one.
• Group-count gate (in is_eligible): a family with one_fixed_size_sketch_for_all_groups is eligible only when card(G) ≤ query_accuracy["subpopulations"] of its row. Each row has one measured point, so larger groupings are never planned. Rows measured at more group counts (S5 follow-up, see the comment on Tracking: make rqe-optimizer the solver for ASAPQuery's planner #143) loosen the gate with no code change.
• instance_memory_bytes takes the config to size (the sketch or its tracker) as its first argument. DDSketch's value-range sizing (rqe-optimizer: size DDSketch memory from the metric's value range #152) applies to the sketch only.
• Dominance pruning now compares the cost model's full costs (ingest, query_latency_ms, merge, storage_bytes) instead of per-group copies. Each candidate's cost vector is computed once and then compared pairwise. card(G) is no longer a common multiplier once shared and per-group sketches compete in the same group.
• More than one exact-delta-set row panics during candidate generation.
• AutoSketch skips families with one_fixed_size_sketch_for_all_groups, even with --allow-undeployable-families, because its ranking compares per-instance memory. Fixing that is rqe-optimizer: AutoSketch baseline should rank shared fixed-size sketches (HydraKLL) by deployment cost #159.

Not in this PR

• The cost table has no hydra-kll or exact-delta-set rows yet: the export hasn't been re-run since feat(bench): exact-delta-set row and HydraKLL in the rqe-optimizer cost export #149. Tests use hand-built rows.
• small_problem doesn't print the key tracker (S8).

Test plan

• cargo test --workspace, clippy (-D warnings) and fmt via pre-commit.
• New tests:
  • every family has properties;
  • HydraKLL plus tracker priced phase by phase;
  • Hydra carries the tracker and is dropped without a tracker row;
  • group-count gate;
  • a shared sketch isn't pruned by a sketch that is smaller per group but costs more overall (the old per-group comparison pruned it).

🤖 Generated with Claude Code

[milindsrivastava1997]

Review notes. The inline comments cover lines in the diff. Three more, on lines outside it:

1. rqe-optimizer/src/lib.rs accuracy_at: Hydra rows have an empty merge_accuracy, so a query merging n windows reads the single-window err. A fixed-size sketch folding n windows of every group holds n times the items in the same space, so for a quantile RAQE with L = 10x, accuracy_ok_for can pass while the true rank error exceeds the SLA. Either measure Hydra at several merge counts or refuse n > 1 for Hydra until those rows exist.
2. scripts/export_rqe_optimizer_costs.sh (~L152): this comment is out of date. It says the optimizer prices every row as card(G) x per-instance cost and that HydraKLL rows "must not become candidates until #142 lands". This PR is #142, so the comment should be updated here.
3. docs/rqe_sketch_deployment_v1.md: the side-by-side table leaves out query memory and latency. Latency ((n-1)·c_mrg + G·c_qry for Hydra vs G·(c_qry + (n-1)·c_mrg) per group) is the main reason Hydra survives pruning, so it belongs in the table.

All 50 rqe-optimizer tests pass at 23ec5f7.

[milindsrivastava1997]

Replies to the three review-body notes (42e3216):

1. Hydra merge accuracy (accuracy_at): valid, and split out to rqe-optimizer: hydra-kll accuracy when a query merges several windows #166: measure Hydra at several merge counts, and/or refuse n > 1 until those rows exist. Not changed in this PR.
2. Stale export-script comment: rewritten. It now says the rows are priced as one shared fixed-size sketch, gated on subpopulations, and that the measured keys fan out (bench: measure hydra-kll without subset fan-out (one key per grouping) #165).
3. Doc table: added the query memory and latency rows.

🤖 Generated with Claude Code