hydra-kll: one key per label set, measured at ~100 / 1k / 10k groups (#165)

[milindsrivastava1997]

Closes #165.

The hydra-kll rows inserted every sample into all 2^D − 1 label subsets, because asap_sketchlib 0.2.2's Hydra::update splits keys on ;. The optimizer models one key per grouping. This PR makes the rows measure that sketch, at several group counts.

Design decisions (settled in a grilling session; please approve each)

1. Only hydra-kll changes. It is the only Hydra variant the optimizer prices. The hydra-cms, hydra-cs, hydra-hll and hydra-univmon rows keep the library's subset fan-out: they benchmark asap_sketchlib's Hydra as it ships, and nothing consumes them.
2. One insert per sample, on the framework Hydra. The hydra-kll rows join label values with \u{1f} (SERIES_SEPARATOR) instead of ;, so Hydra::update sees one part. Queries join with the same separator. The join happens at materialisation (peel_series), outside the timed insert.
3. Scored over the full label set. subpop-rank-error for hydra-kll groups by every label column, not just key1. The polars baseline groups by the full key (no fan_out), so subpopulations is the number of distinct label sets, which is what the optimizer's card(G) ≤ subpopulations gate assumes.
4. Rebased onto main (feat(rqe-optimizer): family properties, shared fixed-size sketches and DeltaSet key tracker #160, rqe-optimizer: split SumOrCount and TopK; deny unknown AtomicCostEntry fields #163). The measured_at_group_count comment in rqe-optimizer/src/candidates.rs, which cited this issue, is updated.
5. Group counts swept in this PR. Three specs differ only in key1's population (2 / 20 / 200 values). With key2 zipf over 50, the sketch holds about 100 / 1k / 10k groups. Row count, key2 and the value distribution are unchanged. The export runs every HYDRA_KLL_CONFIGS entry on each spec, merge-accuracy runs included. No pipeline or optimizer change is needed: same_cell keys on the full table description, so the specs are separate cells, and every cost row is its own candidate, gated by its own subpopulations.

Out of scope:

• Measured vs deployed implementation. ASAPQuery deploys the portable HydraKllSketch (xxh32, its own KLL, k=20 by default), not the framework Hydra measured here. Neither fans out, but their costs can differ. This is tracked in HydraKLL: the costed sketch (sketch-bench Hydra framework) is not the deployed one (portable HydraKllSketch) ASAPQuery#799. We decided to keep measuring the framework Hydra.
• Varying the number of label columns. Without fan-out, it only changes the key's length, not the number of groups.

Changes

• aqpbm-cli/src/rows/materialise.rs: peel_series joins labels with SERIES_SEPARATOR; peel_labeled (;) is unchanged for the other rows.
• aqpbm-cli/src/rows/statistic.rs: subpop_quantile_row uses peel_series and groups by every label column.
• sketch-bench/src/wrappers/hydra_kll/: SERIES_SEPARATOR. The lib wrapper queries the joined key; the polars baseline groups by the full key. The tests use full label sets, and a new test, kll_record_is_not_inserted_under_its_label_subsets, checks there is no fan-out.
• configs/datagen/hydra_kll_disjoint_labels_{100,1k,10k}.yaml: _10k is the old spec renamed, with its fan-out comments fixed.
• scripts/export_rqe_optimizer_costs.sh: loops over HYDRA_KLL_GROUP_COUNTS=(100 1k 10k).
• rqe-optimizer/src/candidates.rs: comment only.

Verification

Subpopulation rank error at rows=3 cols=256 cell_k=200, seed 42:

spec	subpopulations	lib mean / max rank err	polars mean / max
_100	100	0.0008 / 0.031	0 / 0
_1k	1000	0.018 / 0.20	0 / 0
_10k	9950	0.094 / 1.0	0 / 0

I ran one config through flatten and atomic-costs on all three specs: it gave 3 rows, 0 skipped, each with its own merge accuracy. cargo test and the rest of the pre-commit checks pass. I haven't run the full export script or regenerated the cost table.

🤖 Generated with Claude Code

[milindsrivastava1997]

Closed, because my undersanding of Hydra-KLL was wrong. If we want to maintain a set of KLLs, we cannot just maintain a Hydra like structure over it because that might only provide error guarantees for point queries, not for enumerating all keys. Similar to how, CMS only provides guarantees for point queries, and some heavy hitters, ot all queries.